# stream-processing Stream processing is a computing paradigm and method for continuously ingesting, processing, and analyzing data as it arrives rather than in batches. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Redpanda named a Leader in G2's Fall 2026 Event Stream Processing Reports DevFeed: [Redpanda named a Leader in G2's Fall 2026 Event Stream Processing Reports]() Original publisher: [Read original article]() Author: Redpanda Published: 2026-09-01T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [stream-processing](), [streaming-data-processing](), [Usability](), [real-time](), [Streaming](), [Software]() Tags: [2026](), [announcement](), [event](), [performance](), [real-time](), [reports](), [software](), [stream-processing](), [streaming](), [usability]() ### AI overview Redpanda reports that it was recognized in 10 G2 Fall 2026 Event Stream Processing reports, earning first place in eight indexes, including implementation, usability, results, and relationship. The article also highlights Redpanda Shadowing for migration from Confluent and cites a 4.7 out of 5 average rating across 53 G2 reviews. ### Source excerpt Redpanda ranked #1 in Implementation, Usability, Results, and Relationship Indexes in G2's Fall 2026 Event Stream Processing Reports. ## A Tale of Two Flink Autoscalers DevFeed: [A Tale of Two Flink Autoscalers]() Original publisher: [Read original article]() Author: Netflix Technology Blog Published: 2026-08-21T16:01:01Z Content type: article Language: en Sources: [Netflix](), [Netflix TechBlog - Medium]() Topics: [apache-flink](), [autoscaling](), [Netflix](), [stream-processing](), [Kafka](), [Open Source](), [telemetry](), [Network]() Tags: [apache-flink](), [autoscaling](), [aws](), [infrastructure](), [kafka](), [metrics](), [netflix](), [open-source](), [operational-excellence](), [scale](), [stream-processing](), [telemetry]() ### AI overview Netflix describes running two Apache Flink autoscalers: an older in-house system and a newer one from the Apache Flink community. The article explains why autoscaling is necessary for more than 30,000 Flink jobs across AWS regions, discusses workload variability and the cost of resizing stateful jobs, and introduces the metrics-driven design of the first autoscaler. ### Source excerpt Samuel Yeboah, Francesco Di Chiara and Mingliang Liu Today, Netflix runs two Flink autoscalers. That is exactly one more than we want. We built the first one in-house years ago, when there was no mature option suited to our platform. The second came from the Apache Flink community, and it can scale workloads our homegrown system was never designed for. We now run both in production and are steadily converging on the open-source one. Along the way we learned some hard lessons about metrics, cost, and the real price of maintaining infrastructure you could instead adopt, and we hope they are useful whether you run a handful of Flink jobs or tens of thousands. Why autoscaling is not optional at our scale Netflix has run stream processing on Apache Flink since 2017. As of 2026 we operate more than 30,000 Flink jobs across multiple AWS regions. Most are not deployed by hand; they are generated by our managed platform Data Mesh, so the majority of users never touch a Flink job directly. A smaller but growing set are custom jobs, built and operated by teams across the company for use cases like personalization, Ads, and Live events. They range from single-operator jobs that shuttle records between Kafka topics to stateful pipelines with branches, joins, and terabytes of state, and their load swings with daily cycles, launches, and regional failovers. Provisioning every one of those jobs for its peak is wasteful; provisioning for the average causes lag during surges. And in our platform a scaling action is not free: by default it means taking a savepoint, stopping the job gracefully, and restarting it at the new size, which for a large stateful job can take minutes. That leaves a genuinely hard question: how do you give each job the resources it needs, when it needs them, without a human in the loop and without breaking anything? The first autoscaler: watching from outside Our first answer, built around 2019, was an autoscaler shaped like a stream-processing job. It ran on M ## Confluent Cloud for Apache Flink: Engine for Mission-Critical, Real-Time Operational Systems and dbt/SQL-Native Home for Data Science and AI DevFeed: [Confluent Cloud for Apache Flink: Engine for Mission-Critical, Real-Time Operational Systems and dbt/SQL-Native Home for Data Science and AI]() Original publisher: [Read original article]() Author: Yashwanth Dasari Published: 2026-08-18T02:20:00Z Content type: release Language: en Sources: [Confluent: Data in motion]() Topics: [Confluent Cloud](), [apache-flink](), [stream-processing](), [Kafka](), [AI, ML & Data Engineering](), [Cloud](), [SDKs]() Tags: [ai](), [announcement](), [apache-flink](), [api](), [batch](), [confluent-cloud](), [data-science](), [developers](), [kafka](), [real-time]() ### AI overview Confluent announces an evolution of Confluent Cloud for Apache Flink that unifies mission-critical real-time operations with analytics and AI workflows. The release adds a serverless, co-designed Kafka and Flink platform, including the generally available Flink Table API in Java for code-first development. ### Source excerpt Flink now acts as a robust engine for developers through Table API, UDFs, and PTFs while offering a SQL-native, dbt-integrated platform for data science and AI teams. ## Multi-region high availability for Kafka workloads with a single Stretch Cluster DevFeed: [Multi-region high availability for Kafka workloads with a single Stretch Cluster]() Original publisher: [Read original article]() Author: David Yu Published: 2026-08-11T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Kafka](), [Kubernetes](), [Availability](), [Replication](), [Raft](), [Resilience](), [stream-processing]() Tags: [clusters](), [k8s](), [kafka](), [raft](), [release](), [replication](), [resilience](), [stream-processing]() ### AI overview Redpanda Operator 26.2 introduces generally available Stretch Clusters, allowing one logical Redpanda cluster to span multiple Kubernetes clusters and regions. The release provides synchronous replication with Raft-based automatic failover, safer broker rolling restarts, Redpanda Connect pipelines as Kubernetes resources, and Gateway API support for Redpanda Console. ### Source excerpt Redpanda Operator 26.2 brings GA Stretch Clusters for multi-region replication, Redpanda Connect pipelines as K8s resources, Gateway API support, and safer rolling restarts. ## Agentic kill switch is a database problem. So we built Redpanda SQL DevFeed: [Agentic kill switch is a database problem. So we built Redpanda SQL]() Original publisher: [Read original article]() Author: Alexander Gallego Published: 2026-08-03T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Databases](), [SQL](), [stream-processing](), [Google Cloud Platform (GCP)](), [Amazon Web Services](), [systems]() Tags: [agentic](), [announcement](), [aws](), [cloud](), [code](), [database](), [exploit](), [google-cloud](), [governance](), [models](), [olap](), [real-time](), [scale](), [sql](), [stream-processing](), [systems](), [tools]() ### AI overview The article presents Redpanda SQL as a database built for agentic governance. It describes real-time bridge queries, efficient SQL JOINs, a ring-buffer shuffle algorithm, and analytical processing designed to detect agent behavior and provide an agentic kill switch. Redpanda SQL is stated to be available on AWS and Google Cloud. ### Source excerpt Agentic governance needs a new kind of database, so we built Redpanda SQL. Now available on both AWS and Google Cloud. ## Drag, drop, done: a visual composer for Redpanda Connect DevFeed: [Drag, drop, done: a visual composer for Redpanda Connect]() Original publisher: [Read original article]() Author: Prakhar Garg Published: 2026-07-23T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Redpanda-Connect](), [Canvas](), [stream-processing](), [Caching](), [Routing (disambiguation)](), [HTTP](), [n8n](), [workflow automation]() Tags: [api](), [automation](), [building](), [cache](), [caching](), [events](), [front-end](), [http](), [n8n](), [pipeline](), [redpanda-connect](), [routing](), [stream-processing](), [workflow]() ### AI overview Redpanda introduces a visual Pipeline Builder for Redpanda Connect, using a drag-and-drop canvas to compose pipelines from inputs, processors, and outputs. The builder generates YAML while making routing, enrichment, caching, error handling, and retries easier to understand and configure. The article contrasts it with n8n, emphasizing Redpanda Connect's suitability for high-volume event streaming with ordering, backpressure, and exactly-once semantics. ### Source excerpt A first look at the new visual Pipeline Builder for Redpanda Connect (preview), plus a roundup of recent CDC and connector updates. ## Michael Drogalis on Building ShadowTraffic as a Solo B2B Software Company DevFeed: [Michael Drogalis on Building ShadowTraffic as a Solo B2B Software Company]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-15T13:24:12Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [Software Engineering](), [Kafka](), [stream-processing]() Tags: [kafka](), [software-engineering](), [stream-processing]() ### AI overview A conversation with Michael Drogalis about building ShadowTraffic as a solo B2B software company. It discusses realistic production-traffic generation for testing, demos, and development, along with customer discovery, marketing, AI use, and engineering careers. ### Source excerpt A Conversation with Michael Drogalis ## Introducing low-code MCP servers in Redpanda Cloud DevFeed: [Introducing low-code MCP servers in Redpanda Cloud]() Original publisher: [Read original article]() Author: Johannes Brüderl Published: 2025-10-28T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Model Context Protocol (MCP)](), [Low code](), [AI Development](), [Cloud](), [YAML](), [stream-processing](), [PostgreSQL](), [Redis]() Tags: [ai](), [ai-agents](), [announcement](), [anthropic](), [apis](), [applications](), [cloud](), [connectors](), [kafka](), [low-code](), [mcp](), [postgresql](), [redis](), [servers](), [tool](), [yaml]() ### AI overview This article introduces Redpanda Cloud Remote MCP, a managed service for building MCP servers with low-code YAML. It explains how the service uses Redpanda Connect and more than 300 connectors to link data sources, APIs, and databases with AI applications, enabling standardized access to live organizational context. ### Source excerpt Introducing Redpanda Cloud Remote MCP, a managed solution for low-code MCP servers that easily integrates your data sources with AI apps. ## Redpanda open-sources top 16 AI connectors DevFeed: [Redpanda open-sources top 16 AI connectors]() Original publisher: [Read original article]() Author: Mike Broberg Published: 2025-08-18T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Redpanda-Connect](), [Open Source](), [Artificial Intelligence](), [stream-processing](), [OpenAI](), [Streaming](), [Amazon Bedrock](), [Embeddings](), [Retrieval Augmented Generation (RAG)](), [cohere](), [Ollama]() Tags: [ai](), [ai-connector-platforms](), [ai-connectors-for-commercial-products](), [ai-connectors-open-source](), [ai-data-streaming](), [ai-integration-tools](), [announcement](), [bedrock](), [cloud-native-ai-services](), [cohere](), [embeddable-ai-capabilities](), [embeddings](), [ollama](), [open-source](), [open-source-ai-for-business](), [openai](), [openai-connectors](), [rag](), [real-time-ai-streaming](), [redpanda-ai-connectors](), [redpanda-connect](), [stream-processing](), [streaming](), [streaming-ai-processors]() ### AI overview Redpanda announces the open-source release of its top AI connectors under the Apache 2.0 license. The connectors integrate Redpanda Connect with destinations and models including OpenAI, Cohere, Amazon Bedrock, Ollama, and Google Cloud Vertex AI, supporting streaming pipelines and use cases such as generation, summarization, classification, translation, and text embeddings for RAG. ### Source excerpt Redpanda open-sources top AI connectors to the most used destinations, including OpenAI, Cohere, Bedrock, Ollama, and Vertex AI. Learn more. ## Backfilling Postgres TOAST Columns in Debezium Data Change Events DevFeed: [Backfilling Postgres TOAST Columns in Debezium Data Change Events]() Original publisher: [Read original article]() Published: 2025-05-26T14:40:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Databases](), [Replication](), [apache-flink](), [stream-processing](), [SQL](), [API]() Tags: [apache-flink](), [data](), [debezium](), [debezium-connector](), [flink](), [postgres](), [replication](), [sql](), [stateful](), [stream-processing]() ### AI overview This article explains how Postgres logical replication handles TOAST columns in Debezium change events. It discusses Debezium's reselect postprocessor and explores Apache Flink solutions using stateful stream processing, Flink SQL, and process table functions. ### Source excerpt Table of Contents Debezium Reselect Postprocessor Flink DataStream API Flink SQL With OVER Aggregation Flink Process Table Functions Summary and Discussion Postgres logical replication, while powerful for capturing real-time data changes, presents challenges with TOAST columns, whose values can be absent from data change events in specific situations. This post discusses how Debezium addresses this through its built-in reselect post processor, then explores more robust solutions leveraging Apache Flink's capabilities for stateful stream processing, including Flink SQL and the brand-new process table functions (PTFs) in Flink 2.1. ## Streaming and Batch Processing Are Complementary; Pull Versus Push Is the Key Distinction DevFeed: [Streaming and Batch Processing Are Complementary; Pull Versus Push Is the Key Distinction]() Original publisher: [Read original article]() Published: 2025-05-14T08:10:00Z Content type: opinion Language: en Sources: [Gunnar Morling]() Topics: [Streaming](), [stream-processing](), [streaming-data-processing](), [real-time](), [systems](), [data]() Tags: [kafka](), [latency](), [low-latency](), [performance](), [real-time](), [stream-processing](), [streaming](), [systems]() ### AI overview The article argues that streaming systems commonly use batching to improve throughput, so streaming and batch processing are not opposites. It proposes pull versus push semantics as the more meaningful distinction and explains that push-based streaming can provide timely updates, while adding complexity around state, joins, and out-of-order data. ### Source excerpt Often times, "Stream vs. Batch" is discussed as if it's one or the other, but to me this does not make that much sense really. ## KIP-932 explores queue semantics and Share Groups for Apache Kafka DevFeed: [KIP-932 explores queue semantics and Share Groups for Apache Kafka]() Original publisher: [Read original article]() Published: 2025-03-05T11:35:00Z Content type: opinion Language: en Sources: [Gunnar Morling]() Topics: [Kafka](), [Streaming](), [stream-processing](), [Architecture & Design]() Tags: [apache-kafka](), [fraud](), [fraud-detection](), [job](), [kafka](), [microservices](), [stream-processing](), [streaming]() ### AI overview The article examines KIP-932, which explores adding queue semantics to Apache Kafka through Share Groups. It explains how Kafka's partition-based consumer model limits parallelism and ordered processing, and why individual message acknowledgment and rejection are important for queueing workloads such as independent job processing. ### Source excerpt Table of Contents Towards Queue Support in Kafka--Introducing Share Groups Share Groups in Action Retry Behavior and State Management Share Group State Persistence Summary and Outlook In the "Let's Take a Look at...!" blog series I am going to explore interesting projects, developments and technologies in the data and streaming space. This can be KIPs and FLIPs, open-source projects, services, and more. The idea is to get some hands-on experience, learn about potential use cases and applications, and understand the trade-offs involved. If you think there's a specific subject I should take a look at, let me know in the comments below! That guy above? Yep, that's me, whenever someone says "Kafka queue". Because, that's not what Apache Kafka is. At its core, Kafka is a distributed durable event log. Producers write events to a topic, organized in partitions which are distributed amongst the brokers of a Kafka cluster. Consumers, organized in groups, divide the partitions they process amongst themselves, so that each partition of a topic is read by exactly one consumer in the group. ## Kafka for System Design Interviews DevFeed: [Kafka for System Design Interviews]() Original publisher: [Read original article]() Author: Evan King Published: 2024-07-01T03:18:09Z Content type: tutorial Language: en Sources: [Evan King]() Topics: [Kafka](), [Apache-Kafka](), [Event-Streaming](), [stream-processing](), [Scalability](), [data]() Tags: [apache-kafka](), [distributed](), [durability](), [google](), [high-performance](), [kafka](), [meta](), [open-source](), [real-time](), [scalability](), [stream-processing](), [system-design-interview]() ### AI overview A tutorial on Apache Kafka for system design interviews. It explains Kafka as an open-source distributed event streaming platform that can function as a message queue or stream processing system, then uses a real-time sports statistics example to discuss producers, consumers, scaling, and event ordering. ### Source excerpt By the Co-founder of www.hellointerview.comIntro There is a good chance you've heard of Kafka. It's popular. In fact, according to their website, it's used by 80% of the Fortune 100. If it's good enough to help scale the largest companies in the world, it's probably good enough for your next system design interview. In fact, it's one of the top 5 technologies we see used in design interviews. Apache Kafka is an open-source distributed event streaming platform that can be used either as a message queue or as a stream processing system. Kafka excels in delivering high performance, scalability, and durability. It's engineered to handle vast volumes of data in real-time, ensuring that no message is ever lost and that each piece of data is processed as swiftly as possible. In this deep dive, we're going to take a top down approach. Starting with a zoomed out view of Kafka and progressing into more and more detail. If you know the basics, feel free to skip ahead to the more advanced sections. A Motivating Example It's the World Cup (my personal favorite competition). And we run a website that provides real-time statistics on the matches. Each time a goal is scored, a player is booked, or a substitution is made, we want to update our website with the latest information. Events are placed on a queue when they occur. We call the server or process responsible for putting these events on the queue the producer. Downstream, we have a server that reads events off the queue and updates the website. We call this the consumer. Now, imagine the World Cup expanded from just the top 48 teams to a hypothetical 1,000-team tournament, and all the games are now played at the same time. The number of events has increased significantly, and our single server hosting the queue is struggling to keep up. Similarly, our consumer feels like it has its mouth under a firehose and is crashing under the load. We need to scale the system by adding more servers to distribute our queue. But how do we e ## Evolving the Backend Storage for Platform Metrics DevFeed: [Evolving the Backend Storage for Platform Metrics]() Original publisher: [Read original article]() Author: David Murray Published: 2024-05-09T22:00:00Z Content type: article Language: en Sources: [Heroku]() Topics: [Heroku](), [Back end](), [Time Series](), [Apache Cassandra](), [Kafka](), [stream-processing](), [autoscaling]() Tags: [apache-kafka](), [app-architecture](), [autoscaling](), [aws](), [backend](), [cassandra](), [cloud-infrastructure](), [data-analytics](), [database](), [developer-tools](), [engineering](), [heroku](), [kafka](), [metrics](), [performance-optimization](), [platform](), [platform-updates](), [stream-processing](), [time-series]() ### AI overview Heroku describes an infrastructure upgrade that migrated the storage backend for platform metrics. The article explains how MetaaS collects application observations, aggregates them into time-series statistics, and uses Kafka, stream-processing jobs, and Cassandra to support dashboard metrics, alerting, autoscaling, retention, and querying. ### Source excerpt One of our most important goals at Heroku is to be boring. Don't get us wrong, we certainly hope that you're excited about the Heroku developer experience -- as heavy users of Heroku ourselves, we certainly are! But, even more so, we hope that you don't have to spend all that much time thinking about [...] The post Evolving the Backend Storage for Platform Metrics appeared first on Heroku. ## Getting Started With PyFlink on Kubernetes DevFeed: [Getting Started With PyFlink on Kubernetes]() Original publisher: [Read original article]() Published: 2023-12-07T00:00:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [apache-flink](), [Kubernetes](), [stream-processing](), [Kafka](), [Python](), [data-engineering]() Tags: [apache-kafka](), [flink](), [kafka](), [kubernetes](), [operator](), [programming](), [python](), [stream-processing]() ### AI overview A practical guide to running PyFlink on Kubernetes with Apache Flink 1.18. It covers installing the Flink Kubernetes Operator and Kafka with Strimzi, creating a PyFlink job that generates data and writes it to Kafka using Flink SQL, building a container image, and deploying the job to Kubernetes. ### Source excerpt Table of Contents What Is PyFlink and Why Should You Care? Prerequisites Installing the Flink Kubernetes Operator Installing Strimzi and Apache Kafka A Simple PyFlink Job Building a Container Image With Your PyFlink Job Deploying a PyFlink Job On Kubernetes This post originally appeared on the Decodable blog. All rights reserved. The other day, I wanted to get my feet wet with PyFlink. While there is a fair amount of related information out there, I couldn't find really up-to-date documentation on using current versions of PyFlink with Flink on Kubernetes. ## Tinybird: A ksqlDB alternative for stateful stream processing DevFeed: [Tinybird: A ksqlDB alternative for stateful stream processing]() Original publisher: [Read original article]() Author: Alasdair Brown Published: 2023-07-27T00:00:00Z Content type: comparison Language: en Sources: [Tinybird]() Topics: [streaming-data-processing](), [stream-processing](), [Kafka]() Tags: [compare](), [kafka](), [scalable-analytics-architecture](), [stateful](), [stream-processing](), [streaming]() ### AI overview This comparison presents Tinybird as an alternative to ksqlDB for stateful stream processing and streaming SQL, emphasizing an approach intended to avoid the complexity associated with Kafka Streams. ### Source excerpt Looking for a ksqlDB alternative? Tinybird gives you streaming SQL without the Kafka Streams complexity. Compare the approaches. ## Why I Joined Decodable DevFeed: [Why I Joined Decodable]() Original publisher: [Read original article]() Published: 2022-11-03T14:00:00Z Content type: opinion Language: en Sources: [Gunnar Morling]() Topics: [stream-processing](), [streaming-data-processing](), [real-time](), [Serverless](), [Kafka](), [datasets](), [Apache Pulsar](), [amazon]() Tags: [amazon](), [apache](), [apache-kafka](), [kafka](), [latency](), [low-latency](), [news](), [platform](), [real-time](), [serverless](), [stream-processing]() ### AI overview A software engineer explains why he joined Decodable after several years at Red Hat, citing the field of real-time stream processing, the startup environment, and the team. He describes his experience with Debezium and his interest in helping users implement end-to-end streaming use cases. ### Source excerpt Table of Contents The Space: Real-Time Stream Processing The Environment: A Start-up The Team: One of a Kind Outlook It's my first week as a software engineer at Decodable, a start-up building a serverless real-time data platform! When I shared this news on social media yesterday, folks were not only super supportive and excited for me (thank you so much for all the nice words and wishes!), but some also asked about the reasons behind my decision for switching jobs and going to a start-up, after having worked for Red Hat for the last few years. ## Bullet Updates - Windowing, Apache Pulsar PubSub, Configuration-based Data Ingestion, and More DevFeed: [Bullet Updates - Windowing, Apache Pulsar PubSub, Configuration-based Data Ingestion, and More]() Original publisher: [Read original article]() Author: rosaliebeevm-blog Published: 2019-03-08T17:12:50Z Content type: release Language: en Sources: [Yahoo]() Topics: [streaming-data-processing](), [stream-processing](), [Query (disambiguation)](), [Streaming](), [Apache Spark]() Tags: [announcements](), [big-data](), [code](), [java](), [javascript](), [open-source](), [releases](), [spark](), [stream](), [stream-processing](), [streaming](), [updates](), [yahoo-engineering]() ### AI overview This update describes new windowing support in Bullet, an open-source query system for data flowing through streaming systems. It explains time- and record-based windows, including tumbling, sliding, and hopping window patterns, for returning intermediate query results. ### Source excerpt yahoodevelopers: By Akshay Sarma, Principal Engineer, Verizon Media & Brian Xiao, Software Engineer, Verizon Media This is the first of an ongoing series of blog posts sharing releases and announcements for Bullet, an open-sourced lightweight, scalable, pluggable, multi-tenant query system. Bullet allows you to query any data flowing through a streaming system without having to store it first through its UI or API. The queries are injected into the running system and have minimal overhead. Running hundreds of queries generally fit into the overhead of just reading the streaming data. Bullet requires running an instance of its backend on your data. This backend runs on common stream processing frameworks (Storm and Spark Streaming currently supported). The data on which Bullet sits determines what it is used for. For example, our team runs an instance of Bullet on user engagement data (~1M events/sec) to let developers find their own events to validate their code that produces this data. We also use this instance to interactively explore data, throw up quick dashboards to monitor live releases, count unique users, debug issues, and more. Since open sourcing Bullet in 2017, we've been hard at work adding many new features! We'll highlight some of these here and continue sharing update posts for future releases. Windowing Bullet used to operate in a request-response fashion - you would submit a query and wait for the query to meet its termination conditions (usually duration) before receiving results. For short-lived queries, say, a few seconds, this was fine. But as we started fielding more interactive and iterative queries, waiting even a minute for results became too cumbersome. Enter windowing! Bullet now supports time and record-based windowing. With time windowing, you can break up your query into chunks of time over its duration and retrieve results for each chunk. For example, you can calculate the average of a field, and stream back results every second: In th ## Analyzing Twitter Location Data with Heron, Machine Learning, Google's NLP, and BigQuery DevFeed: [Analyzing Twitter Location Data with Heron, Machine Learning, Google's NLP, and BigQuery]() Original publisher: [Read original article]() Author: Christian Hollinger Published: 2018-03-18T00:00:00Z Content type: tutorial Language: en Sources: [Christian Hollinger]() Topics: [streaming-data-processing](), [stream-processing](), [data](), [Natural language processing](), [Machine Learning & Artificial Intelligence](), [BigQuery](), [Google Cloud Platform (GCP)](), [Google]() Tags: [analytics](), [article](), [big-data](), [bigquery](), [cloud](), [data](), [distributed](), [gcp](), [google](), [google-cloud](), [hbase](), [heron](), [java](), [machine-learning](), [natural-language-processing](), [nlp](), [programming](), [storm](), [stream](), [stream-processing]() ### AI overview A tutorial describing a pipeline that processes public Twitter data with Heron, Google's NLP tools, Nominatim reverse geocoding, machine learning clustering, and Google Cloud storage and visualization services. The project maps inferred user locations and explores privacy implications. ### Source excerpt In this article, we will use Heron, the distributed stream processing and analytics engine from Twitter, together with Google's NLP toolkit, Nominatim and some Machine Learning as well as Google's BigTable, BigQuery, and Data Studio to plot Twitter user's assumed location across the US. ## One-box stream processing with CSP DevFeed: [One-box stream processing with CSP]() Original publisher: [Read original article]() Published: 2018-02-18T00:00:00Z Content type: tutorial Language: en Sources: [Adam Bard]() Topics: [streaming-data-processing](), [Clojure](), [stream-processing](), [Concurrent Programming](), [distributed-systems]() Tags: [clojure](), [concurrent](), [core-async](), [coroutine](), [distributed-computing](), [distributed-system](), [flink](), [spark](), [stream](), [stream-processing]() ### AI overview This article presents a small-scale stream-processing architecture in Clojure using core.async and component libraries. It describes modular components connected by asynchronous queues and explains how stream-processing design principles can support reusable, loosely coupled systems without requiring a distributed stream processor. ### Source excerpt If you're like me (that is, employed by an ad tech company), stream processing is usually associated with frameworks like Storm, Flink, Spark Streaming, and other such solutions. However, a lot of real-life software can be described as stream processing - data comes in one end, is transformed or aggregated, and goes somewhere else. Many of these workloads don't justify the overhead of a stream processor, but that doesn't mean they can't benefit from some of the lessons of stream processing systems. ## Live Video Transmuxing/Transcoding: FFmpeg vs TwitchTranscoder, Part I DevFeed: [Live Video Transmuxing/Transcoding: FFmpeg vs TwitchTranscoder, Part I]() Original publisher: [Read original article]() Author: Yueshi Shen Published: 2017-10-10T18:40:03Z Content type: article Language: en Sources: [Twitch]() Topics: [FFmpeg (Fast Forward Moving Picture Experts Group)](), [Twitch](), [Transcodings](), [Streaming](), [HTTP](), [stream-processing]() Tags: [architecture](), [engineering](), [ffmpeg](), [http](), [live-streaming](), [open-source](), [pipeline](), [stream-processing](), [streaming](), [transcoder](), [twitch](), [video]() ### AI overview This first part of a two-part series explains Twitch's live video processing pipeline and compares using FFmpeg with Twitch's in-house TwitchTranscoder. It describes how RTMP uploads are converted into multiple-bitrate HLS variants and outlines the live transcoding requirements that FFmpeg meets or does not meet. ### Source excerpt By: Jeff Gong, Software Engineer, jeffgon@twitch.tv Sahil Dhanju, Software Engineer Intern Chih-Chiang Lu, Senior Software Engineer, chihchil@twitch.tv Yueshi Shen, Principal Research Engineer, yshen@twitch.tv Special thanks go to Christopher Kennedy, Staff Video Engineer at Crunchyroll/Ellation John Nichols, Principal Software Engineer at Xilinx, jnichol@xilinx.com for their information on FFmpeg and reviewing this article. Note: This is the first part of a 2-part series. Please read part 2 after finishing this article. Background Twitch is the world's leading live streaming platform for video games, esports, and other emerging creative content. Every month, more than 2.2 million unique content creators stream or upload video on our website. At its peak, Twitch ingests tens of thousands of concurrent live video streams and delivers them to viewers across the world. Figure 1 depicts the architecture of our live video Content Delivery Network (CDN) which delivers tens of thousands of concurrent live streams internationally. Figure 1: Architecture of Twitch's Live Video CDN Twitch, like many other live streaming services, receives live stream uploads in Real-Time Messaging Protocol (RTMP) from its broadcasters. RTMP is a protocol designed to stream video and audio on the Internet, and is mainly used for point to point communication. To then scale our live stream content to countless viewers, Twitch uses HTTP Live Streaming (HLS), an HTTP-based media streaming communications protocol that most video websites also use. Within the live stream processing pipeline, the transcoder module is in charge of converting an incoming RTMP stream into the HLS format with multiple variants (e.g., 1080p, 720p, etc.). These variants have different bitrates so that viewers with different levels of download bandwidth are able to consume live video streams at the best possible quality for their connection. Figure 2 depicts the input and output of the transcoder module in our live video CD ## Storm vs. Heron, Part 1: Reusing a Storm topology for Heron DevFeed: [Storm vs. Heron, Part 1: Reusing a Storm topology for Heron]() Original publisher: [Read original article]() Author: Christian Hollinger Published: 2016-10-15T00:00:00Z Content type: article Language: en Sources: [Christian Hollinger]() Topics: [stream-processing](), [streaming-data-processing](), [Hadoop](), [hdfs](), [Maven](), [Deployment]() Tags: [apache](), [big-data](), [configuration](), [data](), [deployment](), [github](), [hadoop](), [hdfs](), [heron](), [java](), [maven](), [processing](), [spark](), [storm](), [topology](), [vs]() ### AI overview This article examines upgrading an Apache Storm topology to Twitter Heron. Using a Twitter analysis pipeline that processes tweets and stores results in HDFS, it reports compatibility problems with the storm-hdfs library, including a missing Storm class, and discusses integrating or rewriting the library as possible workarounds. ### Source excerpt This article is part 1 of an upcoming article series, Storm vs. Heron. ## Real-time Big Data at Target DevFeed: [Real-time Big Data at Target]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2015-11-11T06:00:00Z Content type: article Language: en Sources: [Target]() Topics: [big-data](), [Hadoop](), [stream-processing](), [streaming-data-processing](), [Kafka](), [Latency](), [Low Latency](), [Open Source]() Tags: [analytics](), [apache-kafka](), [batch](), [big-data](), [data](), [hadoop](), [kafka](), [latency](), [low-latency](), [open-source](), [real-time](), [storm](), [stream-processing](), [streaming]() ### AI overview Target's Big Data platform team describes its effort to move data into Hadoop in real time using flexible open source components. The article outlines requirements for resilience, message-format support, usability, and low latency, then evaluates Apache Storm and identifies recovery and batching-related latency issues in testing. ### Source excerpt An enterprise as large as Target generates a lot of data and on my Big Data platform team we want to make it as easy as possible for our users to get it into Hadoop in real-time. I want to discuss how we are starting to approach this problem, what we've done so far, and what is still to come. Our requirements We wanted to build a system with flexible open source components. Our experience with proprietary products on Hadoop is that they tend to be inflexible and only work with a narrow set of use cases. That can also be true of open source products, but we have found them to be easier to adapt to our needs. Indeed, that ended up being the case as we went through this journey and made several contributions to Apache projects. More specifically, we wanted to create a system that: was highly resilient to the failure of any individual component supported a variety of message formats delivered data to Hadoop with very low latency made the data immedaiately usable streamed data from Apache Kafka sources A Streaming Framework There are many excellent comparisons of streaming frameworks available, and I won't attempt to recreate them here. The first criteria we considered was what tooling was needed to monitor and administer the streaming framework, with a strong preference to use our primary Hadoop administration tool, Apache Ambari. Apache Storm fit that bill and was also a proven solution for stream processing. If Storm could meet our other requirements, it would be our first choice. To test its resiliency we ran a simple scenario: start a data stream into Hadoop, disable HDFS, and then reenable it. Streaming would obviously fail while HDFS was disabled, but we needed the system to recover gracefully when HDFS came back online. Unfortunately our first test of this scenario left our Storm topology in an unrecoverable state, which required a manual restart. That's not something we could live with. We also needed very fine control over the latency of arriving data. In gener