# streaming-data-processing A data-processing approach that continuously processes data from unbounded streams as it arrives. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Redpanda named a Leader in G2's Fall 2026 Event Stream Processing Reports DevFeed: [Redpanda named a Leader in G2's Fall 2026 Event Stream Processing Reports]() Original publisher: [Read original article]() Author: Redpanda Published: 2026-09-01T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [stream-processing](), [streaming-data-processing](), [Usability](), [real-time](), [Streaming](), [Software]() Tags: [2026](), [announcement](), [event](), [performance](), [real-time](), [reports](), [software](), [stream-processing](), [streaming](), [usability]() ### AI overview Redpanda reports that it was recognized in 10 G2 Fall 2026 Event Stream Processing reports, earning first place in eight indexes, including implementation, usability, results, and relationship. The article also highlights Redpanda Shadowing for migration from Confluent and cites a 4.7 out of 5 average rating across 53 G2 reviews. ### Source excerpt Redpanda ranked #1 in Implementation, Usability, Results, and Relationship Indexes in G2's Fall 2026 Event Stream Processing Reports. ## New in Confluent Intelligence and AI Tools: Making Agents Native to the Stream, Expanded Model Support, New Agent Skills, and Copilot DevFeed: [New in Confluent Intelligence and AI Tools: Making Agents Native to the Stream, Expanded Model Support, New Agent Skills, and Copilot]() Original publisher: [Read original article]() Author: Confluent Staff Published: 2026-08-18T14:00:10Z Content type: article Language: en Sources: [Confluent: Data in motion]() Topics: [Artificial Intelligence](), [apache-flink](), [Kafka](), [streaming-data-processing](), [real-time](), [MCP Server](), [Agent Skills](), [MCP](), [Inference](), [Google](), [ibm]() Tags: [ai](), [ai-ready-data](), [ai-tools](), [apache-flink](), [confluent](), [confluent-cloud](), [google](), [ibm](), [kafka](), [mcp](), [mcp-server](), [real-time](), [skills](), [streaming-data-processing](), [support](), [tools]() ### AI overview Confluent announces updates to Confluent Intelligence and related AI tools for building production AI systems on Apache Kafka and Apache Flink. The release adds expanded time-series model support, a generally available Real-Time Context Engine, updates to the fully managed MCP Server, Agent Skills for AI coding assistants, and Confluent Copilot. The features are intended to provide agents and applications with fresh business context and governed access to live data and infrastructure. ### Source excerpt Explore new AI features and AI tools: support for IBM Granite Time Series models and TimesFM models (EA), enhanced Real-Time Context Engine experience, new Agent Skills, Confluent Copilot ## Real-time streaming for the agentic era with NVIDIA DevFeed: [Real-time streaming for the agentic era with NVIDIA]() Original publisher: [Read original article]() Author: Melissa Czapiga Published: 2026-06-01T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Vera CPU](), [streaming-data-processing](), [Latency](), [Data Infrastructure](), [agent observability](), [systems]() Tags: [agentic](), [agents](), [announcement](), [compute](), [data-infrastructure](), [latency](), [nvidia](), [nvidia-vera](), [performance](), [streaming](), [systems]() ### AI overview Redpanda describes its collaboration with NVIDIA to run streaming workloads on NVIDIA Vera CPUs, reporting 5.5x lower latency for AI agents in mission-critical environments. The article discusses testing on workloads processing billions of messages daily, real-time data infrastructure, and potential benefits for enterprise AI and agentic applications. ### Source excerpt NVIDIA Vera launches today with Redpanda as part of the ecosystem, delivering 5.5x lower latencies for agents running in mission-critical environments. ## Event-Driven Architecture with Apache Kafka DevFeed: [Event-Driven Architecture with Apache Kafka]() Original publisher: [Read original article]() Author: hello@ankit-rana.com Published: 2026-03-16T00:00:00Z Content type: tutorial Language: en Sources: [Ankit Rana | Mechanical Sympathy]() Topics: [event driven](), [Kafka](), [Apache-Kafka](), [Scalability](), [Resilience](), [streaming-data-processing](), [Latency](), [systems]() Tags: [apache-kafka](), [architecture](), [asynchronous](), [consumer](), [distributed-systems](), [event-driven-architecture](), [event-sourcing](), [kafka](), [microservices](), [partitions](), [real-time](), [resilience](), [scalability](), [schemas](), [stream-processing]() ### AI overview This tutorial explains event-driven architecture and how Apache Kafka supports asynchronous, real-time data processing. It covers producers, consumers, immutable events, event sourcing, scalability, resilience, stream-processing pipelines, and challenges such as ordering, debugging, and eventual consistency. ### Source excerpt Event-driven architecture replaces synchronous point-to-point calls with immutable events on a durable log, so producers and consumers scale and fail independently. Kafka provides that log: topics sharded into ordered append-only partitions, replicated across brokers, with consumer groups sharing partitions to scale read throughput. ## Why Flink may be unnecessarily complex for most streaming data processing users DevFeed: [Why Flink may be unnecessarily complex for most streaming data processing users]() Original publisher: [Read original article]() Author: Javi Santana Published: 2025-10-21T00:00:00Z Content type: opinion Language: en Sources: [Tinybird]() Topics: [streaming-data-processing](), [Streaming]() Tags: [flink](), [scalable-analytics-architecture](), [streaming](), [streaming-data-processing]() ### AI overview The article argues that Flink's complexity may make it unnecessary for most people who need streaming data processing. ### Source excerpt Flink might sound like the holy grail of streaming data processing, but for 95% of us, it's just a complex headache we don't need. ## Introducing multi-language dynamic plugins for Redpanda Connect DevFeed: [Introducing multi-language dynamic plugins for Redpanda Connect]() Original publisher: [Read original article]() Author: James Kinley Published: 2025-06-17T00:00:00Z Content type: release Language: en Sources: [Redpanda]() Topics: [gRPC](), [SDKs](), [Go Language](), [Python](), [Processes](), [streaming-data-processing](), [Unix]() Tags: [ai-ml-capabilities-in-streaming-data](), [architecture](), [configuration](), [creating-plugins-in-golang-and-python](), [dynamic-vs-compiled-plugins](), [framework](), [go](), [grpc-plugin-system](), [integration](), [interfaces](), [ipc](), [language-agnostic-plugin-system](), [multi-language-plugin-development](), [pipelines](), [plugin-development](), [plugins](), [process](), [processes](), [product](), [protocol](), [python](), [python-sdk-for-streaming-data](), [redpanda-connect](), [redpanda-connect-dynamic-plugins](), [redpanda-streaming-infrastructure](), [runtime-loaded-plugins-grpc](), [sdk](), [sdks](), [stream-processing](), [streaming-data-pipeline-plugins]() ### AI overview Redpanda Connect introduces Apache 2.0-licensed dynamic plugins in Beta with version 4.56.0. Plugins can be loaded at runtime as external executables that communicate with the main process through gRPC over Unix sockets, enabling plugin development in Go and Python. Official SDKs support both languages, while native Go plugins remain the preferred option for performance-critical workloads. ### Source excerpt Redpanda Connect dynamic plugins framework allows you to create and load plugins at runtime, opening up a world of new integration possibilities beyond Go. ## Streaming and Batch Processing Are Complementary; Pull Versus Push Is the Key Distinction DevFeed: [Streaming and Batch Processing Are Complementary; Pull Versus Push Is the Key Distinction]() Original publisher: [Read original article]() Published: 2025-05-14T08:10:00Z Content type: opinion Language: en Sources: [Gunnar Morling]() Topics: [Streaming](), [stream-processing](), [streaming-data-processing](), [real-time](), [systems](), [data]() Tags: [kafka](), [latency](), [low-latency](), [performance](), [real-time](), [stream-processing](), [streaming](), [systems]() ### AI overview The article argues that streaming systems commonly use batching to improve throughput, so streaming and batch processing are not opposites. It proposes pull versus push semantics as the more meaningful distinction and explains that push-based streaming can provide timely updates, while adding complexity around state, joins, and out-of-order data. ### Source excerpt Often times, "Stream vs. Batch" is discussed as if it's one or the other, but to me this does not make that much sense really. ## Snyk Helps Secure the Golang Bento Project DevFeed: [Snyk Helps Secure the Golang Bento Project]() Original publisher: [Read original article]() Author: Phill Garrett Published: 2025-03-12T04:00:00Z Content type: article Language: en Sources: [Blog RSS Feed | Snyk]() Topics: [Vulnerabilities](), [open-source-security](), [streaming-data-processing](), [Kafka](), [CI/CD]() Tags: [blog](), [ci-cd](), [container-security](), [developer](), [engineering](), [go](), [golang](), [interest](), [kafka](), [maintainers](), [open-source](), [open-source-security](), [pull-request](), [releases](), [security](), [snyk-open-source](), [streaming-data-processing](), [vulnerability]() ### AI overview Snyk describes contributing fixes to the open-source Go project Bento after finding a denial-of-service vulnerability in its SSH dependency. The post also outlines Bento's streaming-data role and Snyk's program for open-source maintainers. ### Source excerpt Discover how Snyk helps secure the open source Golang project Bento by contributing vulnerability fixes and leveraging AI-powered tools. Learn about our efforts to enhance Bento's security and support open source maintainers through the Snyk Secure Developer Program. ## Latest Chainguard Images: FIPS, Harbor stack, Apache, and more! DevFeed: [Latest Chainguard Images: FIPS, Harbor stack, Apache, and more!]() Original publisher: [Read original article]() Published: 2024-06-20T00:00:00Z Content type: article Language: en Sources: [Chainguard: Unchained]() Topics: [chainguard images](), [container images](), [Security](), [open-source-security](), [supply-chain-security](), [airflow](), [streaming-data-processing](), [DataOps]() Tags: [airflow](), [apache](), [apache-zookeeper](), [chainguard](), [chainguard-images](), [container-image](), [container-images](), [fips](), [harbor](), [jitsu](), [open-source](), [pipelines](), [security](), [software-supply-chain](), [stream-processing](), [vulnerabilities]() ### AI overview Chainguard's monthly roundup announces 57 new hardened, minimal container images, including expanded Apache, Jitsu, and Harbor stacks. More than half of the releases are FIPS compliant, bringing the total to nearly 300 FIPS-validated Images. The article highlights security, reduced CVE exposure, software supply chain protection, and images for Apache Airflow, Superset, Zookeeper, and Nifi. ### Source excerpt Upgrade your software security with the latest Chainguard Images, featuring FIPS, Harbor, Apache, and more. Streamline your development and enhance protection. ## When Change Data Capture Can Break Application Encapsulation DevFeed: [When Change Data Capture Can Break Application Encapsulation]() Original publisher: [Read original article]() Published: 2023-11-21T00:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Architecture & Design](), [Software Engineering](), [streaming-data-processing](), [Kafka](), [Database]() Tags: [apache-kafka](), [data](), [debezium](), [schema](), [stream-processing](), [streaming]() ### AI overview This article examines whether exposing database change event feeds through Change Data Capture (CDC) breaks application encapsulation. It explains that CDC can expose internal table models as event APIs and create schema and downstream-consumer risks, then discusses ways to address those risks. ### Source excerpt Table of Contents CDC--A Quick Primer Does CDC Break Encapsulation? Entering Data Contracts Implementation Approaches For Data Contracts The Outbox Pattern Stream Processing Streaming Data Contracts--Beyond the Basics Handling Schema Changes Summary This post originally appeared on the Decodable blog. All rights reserved. Having worked on Debezium--an open-source platform for Change Data Capture (CDC)--for several years, one concern I've heard repeatedly is this: aren't you breaking the encapsulation of your application when you expose change event feeds directly from your database? After all, CDC exposes your internal persistent data model to the outside world, which may have unintended consequences, e.g. in terms of data exposure but also when it comes to changes to the schema of your data, which may break downstream consumers. ## CDC Use Cases: 7 Ways to Put CDC to Work DevFeed: [CDC Use Cases: 7 Ways to Put CDC to Work]() Original publisher: [Read original article]() Published: 2023-11-02T00:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [data-engineering](), [streaming-data-processing](), [Kafka](), [apache-flink](), [MySQL](), [Microservice](), [migration]() Tags: [apache-flink](), [data](), [data-engineering](), [debezium](), [engineering](), [kafka](), [migration](), [mysql](), [real-time]() ### AI overview This article explains change data capture (CDC), focusing on log-based CDC and its low-latency, resource-efficient handling of database changes. It introduces Debezium and describes how CDC works with data streaming and stream-processing tools such as Apache Kafka and Apache Flink, before outlining seven common use cases. ### Source excerpt Table of Contents What is CDC? CDC Tools Analytics Data Platforms Application Caches Full-Text Search Audit Logs Continuous Queries Microservices Data Exchange Monolith-to-Microservices Migration Summary This post originally appeared on the Decodable blog. All rights reserved. Change Data Capture (CDC) is a powerful tool in data engineering and has seen a tremendous uptake in organizations of all kinds over the last few years. This is because it enables the tight integration of transactional databases into many other systems in your business at a very low latency. ## Tinybird: A ksqlDB alternative for stateful stream processing DevFeed: [Tinybird: A ksqlDB alternative for stateful stream processing]() Original publisher: [Read original article]() Author: Alasdair Brown Published: 2023-07-27T00:00:00Z Content type: comparison Language: en Sources: [Tinybird]() Topics: [streaming-data-processing](), [stream-processing](), [Kafka]() Tags: [compare](), [kafka](), [scalable-analytics-architecture](), [stateful](), [stream-processing](), [streaming]() ### AI overview This comparison presents Tinybird as an alternative to ksqlDB for stateful stream processing and streaming SQL, emphasizing an approach intended to avoid the complexity associated with Kafka Streams. ### Source excerpt Looking for a ksqlDB alternative? Tinybird gives you streaming SQL without the Kafka Streams complexity. Compare the approaches. ## Why I Joined Decodable DevFeed: [Why I Joined Decodable]() Original publisher: [Read original article]() Published: 2022-11-03T14:00:00Z Content type: opinion Language: en Sources: [Gunnar Morling]() Topics: [stream-processing](), [streaming-data-processing](), [real-time](), [Serverless](), [Kafka](), [datasets](), [Apache Pulsar](), [amazon]() Tags: [amazon](), [apache](), [apache-kafka](), [kafka](), [latency](), [low-latency](), [news](), [platform](), [real-time](), [serverless](), [stream-processing]() ### AI overview A software engineer explains why he joined Decodable after several years at Red Hat, citing the field of real-time stream processing, the startup environment, and the team. He describes his experience with Debezium and his interest in helping users implement end-to-end streaming use cases. ### Source excerpt Table of Contents The Space: Real-Time Stream Processing The Environment: A Start-up The Team: One of a Kind Outlook It's my first week as a software engineer at Decodable, a start-up building a serverless real-time data platform! When I shared this news on social media yesterday, folks were not only super supportive and excited for me (thank you so much for all the nice words and wishes!), but some also asked about the reasons behind my decision for switching jobs and going to a start-up, after having worked for Red Hat for the last few years. ## Scaling Shopify's BFCM Live Map: An Apache Flink Redesign DevFeed: [Scaling Shopify's BFCM Live Map: An Apache Flink Redesign]() Original publisher: [Read original article]() Author: Berkay Antmen Published: 2021-12-10T19:00:00Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [Shopify](), [apache-flink](), [Data Infrastructure](), [Scalability](), [streaming-data-processing]() Tags: [apache-flink](), [data-infrastructure](), [scalability](), [shopify](), [stateful](), [stream-processing](), [streaming]() ### AI overview Shopify Data Platform Engineering redesigned the infrastructure behind the BFCM live map with Apache Flink to support more than 1.7 million merchants, richer insights, higher data volume, and higher uptime without manual intervention. ### Source excerpt A deep dive into how Shopify Data revamped the data infrastructure powering our BFCM live map using Apache Flink. ## Emerging Kafka Trends in Connectors, Self-Service Environments, and Stream Processing DevFeed: [Emerging Kafka Trends in Connectors, Self-Service Environments, and Stream Processing]() Original publisher: [Read original article]() Published: 2021-05-28T08:30:00Z Content type: opinion Language: en Sources: [Gunnar Morling]() Topics: [Kafka](), [streaming-data-processing](), [DataOps](), [Open Source]() Tags: [kafka](), [open-source](), [pipelines](), [review](), [stream-processing]() ### AI overview The article identifies recurring themes in Kafka community activity: the expanding connector ecosystem and its operational maturity, self-service Kafka environments, and growing adoption of stream processing. It also mentions related concerns including access control, schema management, data lineage, quality, compliance, privacy, and observability. ### Source excerpt Table of Contents Cambrian Explosion of Connectors Democratization of Data Pipelines Stream Processing for Everyone Honorable Mentions Over the course of the last few months, I've had the pleasure to serve on the Kafka Summit program committee and review several hundred session abstracts for the three Summits happening this year (Europe, APAC, Americas). That's not only a big honour, but also a unique opportunity to learn what excites people currently in the Kafka eco-system (and yes, it's a fair amount of work, too ;). While voting on the proposals, and also generally aspiring to stay informed of what's going on in the Kafka community at large, I noticed a few repeating themes and topics which I thought would be interesting to share (without touching on any specific talks of course). At first I meant to put this out via a Twitter thread, but then it became a bit too long for that, so I decided to write this quick blog post instead. Here it goes! ## From pipeline to beyond DevFeed: [From pipeline to beyond]() Original publisher: [Read original article]() Author: GameChanger Published: 2021-05-05T09:00:18Z Content type: article Language: en Sources: [GameChanger]() Topics: [Kafka](), [Amazon S3](), [data](), [streaming-data-processing](), [Back end]() Tags: [analysis](), [backend](), [consul](), [data](), [infrastructure](), [kafka](), [pipeline](), [redshift](), [s3](), [terraform](), [warehouse]() ### AI overview The article examines options for moving data out of Kafka into a data warehouse and archive. It discusses off-the-shelf tools such as Kafka Connect, Secor, and Gobblin, the limitations encountered, and the development of a custom solution. The requirements include preserving Avro data and schemas, writing to S3, and partitioning data by processing time or event time. ### Source excerpt An overview of off-the-shelf solutions for moving data out of Kafka, problems we had making those systems work, and how we wrote our own solution and stood it up for those in a similar situation. You cannot know everything a system will be used for when you start: it is only at the end of its life you can have such certainty. [1] Many moons ago I wrote about our design for upgrading our data pipeline, which lightly touched on how we'd move data out of our pipeline (Kafka) to downstream systems, namely our data warehouse and data archive. At that time we hadn't really been able to dive into focusing on getting the data out of Kafka, because getting data in to Kafka is often much more custom and complex, and we thought we'd be able to use an off the shelf solution like Kafka Connect to move data out, don't even worry about it. We were, uh -- we were wrong. Let me take you on our journey, in case you're on this journey too. The problem space Solution 1: Kafka Connect Solution 2: Secor or Gobblin Solution 3: we'll do this ourselves Tangent: naming things How you can do this yourselves, code edition How you can do this yourselves, infrastructure and metrics edition Takeaways The problem space Programmers are not to be measured by their ingenuity and their logic but by the completeness of their case analysis. [4] At a high level, the problem we needed a solution for was as follows: Data enters the data pipeline from numerous backend systems. This crossover point is producers into the pipeline, which we'd already implemented. Data from the data pipeline needs to move into the data warehouse and the data archive. This crossover point would be a consumer on the pipeline. Ideally we'd like the same consumer for both needs that we can simply configure differently. We want to preserve our data's Avro format along side its schemas. This would allow every system that interacts with the data to use the same language. We want to write our data to S3. data warehouse: This will be our ## Apache Beam for Search: Getting Started by Hacking Time DevFeed: [Apache Beam for Search: Getting Started by Hacking Time]() Original publisher: [Read original article]() Author: Doug Turnbull Published: 2021-01-08T15:00:01Z Content type: tutorial Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [streaming-data-processing](), [data-processing](), [AI Bots]() Tags: [apache](), [apis](), [batch](), [data-processing](), [event](), [events](), [real-time](), [search](), [shopify](), [spark](), [statistics](), [storage](), [stream-processing](), [streaming-data-processing](), [workflows]() ### AI overview An introduction to using Apache Beam for search-related streaming data processing. It explains how unified batch and streaming workflows can process clickstream data for real-time relevance tuning, and introduces event time, delayed events, and out-of-order data as core challenges. ### Source excerpt To create relevant search, processing clickstream data is key: you frequently want to promote search results that are being clicked on and purchased, and demote those things users don't love. Typically search systems think of processing clickstream data as a batch job run over historical data, perhaps using a system like Spark. ## Bullet Updates - Windowing, Apache Pulsar PubSub, Configuration-based Data Ingestion, and More DevFeed: [Bullet Updates - Windowing, Apache Pulsar PubSub, Configuration-based Data Ingestion, and More]() Original publisher: [Read original article]() Author: rosaliebeevm-blog Published: 2019-03-08T17:12:50Z Content type: release Language: en Sources: [Yahoo]() Topics: [streaming-data-processing](), [stream-processing](), [Query (disambiguation)](), [Streaming](), [Apache Spark]() Tags: [announcements](), [big-data](), [code](), [java](), [javascript](), [open-source](), [releases](), [spark](), [stream](), [stream-processing](), [streaming](), [updates](), [yahoo-engineering]() ### AI overview This update describes new windowing support in Bullet, an open-source query system for data flowing through streaming systems. It explains time- and record-based windows, including tumbling, sliding, and hopping window patterns, for returning intermediate query results. ### Source excerpt yahoodevelopers: By Akshay Sarma, Principal Engineer, Verizon Media & Brian Xiao, Software Engineer, Verizon Media This is the first of an ongoing series of blog posts sharing releases and announcements for Bullet, an open-sourced lightweight, scalable, pluggable, multi-tenant query system. Bullet allows you to query any data flowing through a streaming system without having to store it first through its UI or API. The queries are injected into the running system and have minimal overhead. Running hundreds of queries generally fit into the overhead of just reading the streaming data. Bullet requires running an instance of its backend on your data. This backend runs on common stream processing frameworks (Storm and Spark Streaming currently supported). The data on which Bullet sits determines what it is used for. For example, our team runs an instance of Bullet on user engagement data (~1M events/sec) to let developers find their own events to validate their code that produces this data. We also use this instance to interactively explore data, throw up quick dashboards to monitor live releases, count unique users, debug issues, and more. Since open sourcing Bullet in 2017, we've been hard at work adding many new features! We'll highlight some of these here and continue sharing update posts for future releases. Windowing Bullet used to operate in a request-response fashion - you would submit a query and wait for the query to meet its termination conditions (usually duration) before receiving results. For short-lived queries, say, a few seconds, this was fine. But as we started fielding more interactive and iterative queries, waiting even a minute for results became too cumbersome. Enter windowing! Bullet now supports time and record-based windowing. With time windowing, you can break up your query into chunks of time over its duration and retrieve results for each chunk. For example, you can calculate the average of a field, and stream back results every second: In th ## One-box stream processing with CSP DevFeed: [One-box stream processing with CSP]() Original publisher: [Read original article]() Published: 2018-02-18T00:00:00Z Content type: tutorial Language: en Sources: [Adam Bard]() Topics: [streaming-data-processing](), [Clojure](), [stream-processing](), [Concurrent Programming](), [distributed-systems]() Tags: [clojure](), [concurrent](), [core-async](), [coroutine](), [distributed-computing](), [distributed-system](), [flink](), [spark](), [stream](), [stream-processing]() ### AI overview This article presents a small-scale stream-processing architecture in Clojure using core.async and component libraries. It describes modular components connected by asynchronous queues and explains how stream-processing design principles can support reusable, loosely coupled systems without requiring a distributed stream processor. ### Source excerpt If you're like me (that is, employed by an ad tech company), stream processing is usually associated with frameworks like Storm, Flink, Spark Streaming, and other such solutions. However, a lot of real-life software can be described as stream processing - data comes in one end, is transformed or aggregated, and goes somewhere else. Many of these workloads don't justify the overhead of a stream processor, but that doesn't mean they can't benefit from some of the lessons of stream processing systems. ## Real-time Big Data at Target DevFeed: [Real-time Big Data at Target]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2015-11-11T06:00:00Z Content type: article Language: en Sources: [Target]() Topics: [big-data](), [Hadoop](), [stream-processing](), [streaming-data-processing](), [Kafka](), [Latency](), [Low Latency](), [Open Source]() Tags: [analytics](), [apache-kafka](), [batch](), [big-data](), [data](), [hadoop](), [kafka](), [latency](), [low-latency](), [open-source](), [real-time](), [storm](), [stream-processing](), [streaming]() ### AI overview Target's Big Data platform team describes its effort to move data into Hadoop in real time using flexible open source components. The article outlines requirements for resilience, message-format support, usability, and low latency, then evaluates Apache Storm and identifies recovery and batching-related latency issues in testing. ### Source excerpt An enterprise as large as Target generates a lot of data and on my Big Data platform team we want to make it as easy as possible for our users to get it into Hadoop in real-time. I want to discuss how we are starting to approach this problem, what we've done so far, and what is still to come. Our requirements We wanted to build a system with flexible open source components. Our experience with proprietary products on Hadoop is that they tend to be inflexible and only work with a narrow set of use cases. That can also be true of open source products, but we have found them to be easier to adapt to our needs. Indeed, that ended up being the case as we went through this journey and made several contributions to Apache projects. More specifically, we wanted to create a system that: was highly resilient to the failure of any individual component supported a variety of message formats delivered data to Hadoop with very low latency made the data immedaiately usable streamed data from Apache Kafka sources A Streaming Framework There are many excellent comparisons of streaming frameworks available, and I won't attempt to recreate them here. The first criteria we considered was what tooling was needed to monitor and administer the streaming framework, with a strong preference to use our primary Hadoop administration tool, Apache Ambari. Apache Storm fit that bill and was also a proven solution for stream processing. If Storm could meet our other requirements, it would be our first choice. To test its resiliency we ran a simple scenario: start a data stream into Hadoop, disable HDFS, and then reenable it. Streaming would obviously fail while HDFS was disabled, but we needed the system to recover gracefully when HDFS came back online. Unfortunately our first test of this scenario left our Storm topology in an unrecoverable state, which required a manual restart. That's not something we could live with. We also needed very fine control over the latency of arriving data. In gener