# stream-processing Published articles for stream-processing. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Why Kafka retention.ms can delay message deletion DevFeed: [Why Kafka retention.ms can delay message deletion]() Original publisher: [Read original article]() Author: Rogerio Santos Published: 2026-09-16T13:05:06Z Content type: tutorial Language: en Sources: [Red Hat](), [Red Hat Developer]() Topics: [Kafka](), [Apache-Kafka](), [configuration](), [file](), [Event-Streaming]() Tags: [broker](), [configuration](), [diagnose](), [event-driven](), [how-to](), [kafka](), [lifecycle](), [retention](), [storage](), [stream-processing]() ### AI overview This guide explains why Kafka messages can remain readable beyond a topic's retention.ms setting. Kafka deletes closed log segments rather than individual messages, and the active segment and continuous writes can delay removal of older records. ### Source excerpt A customer opened a support case with a deceptively simple complaint: a Kafka topic was configured with a 12-hour retention (retention.ms), yet messages produced on July 24 were still readable 4 days later, on July 28. Nothing was broken. The broker logged no errors. The retention policy was working as designed, but the segment layout and continuous write pattern delayed when the old records could actually be removed. The post Why your Kafka topic ignores retention.ms (and how to fix it) appeared first on Red Hat Developer. ## Redpanda named a Leader in G2's Fall 2026 Event Stream Processing Reports DevFeed: [Redpanda named a Leader in G2's Fall 2026 Event Stream Processing Reports]() Original publisher: [Read original article]() Author: Redpanda Published: 2026-09-01T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [stream-processing](), [streaming-data-processing](), [Usability](), [real-time](), [Streaming](), [Software]() Tags: [2026](), [announcement](), [event](), [performance](), [real-time](), [reports](), [software](), [stream-processing](), [streaming](), [usability]() ### AI overview Redpanda reports that it was recognized in 10 G2 Fall 2026 Event Stream Processing reports, earning first place in eight indexes, including implementation, usability, results, and relationship. The article also highlights Redpanda Shadowing for migration from Confluent and cites a 4.7 out of 5 average rating across 53 G2 reviews. ### Source excerpt Redpanda ranked #1 in Implementation, Usability, Results, and Relationship Indexes in G2's Fall 2026 Event Stream Processing Reports. ## A Tale of Two Flink Autoscalers DevFeed: [A Tale of Two Flink Autoscalers]() Original publisher: [Read original article]() Author: Netflix Technology Blog Published: 2026-08-21T16:01:01Z Content type: article Language: en Sources: [Netflix](), [Netflix TechBlog - Medium]() Topics: [apache-flink](), [autoscaling](), [Netflix](), [stream-processing](), [Kafka](), [Open Source](), [telemetry](), [Network]() Tags: [apache-flink](), [autoscaling](), [aws](), [infrastructure](), [kafka](), [metrics](), [netflix](), [open-source](), [operational-excellence](), [scale](), [stream-processing](), [telemetry]() ### AI overview Netflix describes running two Apache Flink autoscalers: an older in-house system and a newer one from the Apache Flink community. The article explains why autoscaling is necessary for more than 30,000 Flink jobs across AWS regions, discusses workload variability and the cost of resizing stateful jobs, and introduces the metrics-driven design of the first autoscaler. ### Source excerpt Samuel Yeboah, Francesco Di Chiara and Mingliang Liu Today, Netflix runs two Flink autoscalers. That is exactly one more than we want. We built the first one in-house years ago, when there was no mature option suited to our platform. The second came from the Apache Flink community, and it can scale workloads our homegrown system was never designed for. We now run both in production and are steadily converging on the open-source one. Along the way we learned some hard lessons about metrics, cost, and the real price of maintaining infrastructure you could instead adopt, and we hope they are useful whether you run a handful of Flink jobs or tens of thousands. Why autoscaling is not optional at our scale Netflix has run stream processing on Apache Flink since 2017. As of 2026 we operate more than 30,000 Flink jobs across multiple AWS regions. Most are not deployed by hand; they are generated by our managed platform Data Mesh, so the majority of users never touch a Flink job directly. A smaller but growing set are custom jobs, built and operated by teams across the company for use cases like personalization, Ads, and Live events. They range from single-operator jobs that shuttle records between Kafka topics to stateful pipelines with branches, joins, and terabytes of state, and their load swings with daily cycles, launches, and regional failovers. Provisioning every one of those jobs for its peak is wasteful; provisioning for the average causes lag during surges. And in our platform a scaling action is not free: by default it means taking a savepoint, stopping the job gracefully, and restarting it at the new size, which for a large stateful job can take minutes. That leaves a genuinely hard question: how do you give each job the resources it needs, when it needs them, without a human in the loop and without breaking anything? The first autoscaler: watching from outside Our first answer, built around 2019, was an autoscaler shaped like a stream-processing job. It ran on M ## New in Confluent Cloud and WarpStream: Evolving the Data Streaming Platform for AI, Scale, and Control DevFeed: [New in Confluent Cloud and WarpStream: Evolving the Data Streaming Platform for AI, Scale, and Control]() Original publisher: [Read original article]() Author: Mike Agnich Published: 2026-08-18T14:00:11Z Content type: article Language: en Sources: [Confluent: Data in motion]() Topics: [Streaming](), [Kafka](), [Cloud](), [event driven](), [Transactions]() Tags: [confluent](), [confluent-cloud](), [debug](), [kafka](), [networking](), [security](), [stream-processing](), [streaming](), [transactions]() ### AI overview Confluent describes more than 70 new features for its data streaming platform, including new connectors, networking improvements, AI capabilities for event-driven agents, real-time machine learning in streaming pipelines, and AI-assisted development tools. The article also discusses broader Kafka workload support, including stream processing, transactions, exactly-once processing, queues, debugging, and Python client support for Queues for Kafka. ### Source excerpt Accelerate enterprise streaming and AI workloads with new product updates across Kora, connectors, Flink, Tableflow, AI anomaly detection and forecasting, security enhancements, and Warpstream. ## Multi-region high availability for Kafka workloads with a single Stretch Cluster DevFeed: [Multi-region high availability for Kafka workloads with a single Stretch Cluster]() Original publisher: [Read original article]() Author: David Yu Published: 2026-08-11T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Kafka](), [Kubernetes](), [Availability](), [Replication](), [Raft](), [Resilience](), [stream-processing]() Tags: [clusters](), [k8s](), [kafka](), [raft](), [release](), [replication](), [resilience](), [stream-processing]() ### AI overview Redpanda Operator 26.2 introduces generally available Stretch Clusters, allowing one logical Redpanda cluster to span multiple Kubernetes clusters and regions. The release provides synchronous replication with Raft-based automatic failover, safer broker rolling restarts, Redpanda Connect pipelines as Kubernetes resources, and Gateway API support for Redpanda Console. ### Source excerpt Redpanda Operator 26.2 brings GA Stretch Clusters for multi-region replication, Redpanda Connect pipelines as K8s resources, Gateway API support, and safer rolling restarts. ## Agentic kill switch is a database problem. So we built Redpanda SQL DevFeed: [Agentic kill switch is a database problem. So we built Redpanda SQL]() Original publisher: [Read original article]() Author: Alexander Gallego Published: 2026-08-03T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Databases](), [SQL](), [stream-processing](), [Google Cloud Platform (GCP)](), [Amazon Web Services](), [systems]() Tags: [agentic](), [announcement](), [aws](), [cloud](), [code](), [database](), [exploit](), [google-cloud](), [governance](), [models](), [olap](), [real-time](), [scale](), [sql](), [stream-processing](), [systems](), [tools]() ### AI overview The article presents Redpanda SQL as a database built for agentic governance. It describes real-time bridge queries, efficient SQL JOINs, a ring-buffer shuffle algorithm, and analytical processing designed to detect agent behavior and provide an agentic kill switch. Redpanda SQL is stated to be available on AWS and Google Cloud. ### Source excerpt Agentic governance needs a new kind of database, so we built Redpanda SQL. Now available on both AWS and Google Cloud. ## Drag, drop, done: a visual composer for Redpanda Connect DevFeed: [Drag, drop, done: a visual composer for Redpanda Connect]() Original publisher: [Read original article]() Author: Prakhar Garg Published: 2026-07-23T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Redpanda-Connect](), [Canvas](), [stream-processing](), [Caching](), [Routing (disambiguation)](), [HTTP](), [n8n](), [workflow automation]() Tags: [api](), [automation](), [building](), [cache](), [caching](), [events](), [front-end](), [http](), [n8n](), [pipeline](), [redpanda-connect](), [routing](), [stream-processing](), [workflow]() ### AI overview Redpanda introduces a visual Pipeline Builder for Redpanda Connect, using a drag-and-drop canvas to compose pipelines from inputs, processors, and outputs. The builder generates YAML while making routing, enrichment, caching, error handling, and retries easier to understand and configure. The article contrasts it with n8n, emphasizing Redpanda Connect's suitability for high-volume event streaming with ordering, backpressure, and exactly-once semantics. ### Source excerpt A first look at the new visual Pipeline Builder for Redpanda Connect (preview), plus a roundup of recent CDC and connector updates. ## Michael Drogalis on Building ShadowTraffic as a Solo B2B Software Company DevFeed: [Michael Drogalis on Building ShadowTraffic as a Solo B2B Software Company]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-15T13:24:12Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [Software Engineering](), [Kafka](), [stream-processing]() Tags: [kafka](), [software-engineering](), [stream-processing]() ### AI overview A conversation with Michael Drogalis about building ShadowTraffic as a solo B2B software company. It discusses realistic production-traffic generation for testing, demos, and development, along with customer discovery, marketing, AI use, and engineering careers. ### Source excerpt A Conversation with Michael Drogalis ## Event-Driven Architecture with Apache Kafka DevFeed: [Event-Driven Architecture with Apache Kafka]() Original publisher: [Read original article]() Author: hello@ankit-rana.com Published: 2026-03-16T00:00:00Z Content type: tutorial Language: en Sources: [Ankit Rana | Mechanical Sympathy]() Topics: [event driven](), [Kafka](), [Apache-Kafka](), [Scalability](), [Resilience](), [streaming-data-processing](), [Latency](), [systems]() Tags: [apache-kafka](), [architecture](), [asynchronous](), [consumer](), [distributed-systems](), [event-driven-architecture](), [event-sourcing](), [kafka](), [microservices](), [partitions](), [real-time](), [resilience](), [scalability](), [schemas](), [stream-processing]() ### AI overview This tutorial explains event-driven architecture and how Apache Kafka supports asynchronous, real-time data processing. It covers producers, consumers, immutable events, event sourcing, scalability, resilience, stream-processing pipelines, and challenges such as ordering, debugging, and eventual consistency. ### Source excerpt Event-driven architecture replaces synchronous point-to-point calls with immutable events on a durable log, so producers and consumers scale and fail independently. Kafka provides that log: topics sharded into ordered append-only partitions, replicated across brokers, with consumer groups sharing partitions to scale read throughput. ## What's new in Redpanda Cloud: A simpler Serverless now on GCP DevFeed: [What's new in Redpanda Cloud: A simpler Serverless now on GCP]() Original publisher: [Read original article]() Author: Towfiqa Yasmeen Published: 2025-11-25T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Redpanda-Connect](), [Serverless](), [Google Cloud Platform (GCP)](), [Streaming](), [Kafka](), [User experience (UX)]() Tags: [announcement](), [gcp](), [kafka](), [redpanda-connect](), [serverless](), [stream-processing](), [streaming](), [usability](), [ux]() ### AI overview Redpanda Serverless introduces a guided onboarding experience, availability on Google Cloud Platform (GCP), and a simpler path to deploying Kafka-compatible streaming. The update targets developers and platform leaders with managed, usage-based streaming that removes broker, cluster, scaling, and infrastructure-management concerns. ### Source excerpt Redpanda Serverless gives you the shortest path from idea to impact with a new onboarding UX, availability on GCP, and more. Here's what's new. ## Redpanda 25.3 delivers near-instant disaster recovery and more DevFeed: [Redpanda 25.3 delivers near-instant disaster recovery and more]() Original publisher: [Read original article]() Author: Matt Schumpert Published: 2025-11-06T00:00:00Z Content type: release Language: en Sources: [Redpanda]() Topics: [Disaster Recovery](), [Resilience](), [Kafka](), [Streaming](), [Availability](), [configuration](), [Cloud](), [API](), [systems]() Tags: [cloud](), [clusters](), [configuration](), [consumer](), [disaster-recovery](), [high-availability](), [integration](), [kafka](), [process](), [production](), [real-time](), [recovery](), [release](), [resilience](), [server](), [stream-processing](), [streaming](), [systems](), [update]() ### AI overview The article previews Redpanda 25.3, focusing on Shadowing, a disaster recovery and business continuity feature for streaming workloads. Shadowing creates a byte-for-byte, offset-preserving hot standby of a production cluster in another region, including topics, configurations, consumer offsets, ACLs, and schemas. It is designed to provide recovery point and recovery time objectives measured in seconds, with setup through configuration or Redpanda Console and replication using the standard Kafka API. ### Source excerpt Check out how the latest Redpanda release is providing the resilience, efficiency, and interoperability that modern enterprises need for their streaming workloads. ## Introducing the Agentic Data Plane DevFeed: [Introducing the Agentic Data Plane]() Original publisher: [Read original article]() Author: Alexander Gallego Published: 2025-10-28T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Artificial Intelligence](), [Data Infrastructure](), [Streaming](), [Apache Iceberg](), [API](), [Security](), [observability](), [Low Latency]() Tags: [agentic](), [ai](), [ai-agents](), [connectivity](), [data-infrastructure](), [low-latency](), [observability](), [security](), [stream-processing]() ### AI overview Redpanda introduces the Agentic Data Plane, a unified runtime and control plane for safely exposing enterprise data to AI agents. It combines low-latency streaming for events and human-in-the-loop workflows, an Iceberg-native query engine for real-time context, more than 300 connectors, and managed policy and observability capabilities for governed access, intent recording, and replayable audits. ### Source excerpt Redpanda's Agentic Data Plane is a unified runtime and control plane that safely exposes enterprise data to AI agents. Here's how we're building it. ## Redpanda Cloud Summer 2025: Connect everywhere, Iceberg GA in cloud, and more DevFeed: [Redpanda Cloud Summer 2025: Connect everywhere, Iceberg GA in cloud, and more]() Original publisher: [Read original article]() Author: Mike Broberg Published: 2025-08-19T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Cloud](), [Streaming](), [Apache Iceberg](), [Serverless](), [Kafka](), [Cache](), [Google Cloud Platform (GCP)](), [HTTP](), [BigQuery](), [tbox](), [Slack]() Tags: [2026](), [announcement](), [apache-iceberg-integration](), [bigquery](), [byoc-on-azure](), [cloud](), [cloud-event-streaming-architecture](), [http](), [iceberg-ga-in-cloud](), [iceberg-topics-byoc](), [kafka](), [multi-cloud-solution-azure-aws-gcp](), [real-time-data-ingestion-lakehouse](), [redpanda-cloud-summer-2025](), [redpanda-connect](), [redpanda-connect-deployment](), [redpanda-serverless-features](), [serverless](), [serverless-capabilities-expansion](), [slack](), [stream-processing](), [streaming-pipelines-scalability](), [vertex-ai]() ### AI overview Redpanda Cloud's June-July 2025 summer update introduces general availability for Redpanda Connect across deployment types and supported cloud providers, expands Serverless capabilities, and makes Iceberg Topics generally available for AWS and GCP BYOC clusters. It also adds a Spanner change data capture connector, a write-optimized cache, Slack reaction output, and an HTTP gateway input for triggering streaming pipelines. ### Source excerpt Our summer update is sizzling with expanded Serverless capabilities, Iceberg topics on across all major clouds, and Redpanda Connect on all deployments. Read on. ## Redpanda open-sources top 16 AI connectors DevFeed: [Redpanda open-sources top 16 AI connectors]() Original publisher: [Read original article]() Author: Mike Broberg Published: 2025-08-18T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Redpanda-Connect](), [Open Source](), [Artificial Intelligence](), [stream-processing](), [OpenAI](), [Streaming](), [Amazon Bedrock](), [Embeddings](), [Retrieval Augmented Generation (RAG)](), [cohere](), [Ollama]() Tags: [ai](), [ai-connector-platforms](), [ai-connectors-for-commercial-products](), [ai-connectors-open-source](), [ai-data-streaming](), [ai-integration-tools](), [announcement](), [bedrock](), [cloud-native-ai-services](), [cohere](), [embeddable-ai-capabilities](), [embeddings](), [ollama](), [open-source](), [open-source-ai-for-business](), [openai](), [openai-connectors](), [rag](), [real-time-ai-streaming](), [redpanda-ai-connectors](), [redpanda-connect](), [stream-processing](), [streaming](), [streaming-ai-processors]() ### AI overview Redpanda announces the open-source release of its top AI connectors under the Apache 2.0 license. The connectors integrate Redpanda Connect with destinations and models including OpenAI, Cohere, Amazon Bedrock, Ollama, and Google Cloud Vertex AI, supporting streaming pipelines and use cases such as generation, summarization, classification, translation, and text embeddings for RAG. ### Source excerpt Redpanda open-sources top AI connectors to the most used destinations, including OpenAI, Cohere, Bedrock, Ollama, and Vertex AI. Learn more. ## Introducing multi-language dynamic plugins for Redpanda Connect DevFeed: [Introducing multi-language dynamic plugins for Redpanda Connect]() Original publisher: [Read original article]() Author: James Kinley Published: 2025-06-17T00:00:00Z Content type: release Language: en Sources: [Redpanda]() Topics: [gRPC](), [SDKs](), [Go Language](), [Python](), [Processes](), [streaming-data-processing](), [Unix]() Tags: [ai-ml-capabilities-in-streaming-data](), [architecture](), [configuration](), [creating-plugins-in-golang-and-python](), [dynamic-vs-compiled-plugins](), [framework](), [go](), [grpc-plugin-system](), [integration](), [interfaces](), [ipc](), [language-agnostic-plugin-system](), [multi-language-plugin-development](), [pipelines](), [plugin-development](), [plugins](), [process](), [processes](), [product](), [protocol](), [python](), [python-sdk-for-streaming-data](), [redpanda-connect](), [redpanda-connect-dynamic-plugins](), [redpanda-streaming-infrastructure](), [runtime-loaded-plugins-grpc](), [sdk](), [sdks](), [stream-processing](), [streaming-data-pipeline-plugins]() ### AI overview Redpanda Connect introduces Apache 2.0-licensed dynamic plugins in Beta with version 4.56.0. Plugins can be loaded at runtime as external executables that communicate with the main process through gRPC over Unix sockets, enabling plugin development in Go and Python. Official SDKs support both languages, while native Go plugins remain the preferred option for performance-critical workloads. ### Source excerpt Redpanda Connect dynamic plugins framework allows you to create and load plugins at runtime, opening up a world of new integration possibilities beyond Go. ## Backfilling Postgres TOAST Columns in Debezium Data Change Events DevFeed: [Backfilling Postgres TOAST Columns in Debezium Data Change Events]() Original publisher: [Read original article]() Published: 2025-05-26T14:40:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Databases](), [Replication](), [apache-flink](), [stream-processing](), [SQL](), [API]() Tags: [apache-flink](), [data](), [debezium](), [debezium-connector](), [flink](), [postgres](), [replication](), [sql](), [stateful](), [stream-processing]() ### AI overview This article explains how Postgres logical replication handles TOAST columns in Debezium change events. It discusses Debezium's reselect postprocessor and explores Apache Flink solutions using stateful stream processing, Flink SQL, and process table functions. ### Source excerpt Table of Contents Debezium Reselect Postprocessor Flink DataStream API Flink SQL With OVER Aggregation Flink Process Table Functions Summary and Discussion Postgres logical replication, while powerful for capturing real-time data changes, presents challenges with TOAST columns, whose values can be absent from data change events in specific situations. This post discusses how Debezium addresses this through its built-in reselect post processor, then explores more robust solutions leveraging Apache Flink's capabilities for stateful stream processing, including Flink SQL and the brand-new process table functions (PTFs) in Flink 2.1. ## Streaming and Batch Processing Are Complementary; Pull Versus Push Is the Key Distinction DevFeed: [Streaming and Batch Processing Are Complementary; Pull Versus Push Is the Key Distinction]() Original publisher: [Read original article]() Published: 2025-05-14T08:10:00Z Content type: opinion Language: en Sources: [Gunnar Morling]() Topics: [Streaming](), [stream-processing](), [streaming-data-processing](), [real-time](), [systems](), [data]() Tags: [kafka](), [latency](), [low-latency](), [performance](), [real-time](), [stream-processing](), [streaming](), [systems]() ### AI overview The article argues that streaming systems commonly use batching to improve throughput, so streaming and batch processing are not opposites. It proposes pull versus push semantics as the more meaningful distinction and explains that push-based streaming can provide timely updates, while adding complexity around state, joins, and out-of-order data. ### Source excerpt Often times, "Stream vs. Batch" is discussed as if it's one or the other, but to me this does not make that much sense really. ## KIP-932 explores queue semantics and Share Groups for Apache Kafka DevFeed: [KIP-932 explores queue semantics and Share Groups for Apache Kafka]() Original publisher: [Read original article]() Published: 2025-03-05T11:35:00Z Content type: opinion Language: en Sources: [Gunnar Morling]() Topics: [Kafka](), [Streaming](), [stream-processing](), [Architecture & Design]() Tags: [apache-kafka](), [fraud](), [fraud-detection](), [job](), [kafka](), [microservices](), [stream-processing](), [streaming]() ### AI overview The article examines KIP-932, which explores adding queue semantics to Apache Kafka through Share Groups. It explains how Kafka's partition-based consumer model limits parallelism and ordered processing, and why individual message acknowledgment and rejection are important for queueing workloads such as independent job processing. ### Source excerpt Table of Contents Towards Queue Support in Kafka--Introducing Share Groups Share Groups in Action Retry Behavior and State Management Share Group State Persistence Summary and Outlook In the "Let's Take a Look at...!" blog series I am going to explore interesting projects, developments and technologies in the data and streaming space. This can be KIPs and FLIPs, open-source projects, services, and more. The idea is to get some hands-on experience, learn about potential use cases and applications, and understand the trade-offs involved. If you think there's a specific subject I should take a look at, let me know in the comments below! That guy above? Yep, that's me, whenever someone says "Kafka queue". Because, that's not what Apache Kafka is. At its core, Kafka is a distributed durable event log. Producers write events to a topic, organized in partitions which are distributed amongst the brokers of a Kafka cluster. Consumers, organized in groups, divide the partitions they process amongst themselves, so that each partition of a topic is read by exactly one consumer in the group. ## Kafka for System Design Interviews DevFeed: [Kafka for System Design Interviews]() Original publisher: [Read original article]() Author: Evan King Published: 2024-07-01T03:18:09Z Content type: tutorial Language: en Sources: [Evan King]() Topics: [Kafka](), [Apache-Kafka](), [Event-Streaming](), [stream-processing](), [Scalability](), [data]() Tags: [apache-kafka](), [distributed](), [durability](), [google](), [high-performance](), [kafka](), [meta](), [open-source](), [real-time](), [scalability](), [stream-processing](), [system-design-interview]() ### AI overview A tutorial on Apache Kafka for system design interviews. It explains Kafka as an open-source distributed event streaming platform that can function as a message queue or stream processing system, then uses a real-time sports statistics example to discuss producers, consumers, scaling, and event ordering. ### Source excerpt By the Co-founder of www.hellointerview.comIntro There is a good chance you've heard of Kafka. It's popular. In fact, according to their website, it's used by 80% of the Fortune 100. If it's good enough to help scale the largest companies in the world, it's probably good enough for your next system design interview. In fact, it's one of the top 5 technologies we see used in design interviews. Apache Kafka is an open-source distributed event streaming platform that can be used either as a message queue or as a stream processing system. Kafka excels in delivering high performance, scalability, and durability. It's engineered to handle vast volumes of data in real-time, ensuring that no message is ever lost and that each piece of data is processed as swiftly as possible. In this deep dive, we're going to take a top down approach. Starting with a zoomed out view of Kafka and progressing into more and more detail. If you know the basics, feel free to skip ahead to the more advanced sections. A Motivating Example It's the World Cup (my personal favorite competition). And we run a website that provides real-time statistics on the matches. Each time a goal is scored, a player is booked, or a substitution is made, we want to update our website with the latest information. Events are placed on a queue when they occur. We call the server or process responsible for putting these events on the queue the producer. Downstream, we have a server that reads events off the queue and updates the website. We call this the consumer. Now, imagine the World Cup expanded from just the top 48 teams to a hypothetical 1,000-team tournament, and all the games are now played at the same time. The number of events has increased significantly, and our single server hosting the queue is struggling to keep up. Similarly, our consumer feels like it has its mouth under a firehose and is crashing under the load. We need to scale the system by adding more servers to distribute our queue. But how do we e ## Latest Chainguard Images: FIPS, Harbor stack, Apache, and more! DevFeed: [Latest Chainguard Images: FIPS, Harbor stack, Apache, and more!]() Original publisher: [Read original article]() Published: 2024-06-20T00:00:00Z Content type: article Language: en Sources: [Chainguard: Unchained]() Topics: [chainguard images](), [container images](), [Security](), [open-source-security](), [supply-chain-security](), [airflow](), [streaming-data-processing](), [DataOps]() Tags: [airflow](), [apache](), [apache-zookeeper](), [chainguard](), [chainguard-images](), [container-image](), [container-images](), [fips](), [harbor](), [jitsu](), [open-source](), [pipelines](), [security](), [software-supply-chain](), [stream-processing](), [vulnerabilities]() ### AI overview Chainguard's monthly roundup announces 57 new hardened, minimal container images, including expanded Apache, Jitsu, and Harbor stacks. More than half of the releases are FIPS compliant, bringing the total to nearly 300 FIPS-validated Images. The article highlights security, reduced CVE exposure, software supply chain protection, and images for Apache Airflow, Superset, Zookeeper, and Nifi. ### Source excerpt Upgrade your software security with the latest Chainguard Images, featuring FIPS, Harbor, Apache, and more. Streamline your development and enhance protection. ## Evolving the Backend Storage for Platform Metrics DevFeed: [Evolving the Backend Storage for Platform Metrics]() Original publisher: [Read original article]() Author: David Murray Published: 2024-05-09T22:00:00Z Content type: article Language: en Sources: [Heroku]() Topics: [Heroku](), [Back end](), [Time Series](), [Apache Cassandra](), [Kafka](), [stream-processing](), [autoscaling]() Tags: [apache-kafka](), [app-architecture](), [autoscaling](), [aws](), [backend](), [cassandra](), [cloud-infrastructure](), [data-analytics](), [database](), [developer-tools](), [engineering](), [heroku](), [kafka](), [metrics](), [performance-optimization](), [platform](), [platform-updates](), [stream-processing](), [time-series]() ### AI overview Heroku describes an infrastructure upgrade that migrated the storage backend for platform metrics. The article explains how MetaaS collects application observations, aggregates them into time-series statistics, and uses Kafka, stream-processing jobs, and Cassandra to support dashboard metrics, alerting, autoscaling, retention, and querying. ### Source excerpt One of our most important goals at Heroku is to be boring. Don't get us wrong, we certainly hope that you're excited about the Heroku developer experience -- as heavy users of Heroku ourselves, we certainly are! But, even more so, we hope that you don't have to spend all that much time thinking about [...] The post Evolving the Backend Storage for Platform Metrics appeared first on Heroku. ## Getting Started With PyFlink on Kubernetes DevFeed: [Getting Started With PyFlink on Kubernetes]() Original publisher: [Read original article]() Published: 2023-12-07T00:00:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [apache-flink](), [Kubernetes](), [stream-processing](), [Kafka](), [Python](), [data-engineering]() Tags: [apache-kafka](), [flink](), [kafka](), [kubernetes](), [operator](), [programming](), [python](), [stream-processing]() ### AI overview A practical guide to running PyFlink on Kubernetes with Apache Flink 1.18. It covers installing the Flink Kubernetes Operator and Kafka with Strimzi, creating a PyFlink job that generates data and writes it to Kafka using Flink SQL, building a container image, and deploying the job to Kubernetes. ### Source excerpt Table of Contents What Is PyFlink and Why Should You Care? Prerequisites Installing the Flink Kubernetes Operator Installing Strimzi and Apache Kafka A Simple PyFlink Job Building a Container Image With Your PyFlink Job Deploying a PyFlink Job On Kubernetes This post originally appeared on the Decodable blog. All rights reserved. The other day, I wanted to get my feet wet with PyFlink. While there is a fair amount of related information out there, I couldn't find really up-to-date documentation on using current versions of PyFlink with Flink on Kubernetes. ## When Change Data Capture Can Break Application Encapsulation DevFeed: [When Change Data Capture Can Break Application Encapsulation]() Original publisher: [Read original article]() Published: 2023-11-21T00:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Architecture & Design](), [Software Engineering](), [streaming-data-processing](), [Kafka](), [Database]() Tags: [apache-kafka](), [data](), [debezium](), [schema](), [stream-processing](), [streaming]() ### AI overview This article examines whether exposing database change event feeds through Change Data Capture (CDC) breaks application encapsulation. It explains that CDC can expose internal table models as event APIs and create schema and downstream-consumer risks, then discusses ways to address those risks. ### Source excerpt Table of Contents CDC--A Quick Primer Does CDC Break Encapsulation? Entering Data Contracts Implementation Approaches For Data Contracts The Outbox Pattern Stream Processing Streaming Data Contracts--Beyond the Basics Handling Schema Changes Summary This post originally appeared on the Decodable blog. All rights reserved. Having worked on Debezium--an open-source platform for Change Data Capture (CDC)--for several years, one concern I've heard repeatedly is this: aren't you breaking the encapsulation of your application when you expose change event feeds directly from your database? After all, CDC exposes your internal persistent data model to the outside world, which may have unintended consequences, e.g. in terms of data exposure but also when it comes to changes to the schema of your data, which may break downstream consumers. ## Tinybird: A ksqlDB alternative for stateful stream processing DevFeed: [Tinybird: A ksqlDB alternative for stateful stream processing]() Original publisher: [Read original article]() Author: Alasdair Brown Published: 2023-07-27T00:00:00Z Content type: comparison Language: en Sources: [Tinybird]() Topics: [streaming-data-processing](), [stream-processing](), [Kafka]() Tags: [compare](), [kafka](), [scalable-analytics-architecture](), [stateful](), [stream-processing](), [streaming]() ### AI overview This comparison presents Tinybird as an alternative to ksqlDB for stateful stream processing and streaming SQL, emphasizing an approach intended to avoid the complexity associated with Kafka Streams. ### Source excerpt Looking for a ksqlDB alternative? Tinybird gives you streaming SQL without the Kafka Streams complexity. Compare the approaches. [Next page]()