# Kafka Published articles for Kafka. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Why Kafka retention.ms can delay message deletion DevFeed: [Why Kafka retention.ms can delay message deletion]() Original publisher: [Read original article]() Author: Rogerio Santos Published: 2026-09-16T13:05:06Z Content type: tutorial Language: en Sources: [Red Hat](), [Red Hat Developer]() Topics: [Kafka](), [Apache-Kafka](), [configuration](), [file](), [Event-Streaming]() Tags: [broker](), [configuration](), [diagnose](), [event-driven](), [how-to](), [kafka](), [lifecycle](), [retention](), [storage](), [stream-processing]() ### AI overview This guide explains why Kafka messages can remain readable beyond a topic's retention.ms setting. Kafka deletes closed log segments rather than individual messages, and the active segment and continuous writes can delay removal of older records. ### Source excerpt A customer opened a support case with a deceptively simple complaint: a Kafka topic was configured with a 12-hour retention (retention.ms), yet messages produced on July 24 were still readable 4 days later, on July 28. Nothing was broken. The broker logged no errors. The retention policy was working as designed, but the segment layout and continuous write pattern delayed when the old records could actually be removed. The post Why your Kafka topic ignores retention.ms (and how to fix it) appeared first on Red Hat Developer. ## How We Built Automated Capacity Testing for Kafka Consumers DevFeed: [How We Built Automated Capacity Testing for Kafka Consumers]() Original publisher: [Read original article]() Author: Kaan Karakaya Published: 2026-09-14T09:46:34Z Content type: tutorial Language: en Sources: [Booking.com Development - Medium]() Topics: [Kafka](), [Testing](), [HTTP]() Tags: [automated](), [capacity](), [health-checks](), [kafka](), [load](), [load-balancer](), [parallelism](), [partition](), [recovery](), [scale](), [site-reliability-engineer](), [sre](), [testing]() ### AI overview This article describes Booking.com's extension of an existing capacity-testing platform for Kafka consumers. It explains how changing partition assignment can provide a controlled, measurable way to test consumer throughput and whether remaining consumers can absorb reassigned work after an instance or failure domain disappears. ### Source excerpt Photo by GuerrillaBuzz on Unsplash Kafka makes it easy to distribute work across consumer instances. It is much harder to prove, safely and repeatedly, how those instances behave when the distribution changes and one of them has to carry more than its usual share. For teams that run Kafka at scale, this is a practical reliability question: how much load can a consumer instance actually handle? We had automated capacity testing for HTTP services, but Kafka consumers were still tested with manual drills. Those drills could tell us something, but they were disruptive, difficult to reproduce, and risky precisely when the system was close to its limit. We wanted a controlled way to answer three questions: What is the maximum sustainable throughput of a consumer instance? If an instance or failure domain disappears, can the remaining consumers absorb the reassigned work? Are we overprovisioning resources because we do not know the real limit? The result was an extension to our capacity-testing platform that turns Kafka partition assignment into a safe, measurable load-control mechanism. Why HTTP capacity testing did not translate Our existing platform was designed for request-response services behind a load balancer. A scheduled test selects one instance, routes an increasing share of traffic to it, runs health checks after each step, and records the highest ratio the instance can sustain. After the test, traffic returns to its normal distribution and the result is reported to the service owner. Kafka has no equivalent traffic knob. Consumers pull records, and the unit of parallelism is the partition. Within a consumer group, each partition is owned by one consumer at a time. If a topic has 12 partitions and four equally loaded instances, each instance owns about three. When one instance disappears, a rebalance gives the survivors more partitions -- and the extra work arrives as a step change, not as a smooth increase from a load balancer. The key translation: for an HTTP ## KCP: How to Migrate to Confluent Cloud in Days, Not Weeks DevFeed: [KCP: How to Migrate to Confluent Cloud in Days, Not Weeks]() Original publisher: [Read original article]() Author: Ahmed Saef Zamzam Published: 2026-09-14T07:00:00Z Content type: tutorial Language: en Sources: [Confluent: Data in motion]() Topics: [Kafka](), [migration](), [Confluent Cloud](), [Replication](), [Infrastructure as code](), [Provisioning](), [Access Control](), [Structured-data]() Tags: [access-control](), [confluent](), [confluent-cloud](), [data-replication](), [infrastructure-as-code-iac](), [kafka](), [migration](), [open-source](), [provisioning](), [replication](), [schema]() ### AI overview The article explains how Confluent's open source KCP tool automates migration from Amazon MSK to Confluent Cloud. KCP supports discovery, infrastructure provisioning, ACL and schema mapping, and migration, while Cluster Linking provides offset-preserving data replication. Support for self-managed Kafka migrations is described as coming soon. ### Source excerpt Use Kafka Copy Paste (KCP) to automate Kafka migration with infrastructure generation, ACL and schema mapping, and offset-preserving data replication. ## Data Engineering Weekly #287 DevFeed: [Data Engineering Weekly #287]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-09-14T02:52:23Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [Kafka](), [data observability](), [Embeddings](), [Multi-tenancy](), [Event-Streaming](), [OpenAI](), [Library](), [Code]() Tags: [architecture](), [architectures](), [data-engineering](), [data-quality](), [embedding](), [engineering](), [kafka](), [multi-tenancy](), [object-storage](), [observability](), [openai](), [streaming](), [vector]() ### AI overview Data Engineering Weekly #287 covers building data platforms from scratch, including composable architectures, data quality, and observability. It also previews talks on governed machine-executable ontologies for marketing activation and fair, order-preserving Kafka consumption for many tenants. The issue links to OpenAI's storage platform scaling for ChatGPT and Pinterest's embedding retrieval platform. ### Source excerpt The Weekly Data Engineering Newsletter ## Netflix Reworks Conductor for 420 Million Monthly Workflow Executions and 10X Larger Workflows DevFeed: [Netflix Reworks Conductor for 420 Million Monthly Workflow Executions and 10X Larger Workflows]() Original publisher: [Read original article]() Author: Leela Kumili Published: 2026-09-11T14:17:00Z Content type: news Language: en Sources: [InfoQ]() Topics: [Orchestration](), [Amazon S3](), [Kafka](), [Apache Iceberg]() Tags: [amazon-s3](), [apache-iceberg](), [apache-kafka](), [architecture-design](), [asynchronous-architecture](), [cassandra](), [cloud-architecture](), [concurrency](), [development](), [devops](), [distributed-systems](), [elasticsearch](), [java-operator-sdk](), [kafka](), [latency](), [microservices](), [netflix](), [netflix-conductor-4-workflow](), [news](), [orchestration](), [s3](), [scalability](), [windows-workflow-foundation](), [workflow](), [workflow-bpm](), [workflow-foundation]() ### AI overview Netflix reworked Conductor 4.0 to scale workflow orchestration to roughly 200,000 definitions and 420 million monthly executions. The redesign raises supported workflow size to 30,000 tasks and reports a roughly 40% reduction in p99 evaluation latency by loading only task data needed for each decision. ### Source excerpt Netflix has reworked its Conductor workflow orchestration engine to handle larger workloads, increasing supported workflow size from about 2,500 to 30,000 tasks and reducing p99 workflow evaluation latency by about 40%. Conductor 4.0 separates workflow metadata from task data, moves evaluation to asynchronous processing, and introduces dynamic worker allocation and concurrency controls. By Leela Kumili ## Troubleshoot Kafka issues across every layer of your stack with Kafka Console DevFeed: [Troubleshoot Kafka issues across every layer of your stack with Kafka Console]() Original publisher: [Read original article]() Author: Tori Engler; Shelly Matskel Published: 2026-09-10T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [Kafka](), [configuration]() Tags: [apache](), [data-streams-monitoring](), [debugging](), [event-streaming](), [infrastructure](), [kafka](), [monitor](), [performance](), [root-cause-analysis]() ### AI overview An overview of Datadog Kafka Console for diagnosing Kafka health and performance issues, inspecting messages, and tuning configurations. ### Source excerpt Learn how Kafka Console helps you identify Kafka issues, inspect messages, and tune configurations with infrastructure and app context. ## APIs & Service Communication: The Essential Guide DevFeed: [APIs & Service Communication: The Essential Guide]() Original publisher: [Read original article]() Author: Nikki Siapno Published: 2026-09-08T13:03:41Z Content type: tutorial Language: en Sources: [Level Up Coding System Design Newsletter]() Topics: [API](), [Microservice](), [GraphQL](), [gRPC](), [Kafka](), [WebSocket]() Tags: [api](), [apis](), [graphql](), [http](), [kafka](), [rest]() ### AI overview A tutorial comparing communication patterns for distributed systems. It explains how request-response APIs, event streams, and bidirectional connections affect responsiveness, service coupling, failure propagation, operational complexity, reliability, and scalability, then introduces REST and GraphQL with their main strengths, tradeoffs, and use cases. ### Source excerpt REST, gRPC, Kafka, WebSockets, Pub/Sub, and more. How do they all fit together? ## Five Years of Kafka at Razorpay's UPI Switch DevFeed: [Five Years of Kafka at Razorpay's UPI Switch]() Original publisher: [Read original article]() Author: Kshitij Nawandar Published: 2026-09-07T09:09:58Z Content type: article Language: en Sources: [Razorpay Engineering - Medium]() Topics: [Kafka](), [Architecture & Design](), [event driven](), [Amazon Simple Queue Service (SQS)](), [distributed-systems](), [Microservices](), [Amazon Web Services]() Tags: [aws](), [aws-sns](), [distributed-systems](), [event-driven](), [kafka](), [microservices](), [razorpay](), [redshift](), [sns](), [sqs]() ### AI overview Razorpay describes five years of evolution in its UPI Switch, from a monolith using AWS SQS to Kafka-based infrastructure. The article covers architectural decisions, operational challenges, and optimization work affecting payment-processing performance, reliability, and scale. ### Source excerpt Preface The UPI Switch at Razorpay has evolved significantly in the five years since we started building it. The Switch is the platform that enables real-time payment processing with NPCI. When the team began, it was little more than an idea. Today it powers more than 70% of Razorpay's total UPI volume. Because UPI is inherently asynchronous, a messaging system sits at the heart of the Switch and has a direct impact on performance, reliability, and scale. What began as a straightforward queue became the core of the system, shaping how every new feature was designed and delivered. This post covers that evolution: the decisions that enabled growth, the ones that slowed us down, the operational issues that forced us to rethink our assumptions, and the optimizations that ultimately stabilized our Kafka-based infrastructure. This is the story of what we got right, what we got wrong, and how we eventually built something stable enough to grow on. The First Version: Monolith and SQS When we began building the UPI Switch, we weren't thinking about massive scale, distributed systems, or elegant event routing. So we built Switch v1 as a monolith. No microservices, no distributed orchestration: just one solid block of code doing everything. That was the right call. We needed to move fast, experiment, and learn, and we followed the Keep It Simple, Stupid (KISS) principle deliberately. For messaging, we picked AWS SQS: reliable, managed, and low on cognitive load. We didn't need ordering guarantees at the time, so a standard queue worked fine. We started with just two queues, and this setup held its ground. It handled a peak of 400 TPS during the IPL. The limitations showed up as the ecosystem grew. A single event, like a successful payment, needed to fan out into multiple workflows: Update NPCI with an API call Send callbacks to merchants about payment status Push structured data into our warehouse (AWS Redshift) To handle this, we started bolting on AWS SNS plus SQS for fan-ou ## How INTEGER and INT Produced Different Schemas in Debezium DevFeed: [How INTEGER and INT Produced Different Schemas in Debezium]() Original publisher: [Read original article]() Author: Prathit Malik Published: 2026-09-02T07:02:02Z Content type: article Language: en Sources: [Grofers]() Topics: [MySQL](), [migration](), [Streaming](), [Kafka](), [SQL](), [Databases]() Tags: [big-data](), [blinkit](), [change](), [database](), [debezium](), [jdbc](), [kafka](), [migration](), [mysql](), [open-source](), [schema](), [sql](), [streaming]() ### AI overview This article investigates a Debezium CDC pipeline failure caused by MySQL INT and INTEGER synonyms being treated as different types. The resulting schema mismatch produced an Integer where a downstream consumer expected a Long, causing a ClassCastException on every batch. It explains how streaming and snapshot schema handling differ and how routine migrations exposed the problem. ### Source excerpt A Debezium investigation: how MySQL synonyms INT and INTEGER were treated as different types. One of our CDC pipelines started failing with a ClassCastException on every batch. java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Long The pipeline was producing an Integer, but the downstream consumer expected a Long. Every run failed in the same way, which pointed us toward a schema mismatch rather than an issue with individual records. Background: how CDC works To see why a mismatch like that can hide for years, it helps to know how CDC actually works. Most companies replicate their transactional database (MySQL, Postgres, something similar) into a separate data lake for analytics, rather than querying the source directly, and Change Data Capture (CDC) is what keeps that copy in sync: it tails the database's transaction log and replays every insert, update, and delete downstream. Debezium is the most widely used open-source CDC tool for MySQL, and it builds a table's schema in one of two ways that are supposed to agree but do not always. Streaming mode: the first time it sees a CREATE TABLE or ALTER TABLE in the binlog, it parses the raw SQL text and writes the result to its own internal Kafka topic, database.history.kafka.topic. Every restart after that rebuilds the in-memory schema by replaying that topic, not by re-reading the binlog. Snapshot mode: reads the table definition fresh through MySQL's JDBC metadata interface, which normalizes types, every time it runs. Schema Registry sits downstream of both: each connector writes whatever schema it built into the registry, but neither connector reads its own schema back from it. Keep that in mind; it matters later. The trigger: a routine migration With that in mind, here's what actually happened to us. Rewind two years: one of our upstream service teams added a few columns to a source table as part of a standard schema change. ALTER TABLE ADD COLUMN length double NU ## FusionAuth 1.69 adds an AI agent entity type and lifecycle webhooks DevFeed: [FusionAuth 1.69 adds an AI agent entity type and lifecycle webhooks]() Original publisher: [Read original article]() Author: WorkOS Published: 2026-09-01T17:46:01Z Content type: opinion Language: en Sources: [WorkOS Blog]() Topics: [AI Agent](), [Authorization](), [releases](), [Security](), [Authentication]() Tags: [ai-agent](), [authentication](), [authorization](), [kafka](), [release](), [release-notes](), [security](), [webhooks]() ### AI overview The article examines FusionAuth 1.69, which adds an AI agent entity type and lifecycle webhooks. It argues that identity, authorization, and audit data need to be connected to answer which agent accessed a resource, under whose authority, and within what scope. It also reviews related webhook, audit-log, issuer-identification, and signature-verification features. ### Source excerpt FusionAuth 1.69 adds an AI agent entity type and lifecycle webhooks. Whether that answers the question your auditors ask depends on whose events they are. ## Where Kafka Fits in Modern Systems DevFeed: [Where Kafka Fits in Modern Systems]() Original publisher: [Read original article]() Author: Nikki Siapno Published: 2026-08-25T12:02:09Z Content type: tutorial Language: en Sources: [Level Up Coding System Design Newsletter]() Topics: [Kafka](), [Event-Streaming](), [systems](), [distributed-systems](), [data]() Tags: [architecture](), [consumer](), [distributed-systems](), [durability](), [event-streaming](), [kafka](), [parallelism](), [partition](), [pipelines](), [real-time](), [systems]() ### AI overview This article explains when Apache Kafka is appropriate for distributed systems. It presents Kafka as a durable event log whose replayability, consumer fan-out, and partition-based parallelism support real-time data pipelines and shared event-driven architectures, while warning that Kafka can add unnecessary operational complexity for simpler asynchronous workflows. ### Source excerpt When should you actually use Kafka? And when is it just unnecessary complexity? ## Push-button migration from Confluent to Redpanda with Shadowing DevFeed: [Push-button migration from Confluent to Redpanda with Shadowing]() Original publisher: [Read original article]() Author: Trevor Blackford Published: 2026-08-25T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [migration](), [Replication](), [Kafka](), [Confluent Cloud](), [Confluent Platform](), [Disaster Recovery](), [Usability]() Tags: [big-bang](), [confluent-cloud](), [confluent-platform](), [data-centers](), [disaster-recovery](), [kafka](), [migration](), [replication](), [schema]() ### AI overview The article presents Redpanda Shadowing 26.2 as a low-risk migration path from Confluent Cloud, Confluent Platform, or other Apache Kafka-compatible clusters. It replicates topic data, schemas, offsets, and ACLs while preserving offsets, allowing teams to validate workloads and move applications individually instead of using a big-bang cutover. ### Source excerpt Migrate off Confluent without the "big cutover weekend." Redpanda Shadowing carries topic data, schemas, offsets, and ACLs on a single link. Available on Self-Managed , BYOC, and Dedicated. ## API Mocking and Testing With Microcks DevFeed: [API Mocking and Testing With Microcks]() Original publisher: [Read original article]() Author: Andrei Branza Published: 2026-08-22T23:20:24Z Content type: tutorial Language: en Sources: [Baeldung]() Topics: [Mocking](), [Testcontainers](), [API](), [OpenAPI Specification](), [Microservices](), [Docker Compose](), [Docker](), [Kafka](), [Amazon Simple Queue Service (SQS)](), [GraphQL](), [gRPC](), [Jenkins]() Tags: [api](), [compose](), [docker](), [how-to](), [kafka](), [kubernetes](), [mocking](), [openapi](), [popular](), [testcontainers](), [testing](), [testing-popular-testcontainers]() ### AI overview This tutorial explains how to use Microcks with Testcontainers to generate API mocks from contracts and run conformance tests against real implementations. It covers importing an OpenAPI contract, starting Microcks in a JUnit 5 test, calling generated mock endpoints, and integrating mocking and testing into delivery workflows. ### Source excerpt Learn how to use Testcontainers integration to spin up Microcks inside a JUnit5 test. The post API Mocking and Testing With Microcks first appeared on Baeldung. ## A Tale of Two Flink Autoscalers DevFeed: [A Tale of Two Flink Autoscalers]() Original publisher: [Read original article]() Author: Netflix Technology Blog Published: 2026-08-21T16:01:01Z Content type: article Language: en Sources: [Netflix](), [Netflix TechBlog - Medium]() Topics: [apache-flink](), [autoscaling](), [Netflix](), [stream-processing](), [Kafka](), [Open Source](), [telemetry](), [Network]() Tags: [apache-flink](), [autoscaling](), [aws](), [infrastructure](), [kafka](), [metrics](), [netflix](), [open-source](), [operational-excellence](), [scale](), [stream-processing](), [telemetry]() ### AI overview Netflix describes running two Apache Flink autoscalers: an older in-house system and a newer one from the Apache Flink community. The article explains why autoscaling is necessary for more than 30,000 Flink jobs across AWS regions, discusses workload variability and the cost of resizing stateful jobs, and introduces the metrics-driven design of the first autoscaler. ### Source excerpt Samuel Yeboah, Francesco Di Chiara and Mingliang Liu Today, Netflix runs two Flink autoscalers. That is exactly one more than we want. We built the first one in-house years ago, when there was no mature option suited to our platform. The second came from the Apache Flink community, and it can scale workloads our homegrown system was never designed for. We now run both in production and are steadily converging on the open-source one. Along the way we learned some hard lessons about metrics, cost, and the real price of maintaining infrastructure you could instead adopt, and we hope they are useful whether you run a handful of Flink jobs or tens of thousands. Why autoscaling is not optional at our scale Netflix has run stream processing on Apache Flink since 2017. As of 2026 we operate more than 30,000 Flink jobs across multiple AWS regions. Most are not deployed by hand; they are generated by our managed platform Data Mesh, so the majority of users never touch a Flink job directly. A smaller but growing set are custom jobs, built and operated by teams across the company for use cases like personalization, Ads, and Live events. They range from single-operator jobs that shuttle records between Kafka topics to stateful pipelines with branches, joins, and terabytes of state, and their load swings with daily cycles, launches, and regional failovers. Provisioning every one of those jobs for its peak is wasteful; provisioning for the average causes lag during surges. And in our platform a scaling action is not free: by default it means taking a savepoint, stopping the job gracefully, and restarting it at the new size, which for a large stateful job can take minutes. That leaves a genuinely hard question: how do you give each job the resources it needs, when it needs them, without a human in the loop and without breaking anything? The first autoscaler: watching from outside Our first answer, built around 2019, was an autoscaler shaped like a stream-processing job. It ran on M ## POSETTE Talk Recap - Postgres Isn't Slow. Your Storage Is DevFeed: [POSETTE Talk Recap - Postgres Isn't Slow. Your Storage Is]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-08-20T16:11:08Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [PostgreSQL](), [benchmarking](), [IO](), [Latency](), [Replication](), [systems](), [clickhouse](), [Kafka]() Tags: [benchmark](), [clickhouse](), [cpu](), [kafka](), [latency](), [local](), [performance](), [postgres](), [postgresql](), [replication](), [storage](), [systems]() ### AI overview This article recaps a POSETTE 2026 talk about how storage performance affects PostgreSQL workloads at scale. It describes a benchmark comparing a 3.3-billion-row workload on gp3 EBS and local NVMe, covering ingestion, read latency, autovacuum, checkpoints, and logical replication, and introduces production patterns for running PostgreSQL on local NVMe. ### Source excerpt A recap of Sai Srirampur's POSETTE 2026 talk on storage performance in Postgres, with a local NVMe vs. EBS benchmark and the production setup behind it. ## New in Confluent Cloud and WarpStream: Evolving the Data Streaming Platform for AI, Scale, and Control DevFeed: [New in Confluent Cloud and WarpStream: Evolving the Data Streaming Platform for AI, Scale, and Control]() Original publisher: [Read original article]() Author: Mike Agnich Published: 2026-08-18T14:00:11Z Content type: article Language: en Sources: [Confluent: Data in motion]() Topics: [Streaming](), [Kafka](), [Cloud](), [event driven](), [Transactions]() Tags: [confluent](), [confluent-cloud](), [debug](), [kafka](), [networking](), [security](), [stream-processing](), [streaming](), [transactions]() ### AI overview Confluent describes more than 70 new features for its data streaming platform, including new connectors, networking improvements, AI capabilities for event-driven agents, real-time machine learning in streaming pipelines, and AI-assisted development tools. The article also discusses broader Kafka workload support, including stream processing, transactions, exactly-once processing, queues, debugging, and Python client support for Queues for Kafka. ### Source excerpt Accelerate enterprise streaming and AI workloads with new product updates across Kora, connectors, Flink, Tableflow, AI anomaly detection and forecasting, security enhancements, and Warpstream. ## New in Confluent Intelligence and AI Tools: Making Agents Native to the Stream, Expanded Model Support, New Agent Skills, and Copilot DevFeed: [New in Confluent Intelligence and AI Tools: Making Agents Native to the Stream, Expanded Model Support, New Agent Skills, and Copilot]() Original publisher: [Read original article]() Author: Confluent Staff Published: 2026-08-18T14:00:10Z Content type: article Language: en Sources: [Confluent: Data in motion]() Topics: [Artificial Intelligence](), [apache-flink](), [Kafka](), [streaming-data-processing](), [real-time](), [MCP Server](), [Agent Skills](), [MCP](), [Inference](), [Google](), [ibm]() Tags: [ai](), [ai-ready-data](), [ai-tools](), [apache-flink](), [confluent](), [confluent-cloud](), [google](), [ibm](), [kafka](), [mcp](), [mcp-server](), [real-time](), [skills](), [streaming-data-processing](), [support](), [tools]() ### AI overview Confluent announces updates to Confluent Intelligence and related AI tools for building production AI systems on Apache Kafka and Apache Flink. The release adds expanded time-series model support, a generally available Real-Time Context Engine, updates to the fully managed MCP Server, Agent Skills for AI coding assistants, and Confluent Copilot. The features are intended to provide agents and applications with fresh business context and governed access to live data and infrastructure. ### Source excerpt Explore new AI features and AI tools: support for IBM Granite Time Series models and TimesFM models (EA), enhanced Real-Time Context Engine experience, new Agent Skills, Confluent Copilot ## Confluent Cloud for Apache Flink: Engine for Mission-Critical, Real-Time Operational Systems and dbt/SQL-Native Home for Data Science and AI DevFeed: [Confluent Cloud for Apache Flink: Engine for Mission-Critical, Real-Time Operational Systems and dbt/SQL-Native Home for Data Science and AI]() Original publisher: [Read original article]() Author: Yashwanth Dasari Published: 2026-08-18T02:20:00Z Content type: release Language: en Sources: [Confluent: Data in motion]() Topics: [Confluent Cloud](), [apache-flink](), [stream-processing](), [Kafka](), [AI, ML & Data Engineering](), [Cloud](), [SDKs]() Tags: [ai](), [announcement](), [apache-flink](), [api](), [batch](), [confluent-cloud](), [data-science](), [developers](), [kafka](), [real-time]() ### AI overview Confluent announces an evolution of Confluent Cloud for Apache Flink that unifies mission-critical real-time operations with analytics and AI workflows. The release adds a serverless, co-designed Kafka and Flink platform, including the generally available Flink Table API in Java for code-first development. ### Source excerpt Flink now acts as a robust engine for developers through Table API, UDFs, and PTFs while offering a SQL-native, dbt-integrated platform for data science and AI teams. ## Agent Night demo recap: Brian Douglas's self-healing Pokémon agent DevFeed: [Agent Night demo recap: Brian Douglas's self-healing Pokémon agent]() Original publisher: [Read original article]() Author: WorkOS Published: 2026-08-17T18:34:12Z Content type: news Language: en Sources: [WorkOS Blog]() Topics: [Demo](), [observability ai agents](), [telemetry](), [Claude Code](), [Python](), [DuckDB](), [Kafka](), [Code](), [Open Source]() Tags: [agent](), [claude-code](), [code](), [coding](), [demo](), [duckdb](), [kafka](), [open-source](), [python](), [self-healing](), [telemetry](), [traces]() ### AI overview A recap of Brian Douglas's Agent Night demo shows how Paper Compute ran Pokémon as a self-healing agent for six months. The setup uses Claude Code, a headless Python emulator, and traces from each session and turn. The article explains how labeled traces and per-session reflections can make the agent's history searchable and useful for improvement. ### Source excerpt Brian Douglas of Paper Compute closed Agent Night with six months of running Pokémon as a self-healing agent: traces, anomalies, dreams, and inceptions. ## Data Engineering Weekly #283 DevFeed: [Data Engineering Weekly #283]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-17T02:59:40Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [AI, ML & Data Engineering](), [Data Quality](), [Microservice](), [Apache Spark](), [Kafka](), [Feature Engineering](), [audit trail](), [Protocol (disambiguation)]() Tags: [ai](), [anthropic](), [article](), [data-engineering](), [data-quality](), [engineering](), [feature-engineering](), [kafka](), [newsletter](), [pipelines](), [services](), [testing]() ### AI overview Data Engineering Weekly #283 is a newsletter covering data platform fundamentals, multiagent system coordination, payments platform data contracts, financial data quality, declarative data engineering, and cost-efficient export workloads. ### Source excerpt The Weekly Data Engineering Newsletter ## How Neuron Systems Served 2.3 Million Fans Across 104 World Cup Matches with AI on Confluent DevFeed: [How Neuron Systems Served 2.3 Million Fans Across 104 World Cup Matches with AI on Confluent]() Original publisher: [Read original article]() Author: Shalini Ananda, PhD Published: 2026-08-14T06:48:05Z Content type: article Language: en Sources: [Confluent: Data in motion]() Topics: [Artificial Intelligence](), [Kafka](), [Streaming](), [real-time](), [observability](), [systems]() Tags: [agents](), [ai](), [architecture](), [confluent](), [kafka](), [observability](), [production](), [real-time](), [scale](), [solutions](), [streaming]() ### AI overview Neuron Systems used six agents and Confluent's Data Streaming Platform to provide multilingual live commentary for all 104 FIFA World Cup 2026 matches. The platform processed 42.1 million production events, maintained a median glass-to-glass latency of 42 milliseconds, and supported millions of fans while keeping commentary synchronized. ### Source excerpt Every one of the 42.1 million production events flowed through Confluent's Data Streaming Platform and that single decision is what let a small team serve a global tournament without the commentary ever falling out of sync. ## Data pipeline monitoring 101: Tracking health and performance across the data stack DevFeed: [Data pipeline monitoring 101: Tracking health and performance across the data stack]() Original publisher: [Read original article]() Author: Aaron Kaplan; Ryan Warrier Published: 2026-08-14T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [Monitoring](), [data observability](), [observability](), [data](), [data streams monitoring](), [Kafka](), [Apache Spark](), [event driven](), [Streaming]() Tags: [ai-ml](), [analytics](), [data](), [data-analytics](), [data-engineering](), [data-observability](), [data-streams-monitoring](), [event-driven](), [kafka](), [learn](), [monitoring](), [observability](), [performance](), [spark](), [streaming]() ### AI overview This article introduces end-to-end monitoring for modern data pipelines. It explains how to track pipeline health, performance, data quality, and availability across varied architectures and technology layers, with examples including Kafka, Flink, Apache Spark, data lakes, warehouses, and lakehouses. ### Source excerpt Learn about monitoring the end-to-end health and performance of modern data pipelines. ## Designing for failure: Building resilient systems on AWS DevFeed: [Designing for failure: Building resilient systems on AWS]() Original publisher: [Read original article]() Author: Dhvani Vora Published: 2026-08-13T20:03:28Z Content type: tutorial Language: en Sources: [AWS Compute Blog]() Topics: [incident](), [Incident response]() Tags: [amazon-ec2](), [aws](), [aws-architecture-blog-resilience](), [aws-management-tools-blog-aws-resilience-hub](), [best-practices](), [cassandra](), [distributed-systems](), [hardware](), [incident](), [intermediate-200](), [kafka](), [observability](), [resilience](), [systems]() ### AI overview An AWS resilience guide examines a correlated EC2 hardware failure that took down three colocated Cassandra nodes. It covers placement groups, alarms, automated recovery, and observability patterns for distributed data systems. ### Source excerpt Learn how to prevent correlated hardware failures in distributed systems on Amazon EC2. This post walks through real incident response patterns, including Partition Placement Groups, composite alarms, automated recovery with Auto Scaling, and observability best practices. ## Multi-region high availability for Kafka workloads with a single Stretch Cluster DevFeed: [Multi-region high availability for Kafka workloads with a single Stretch Cluster]() Original publisher: [Read original article]() Author: David Yu Published: 2026-08-11T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Kafka](), [Kubernetes](), [Availability](), [Replication](), [Raft](), [Resilience](), [stream-processing]() Tags: [clusters](), [k8s](), [kafka](), [raft](), [release](), [replication](), [resilience](), [stream-processing]() ### AI overview Redpanda Operator 26.2 introduces generally available Stretch Clusters, allowing one logical Redpanda cluster to span multiple Kubernetes clusters and regions. The release provides synchronous replication with Raft-based automatic failover, safer broker rolling restarts, Redpanda Connect pipelines as Kubernetes resources, and Gateway API support for Redpanda Console. ### Source excerpt Redpanda Operator 26.2 brings GA Stretch Clusters for multi-region replication, Redpanda Connect pipelines as K8s resources, Gateway API support, and safer rolling restarts. [Next page]()