# Kafka Open-source distributed event streaming platform for data pipelines, streaming analytics, data integration, and mission-critical applications. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Why Kafka retention.ms can delay message deletion DevFeed: [Why Kafka retention.ms can delay message deletion]() Original publisher: [Read original article]() Author: Rogerio Santos Published: 2026-09-16T13:05:06Z Content type: tutorial Language: en Sources: [Red Hat](), [Red Hat Developer]() Topics: [Kafka](), [Apache-Kafka](), [configuration](), [file](), [Event-Streaming]() Tags: [broker](), [configuration](), [diagnose](), [event-driven](), [how-to](), [kafka](), [lifecycle](), [retention](), [storage](), [stream-processing]() ### AI overview This guide explains why Kafka messages can remain readable beyond a topic's retention.ms setting. Kafka deletes closed log segments rather than individual messages, and the active segment and continuous writes can delay removal of older records. ### Source excerpt A customer opened a support case with a deceptively simple complaint: a Kafka topic was configured with a 12-hour retention (retention.ms), yet messages produced on July 24 were still readable 4 days later, on July 28. Nothing was broken. The broker logged no errors. The retention policy was working as designed, but the segment layout and continuous write pattern delayed when the old records could actually be removed. The post Why your Kafka topic ignores retention.ms (and how to fix it) appeared first on Red Hat Developer. ## How We Built Automated Capacity Testing for Kafka Consumers DevFeed: [How We Built Automated Capacity Testing for Kafka Consumers]() Original publisher: [Read original article]() Author: Kaan Karakaya Published: 2026-09-14T09:46:34Z Content type: tutorial Language: en Sources: [Booking.com Development - Medium]() Topics: [Kafka](), [Testing](), [HTTP]() Tags: [automated](), [capacity](), [health-checks](), [kafka](), [load](), [load-balancer](), [parallelism](), [partition](), [recovery](), [scale](), [site-reliability-engineer](), [sre](), [testing]() ### AI overview This article describes Booking.com's extension of an existing capacity-testing platform for Kafka consumers. It explains how changing partition assignment can provide a controlled, measurable way to test consumer throughput and whether remaining consumers can absorb reassigned work after an instance or failure domain disappears. ### Source excerpt Photo by GuerrillaBuzz on Unsplash Kafka makes it easy to distribute work across consumer instances. It is much harder to prove, safely and repeatedly, how those instances behave when the distribution changes and one of them has to carry more than its usual share. For teams that run Kafka at scale, this is a practical reliability question: how much load can a consumer instance actually handle? We had automated capacity testing for HTTP services, but Kafka consumers were still tested with manual drills. Those drills could tell us something, but they were disruptive, difficult to reproduce, and risky precisely when the system was close to its limit. We wanted a controlled way to answer three questions: What is the maximum sustainable throughput of a consumer instance? If an instance or failure domain disappears, can the remaining consumers absorb the reassigned work? Are we overprovisioning resources because we do not know the real limit? The result was an extension to our capacity-testing platform that turns Kafka partition assignment into a safe, measurable load-control mechanism. Why HTTP capacity testing did not translate Our existing platform was designed for request-response services behind a load balancer. A scheduled test selects one instance, routes an increasing share of traffic to it, runs health checks after each step, and records the highest ratio the instance can sustain. After the test, traffic returns to its normal distribution and the result is reported to the service owner. Kafka has no equivalent traffic knob. Consumers pull records, and the unit of parallelism is the partition. Within a consumer group, each partition is owned by one consumer at a time. If a topic has 12 partitions and four equally loaded instances, each instance owns about three. When one instance disappears, a rebalance gives the survivors more partitions -- and the extra work arrives as a step change, not as a smooth increase from a load balancer. The key translation: for an HTTP ## KCP: How to Migrate to Confluent Cloud in Days, Not Weeks DevFeed: [KCP: How to Migrate to Confluent Cloud in Days, Not Weeks]() Original publisher: [Read original article]() Author: Ahmed Saef Zamzam Published: 2026-09-14T07:00:00Z Content type: tutorial Language: en Sources: [Confluent: Data in motion]() Topics: [Kafka](), [migration](), [Confluent Cloud](), [Replication](), [Infrastructure as code](), [Provisioning](), [Access Control](), [Structured-data]() Tags: [access-control](), [confluent](), [confluent-cloud](), [data-replication](), [infrastructure-as-code-iac](), [kafka](), [migration](), [open-source](), [provisioning](), [replication](), [schema]() ### AI overview The article explains how Confluent's open source KCP tool automates migration from Amazon MSK to Confluent Cloud. KCP supports discovery, infrastructure provisioning, ACL and schema mapping, and migration, while Cluster Linking provides offset-preserving data replication. Support for self-managed Kafka migrations is described as coming soon. ### Source excerpt Use Kafka Copy Paste (KCP) to automate Kafka migration with infrastructure generation, ACL and schema mapping, and offset-preserving data replication. ## Data Engineering Weekly #287 DevFeed: [Data Engineering Weekly #287]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-09-14T02:52:23Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [Kafka](), [data observability](), [Embeddings](), [Multi-tenancy](), [Event-Streaming](), [OpenAI](), [Library](), [Code]() Tags: [architecture](), [architectures](), [data-engineering](), [data-quality](), [embedding](), [engineering](), [kafka](), [multi-tenancy](), [object-storage](), [observability](), [openai](), [streaming](), [vector]() ### AI overview Data Engineering Weekly #287 covers building data platforms from scratch, including composable architectures, data quality, and observability. It also previews talks on governed machine-executable ontologies for marketing activation and fair, order-preserving Kafka consumption for many tenants. The issue links to OpenAI's storage platform scaling for ChatGPT and Pinterest's embedding retrieval platform. ### Source excerpt The Weekly Data Engineering Newsletter ## Netflix Reworks Conductor for 420 Million Monthly Workflow Executions and 10X Larger Workflows DevFeed: [Netflix Reworks Conductor for 420 Million Monthly Workflow Executions and 10X Larger Workflows]() Original publisher: [Read original article]() Author: Leela Kumili Published: 2026-09-11T14:17:00Z Content type: news Language: en Sources: [InfoQ]() Topics: [Orchestration](), [Amazon S3](), [Kafka](), [Apache Iceberg]() Tags: [amazon-s3](), [apache-iceberg](), [apache-kafka](), [architecture-design](), [asynchronous-architecture](), [cassandra](), [cloud-architecture](), [concurrency](), [development](), [devops](), [distributed-systems](), [elasticsearch](), [java-operator-sdk](), [kafka](), [latency](), [microservices](), [netflix](), [netflix-conductor-4-workflow](), [news](), [orchestration](), [s3](), [scalability](), [windows-workflow-foundation](), [workflow](), [workflow-bpm](), [workflow-foundation]() ### AI overview Netflix reworked Conductor 4.0 to scale workflow orchestration to roughly 200,000 definitions and 420 million monthly executions. The redesign raises supported workflow size to 30,000 tasks and reports a roughly 40% reduction in p99 evaluation latency by loading only task data needed for each decision. ### Source excerpt Netflix has reworked its Conductor workflow orchestration engine to handle larger workloads, increasing supported workflow size from about 2,500 to 30,000 tasks and reducing p99 workflow evaluation latency by about 40%. Conductor 4.0 separates workflow metadata from task data, moves evaluation to asynchronous processing, and introduces dynamic worker allocation and concurrency controls. By Leela Kumili ## Troubleshoot Kafka issues across every layer of your stack with Kafka Console DevFeed: [Troubleshoot Kafka issues across every layer of your stack with Kafka Console]() Original publisher: [Read original article]() Author: Tori Engler; Shelly Matskel Published: 2026-09-10T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [Kafka](), [configuration]() Tags: [apache](), [data-streams-monitoring](), [debugging](), [event-streaming](), [infrastructure](), [kafka](), [monitor](), [performance](), [root-cause-analysis]() ### AI overview An overview of Datadog Kafka Console for diagnosing Kafka health and performance issues, inspecting messages, and tuning configurations. ### Source excerpt Learn how Kafka Console helps you identify Kafka issues, inspect messages, and tune configurations with infrastructure and app context. ## Five Years of Kafka at Razorpay's UPI Switch DevFeed: [Five Years of Kafka at Razorpay's UPI Switch]() Original publisher: [Read original article]() Author: Kshitij Nawandar Published: 2026-09-07T09:09:58Z Content type: article Language: en Sources: [Razorpay Engineering - Medium]() Topics: [Kafka](), [Architecture & Design](), [event driven](), [Amazon Simple Queue Service (SQS)](), [distributed-systems](), [Microservices](), [Amazon Web Services]() Tags: [aws](), [aws-sns](), [distributed-systems](), [event-driven](), [kafka](), [microservices](), [razorpay](), [redshift](), [sns](), [sqs]() ### AI overview Razorpay describes five years of evolution in its UPI Switch, from a monolith using AWS SQS to Kafka-based infrastructure. The article covers architectural decisions, operational challenges, and optimization work affecting payment-processing performance, reliability, and scale. ### Source excerpt Preface The UPI Switch at Razorpay has evolved significantly in the five years since we started building it. The Switch is the platform that enables real-time payment processing with NPCI. When the team began, it was little more than an idea. Today it powers more than 70% of Razorpay's total UPI volume. Because UPI is inherently asynchronous, a messaging system sits at the heart of the Switch and has a direct impact on performance, reliability, and scale. What began as a straightforward queue became the core of the system, shaping how every new feature was designed and delivered. This post covers that evolution: the decisions that enabled growth, the ones that slowed us down, the operational issues that forced us to rethink our assumptions, and the optimizations that ultimately stabilized our Kafka-based infrastructure. This is the story of what we got right, what we got wrong, and how we eventually built something stable enough to grow on. The First Version: Monolith and SQS When we began building the UPI Switch, we weren't thinking about massive scale, distributed systems, or elegant event routing. So we built Switch v1 as a monolith. No microservices, no distributed orchestration: just one solid block of code doing everything. That was the right call. We needed to move fast, experiment, and learn, and we followed the Keep It Simple, Stupid (KISS) principle deliberately. For messaging, we picked AWS SQS: reliable, managed, and low on cognitive load. We didn't need ordering guarantees at the time, so a standard queue worked fine. We started with just two queues, and this setup held its ground. It handled a peak of 400 TPS during the IPL. The limitations showed up as the ecosystem grew. A single event, like a successful payment, needed to fan out into multiple workflows: Update NPCI with an API call Send callbacks to merchants about payment status Push structured data into our warehouse (AWS Redshift) To handle this, we started bolting on AWS SNS plus SQS for fan-ou ## How INTEGER and INT Produced Different Schemas in Debezium DevFeed: [How INTEGER and INT Produced Different Schemas in Debezium]() Original publisher: [Read original article]() Author: Prathit Malik Published: 2026-09-02T07:02:02Z Content type: article Language: en Sources: [Grofers]() Topics: [MySQL](), [migration](), [Streaming](), [Kafka](), [SQL](), [Databases]() Tags: [big-data](), [blinkit](), [change](), [database](), [debezium](), [jdbc](), [kafka](), [migration](), [mysql](), [open-source](), [schema](), [sql](), [streaming]() ### AI overview This article investigates a Debezium CDC pipeline failure caused by MySQL INT and INTEGER synonyms being treated as different types. The resulting schema mismatch produced an Integer where a downstream consumer expected a Long, causing a ClassCastException on every batch. It explains how streaming and snapshot schema handling differ and how routine migrations exposed the problem. ### Source excerpt A Debezium investigation: how MySQL synonyms INT and INTEGER were treated as different types. One of our CDC pipelines started failing with a ClassCastException on every batch. java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Long The pipeline was producing an Integer, but the downstream consumer expected a Long. Every run failed in the same way, which pointed us toward a schema mismatch rather than an issue with individual records. Background: how CDC works To see why a mismatch like that can hide for years, it helps to know how CDC actually works. Most companies replicate their transactional database (MySQL, Postgres, something similar) into a separate data lake for analytics, rather than querying the source directly, and Change Data Capture (CDC) is what keeps that copy in sync: it tails the database's transaction log and replays every insert, update, and delete downstream. Debezium is the most widely used open-source CDC tool for MySQL, and it builds a table's schema in one of two ways that are supposed to agree but do not always. Streaming mode: the first time it sees a CREATE TABLE or ALTER TABLE in the binlog, it parses the raw SQL text and writes the result to its own internal Kafka topic, database.history.kafka.topic. Every restart after that rebuilds the in-memory schema by replaying that topic, not by re-reading the binlog. Snapshot mode: reads the table definition fresh through MySQL's JDBC metadata interface, which normalizes types, every time it runs. Schema Registry sits downstream of both: each connector writes whatever schema it built into the registry, but neither connector reads its own schema back from it. Keep that in mind; it matters later. The trigger: a routine migration With that in mind, here's what actually happened to us. Rewind two years: one of our upstream service teams added a few columns to a source table as part of a standard schema change. ALTER TABLE ADD COLUMN length double NU ## A universal interface: How QuintoAndar made ClickHouse plug-and-play with managed Postgres DevFeed: [A universal interface: How QuintoAndar made ClickHouse plug-and-play with managed Postgres]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-08-27T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [data](), [Cloud](), [Kafka](), [databricks](), [API]() Tags: [api](), [clickhouse](), [cloud](), [data](), [databricks](), [house](), [latency](), [low-latency](), [tech-lead]() ### AI overview QuintoAndar rebuilt its customer data platform on ClickHouse Cloud, consolidating roughly 900 million monthly events from 14 million users and serving about 300 million API requests. The team replaced separate low-latency and batch-processing pipelines by writing events directly to ClickHouse. ClickHouse Managed Postgres provided compatibility with Hightouch and other Postgres-compatible tools. ### Source excerpt QuintoAndar unified ~900 million monthly events in ClickHouse Cloud and used ClickHouse Managed Postgres to make that data accessible to Hightouch and any Postgres-compatible tool. ## Where Kafka Fits in Modern Systems DevFeed: [Where Kafka Fits in Modern Systems]() Original publisher: [Read original article]() Author: Nikki Siapno Published: 2026-08-25T12:02:09Z Content type: tutorial Language: en Sources: [Level Up Coding System Design Newsletter]() Topics: [Kafka](), [Event-Streaming](), [systems](), [distributed-systems](), [data]() Tags: [architecture](), [consumer](), [distributed-systems](), [durability](), [event-streaming](), [kafka](), [parallelism](), [partition](), [pipelines](), [real-time](), [systems]() ### AI overview This article explains when Apache Kafka is appropriate for distributed systems. It presents Kafka as a durable event log whose replayability, consumer fan-out, and partition-based parallelism support real-time data pipelines and shared event-driven architectures, while warning that Kafka can add unnecessary operational complexity for simpler asynchronous workflows. ### Source excerpt When should you actually use Kafka? And when is it just unnecessary complexity? ## Push-button migration from Confluent to Redpanda with Shadowing DevFeed: [Push-button migration from Confluent to Redpanda with Shadowing]() Original publisher: [Read original article]() Author: Trevor Blackford Published: 2026-08-25T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [migration](), [Replication](), [Kafka](), [Confluent Cloud](), [Confluent Platform](), [Disaster Recovery](), [Usability]() Tags: [big-bang](), [confluent-cloud](), [confluent-platform](), [data-centers](), [disaster-recovery](), [kafka](), [migration](), [replication](), [schema]() ### AI overview The article presents Redpanda Shadowing 26.2 as a low-risk migration path from Confluent Cloud, Confluent Platform, or other Apache Kafka-compatible clusters. It replicates topic data, schemas, offsets, and ACLs while preserving offsets, allowing teams to validate workloads and move applications individually instead of using a big-bang cutover. ### Source excerpt Migrate off Confluent without the "big cutover weekend." Redpanda Shadowing carries topic data, schemas, offsets, and ACLs on a single link. Available on Self-Managed , BYOC, and Dedicated. ## A Tale of Two Flink Autoscalers DevFeed: [A Tale of Two Flink Autoscalers]() Original publisher: [Read original article]() Author: Netflix Technology Blog Published: 2026-08-21T16:01:01Z Content type: article Language: en Sources: [Netflix](), [Netflix TechBlog - Medium]() Topics: [apache-flink](), [autoscaling](), [Netflix](), [stream-processing](), [Kafka](), [Open Source](), [telemetry](), [Network]() Tags: [apache-flink](), [autoscaling](), [aws](), [infrastructure](), [kafka](), [metrics](), [netflix](), [open-source](), [operational-excellence](), [scale](), [stream-processing](), [telemetry]() ### AI overview Netflix describes running two Apache Flink autoscalers: an older in-house system and a newer one from the Apache Flink community. The article explains why autoscaling is necessary for more than 30,000 Flink jobs across AWS regions, discusses workload variability and the cost of resizing stateful jobs, and introduces the metrics-driven design of the first autoscaler. ### Source excerpt Samuel Yeboah, Francesco Di Chiara and Mingliang Liu Today, Netflix runs two Flink autoscalers. That is exactly one more than we want. We built the first one in-house years ago, when there was no mature option suited to our platform. The second came from the Apache Flink community, and it can scale workloads our homegrown system was never designed for. We now run both in production and are steadily converging on the open-source one. Along the way we learned some hard lessons about metrics, cost, and the real price of maintaining infrastructure you could instead adopt, and we hope they are useful whether you run a handful of Flink jobs or tens of thousands. Why autoscaling is not optional at our scale Netflix has run stream processing on Apache Flink since 2017. As of 2026 we operate more than 30,000 Flink jobs across multiple AWS regions. Most are not deployed by hand; they are generated by our managed platform Data Mesh, so the majority of users never touch a Flink job directly. A smaller but growing set are custom jobs, built and operated by teams across the company for use cases like personalization, Ads, and Live events. They range from single-operator jobs that shuttle records between Kafka topics to stateful pipelines with branches, joins, and terabytes of state, and their load swings with daily cycles, launches, and regional failovers. Provisioning every one of those jobs for its peak is wasteful; provisioning for the average causes lag during surges. And in our platform a scaling action is not free: by default it means taking a savepoint, stopping the job gracefully, and restarting it at the new size, which for a large stateful job can take minutes. That leaves a genuinely hard question: how do you give each job the resources it needs, when it needs them, without a human in the loop and without breaking anything? The first autoscaler: watching from outside Our first answer, built around 2019, was an autoscaler shaped like a stream-processing job. It ran on M ## New in Confluent Cloud and WarpStream: Evolving the Data Streaming Platform for AI, Scale, and Control DevFeed: [New in Confluent Cloud and WarpStream: Evolving the Data Streaming Platform for AI, Scale, and Control]() Original publisher: [Read original article]() Author: Mike Agnich Published: 2026-08-18T14:00:11Z Content type: article Language: en Sources: [Confluent: Data in motion]() Topics: [Streaming](), [Kafka](), [Cloud](), [event driven](), [Transactions]() Tags: [confluent](), [confluent-cloud](), [debug](), [kafka](), [networking](), [security](), [stream-processing](), [streaming](), [transactions]() ### AI overview Confluent describes more than 70 new features for its data streaming platform, including new connectors, networking improvements, AI capabilities for event-driven agents, real-time machine learning in streaming pipelines, and AI-assisted development tools. The article also discusses broader Kafka workload support, including stream processing, transactions, exactly-once processing, queues, debugging, and Python client support for Queues for Kafka. ### Source excerpt Accelerate enterprise streaming and AI workloads with new product updates across Kora, connectors, Flink, Tableflow, AI anomaly detection and forecasting, security enhancements, and Warpstream. ## New in Confluent Intelligence and AI Tools: Making Agents Native to the Stream, Expanded Model Support, New Agent Skills, and Copilot DevFeed: [New in Confluent Intelligence and AI Tools: Making Agents Native to the Stream, Expanded Model Support, New Agent Skills, and Copilot]() Original publisher: [Read original article]() Author: Confluent Staff Published: 2026-08-18T14:00:10Z Content type: article Language: en Sources: [Confluent: Data in motion]() Topics: [Artificial Intelligence](), [apache-flink](), [Kafka](), [streaming-data-processing](), [real-time](), [MCP Server](), [Agent Skills](), [MCP](), [Inference](), [Google](), [ibm]() Tags: [ai](), [ai-ready-data](), [ai-tools](), [apache-flink](), [confluent](), [confluent-cloud](), [google](), [ibm](), [kafka](), [mcp](), [mcp-server](), [real-time](), [skills](), [streaming-data-processing](), [support](), [tools]() ### AI overview Confluent announces updates to Confluent Intelligence and related AI tools for building production AI systems on Apache Kafka and Apache Flink. The release adds expanded time-series model support, a generally available Real-Time Context Engine, updates to the fully managed MCP Server, Agent Skills for AI coding assistants, and Confluent Copilot. The features are intended to provide agents and applications with fresh business context and governed access to live data and infrastructure. ### Source excerpt Explore new AI features and AI tools: support for IBM Granite Time Series models and TimesFM models (EA), enhanced Real-Time Context Engine experience, new Agent Skills, Confluent Copilot ## Confluent Cloud for Apache Flink: Engine for Mission-Critical, Real-Time Operational Systems and dbt/SQL-Native Home for Data Science and AI DevFeed: [Confluent Cloud for Apache Flink: Engine for Mission-Critical, Real-Time Operational Systems and dbt/SQL-Native Home for Data Science and AI]() Original publisher: [Read original article]() Author: Yashwanth Dasari Published: 2026-08-18T02:20:00Z Content type: release Language: en Sources: [Confluent: Data in motion]() Topics: [Confluent Cloud](), [apache-flink](), [stream-processing](), [Kafka](), [AI, ML & Data Engineering](), [Cloud](), [SDKs]() Tags: [ai](), [announcement](), [apache-flink](), [api](), [batch](), [confluent-cloud](), [data-science](), [developers](), [kafka](), [real-time]() ### AI overview Confluent announces an evolution of Confluent Cloud for Apache Flink that unifies mission-critical real-time operations with analytics and AI workflows. The release adds a serverless, co-designed Kafka and Flink platform, including the generally available Flink Table API in Java for code-first development. ### Source excerpt Flink now acts as a robust engine for developers through Table API, UDFs, and PTFs while offering a SQL-native, dbt-integrated platform for data science and AI teams. ## Agent Night demo recap: Brian Douglas's self-healing Pokémon agent DevFeed: [Agent Night demo recap: Brian Douglas's self-healing Pokémon agent]() Original publisher: [Read original article]() Author: WorkOS Published: 2026-08-17T18:34:12Z Content type: news Language: en Sources: [WorkOS Blog]() Topics: [Demo](), [observability ai agents](), [telemetry](), [Claude Code](), [Python](), [DuckDB](), [Kafka](), [Code](), [Open Source]() Tags: [agent](), [claude-code](), [code](), [coding](), [demo](), [duckdb](), [kafka](), [open-source](), [python](), [self-healing](), [telemetry](), [traces]() ### AI overview A recap of Brian Douglas's Agent Night demo shows how Paper Compute ran Pokémon as a self-healing agent for six months. The setup uses Claude Code, a headless Python emulator, and traces from each session and turn. The article explains how labeled traces and per-session reflections can make the agent's history searchable and useful for improvement. ### Source excerpt Brian Douglas of Paper Compute closed Agent Night with six months of running Pokémon as a self-healing agent: traces, anomalies, dreams, and inceptions. ## Data Engineering Weekly #283 DevFeed: [Data Engineering Weekly #283]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-17T02:59:40Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [AI, ML & Data Engineering](), [Data Quality](), [Microservice](), [Apache Spark](), [Kafka](), [Feature Engineering](), [audit trail](), [Protocol (disambiguation)]() Tags: [ai](), [anthropic](), [article](), [data-engineering](), [data-quality](), [engineering](), [feature-engineering](), [kafka](), [newsletter](), [pipelines](), [services](), [testing]() ### AI overview Data Engineering Weekly #283 is a newsletter covering data platform fundamentals, multiagent system coordination, payments platform data contracts, financial data quality, declarative data engineering, and cost-efficient export workloads. ### Source excerpt The Weekly Data Engineering Newsletter ## How Neuron Systems Served 2.3 Million Fans Across 104 World Cup Matches with AI on Confluent DevFeed: [How Neuron Systems Served 2.3 Million Fans Across 104 World Cup Matches with AI on Confluent]() Original publisher: [Read original article]() Author: Shalini Ananda, PhD Published: 2026-08-14T06:48:05Z Content type: article Language: en Sources: [Confluent: Data in motion]() Topics: [Artificial Intelligence](), [Kafka](), [Streaming](), [real-time](), [observability](), [systems]() Tags: [agents](), [ai](), [architecture](), [confluent](), [kafka](), [observability](), [production](), [real-time](), [scale](), [solutions](), [streaming]() ### AI overview Neuron Systems used six agents and Confluent's Data Streaming Platform to provide multilingual live commentary for all 104 FIFA World Cup 2026 matches. The platform processed 42.1 million production events, maintained a median glass-to-glass latency of 42 milliseconds, and supported millions of fans while keeping commentary synchronized. ### Source excerpt Every one of the 42.1 million production events flowed through Confluent's Data Streaming Platform and that single decision is what let a small team serve a global tournament without the commentary ever falling out of sync. ## Multi-region high availability for Kafka workloads with a single Stretch Cluster DevFeed: [Multi-region high availability for Kafka workloads with a single Stretch Cluster]() Original publisher: [Read original article]() Author: David Yu Published: 2026-08-11T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Kafka](), [Kubernetes](), [Availability](), [Replication](), [Raft](), [Resilience](), [stream-processing]() Tags: [clusters](), [k8s](), [kafka](), [raft](), [release](), [replication](), [resilience](), [stream-processing]() ### AI overview Redpanda Operator 26.2 introduces generally available Stretch Clusters, allowing one logical Redpanda cluster to span multiple Kubernetes clusters and regions. The release provides synchronous replication with Raft-based automatic failover, safer broker rolling restarts, Redpanda Connect pipelines as Kubernetes resources, and Gateway API support for Redpanda Console. ### Source excerpt Redpanda Operator 26.2 brings GA Stretch Clusters for multi-region replication, Redpanda Connect pipelines as K8s resources, Gateway API support, and safer rolling restarts. ## v20.2.3 Tentacle released DevFeed: [v20.2.3 Tentacle released]() Original publisher: [Read original article]() Author: Yuri Weinstein Published: 2026-08-05T00:00:00Z Content type: release Language: en Sources: [Ceph Blog]() Topics: [Deployment](), [Server](), [Hardware](), [TLS (Transport Layer Security)](), [Kafka](), [Grafana Cloud Metrics](), [Monitoring](), [Command-line interface](), [IO]() Tags: [2026](), [authentication](), [blog-post](), [cli](), [en-article](), [en-blog-post](), [hardware](), [io](), [kafka](), [metrics](), [monitoring](), [release](), [storage](), [tentacle](), [tls]() ### AI overview Ceph v20.2.3, the third minor release in the Tentacle series, is a recommended update containing fixes and improvements across metadata servers, object storage daemons, RADOS Gateway, client libraries, the dashboard, ceph-volume, and NVMe-oF. Changes include storage reliability fixes, TLS 1.3 cipher customization, encrypted object copying, Kafka notification authentication, hardware monitoring, CSV host uploads, and role-management updates. ### Source excerpt This is the third minor release in the Tentacle series. We recommend that all users update to this release. Release Date ¶ August 5, 2026 Notable Changes ¶ MDS (Metadata Server) ¶ Fixed a crash and shutdown hang that could occur when ephemeral pins were active and max_mds was set to 0. Session reclaim could miss blocklisting an old session; this is now handled correctly. Fixed a case where scrub was unable to identify damage from an irreparable hard link. OSD (Object Storage Daemon) ¶ BlueStore: Fixed a BlueFS WAL envelope-mode bug that caused write buffer misalignment. Erasure Coding: Fixed truncate+write planning for EC shard sizes. Fixed a condition that prevented rolling forward of PG log entries. Prevented OSDMap::check_health() from asserting when new OSDs are found in a subtree. Scrub: "repairing" scrubs -- deep scrubs re-run specifically to fix a previously detected inconsistency -- are no longer blocked by the configured allowed scrub hours/days; they can now run at any time. RGW (RADOS Gateway) ¶ Added the ssl_ciphersuites option for the Beast front-end to support TLS 1.3 cipher customization. Fixed PutObject's canned_acl comparison bug affecting BlockPublicAcls. Fixed crashes occurring on realm reload. Implemented CopyObject support for encrypted objects. Added mTLS client-certificate authentication support for Kafka notifications. Fixed removal of delete markers in lifecycle processing. RADOS / librados / neorados ¶ neorados: Fixed an infinite trim loop on empty data log shards. Fixed cancellation-slot cleanup in librados/asio's associated executor. Dashboard ¶ Added a hardware-monitoring dashboard backed by node-proxy metrics. Fixed a bind-address regression caused by CherryPy isolation. Added support for adding hosts via CSV upload. NVMeoF: multiple CLI and UI backports (EC pool support, IO statistics, role management). Object: The global RGW Roles tab has been removed. Role management is now scoped under RGW Accounts, where roles can be listed, created, ## From Batch Snapshots to Near-Real-Time Data DevFeed: [From Batch Snapshots to Near-Real-Time Data]() Original publisher: [Read original article]() Author: Doximity Published: 2026-08-04T16:29:00Z Content type: article Language: en Sources: [Doximity]() Topics: [data](), [Kafka](), [Streaming](), [MySQL](), [Amazon Aurora]() Tags: [amazon-aurora](), [architecture](), [data](), [kafka](), [mysql](), [pipelines](), [streaming]() ### AI overview Doximity describes combining batch database snapshots with Change Data Capture to make data available in minutes while retaining batch-based consistency and recovery guarantees. The design uses Kafka and includes a trusted snapshot, metadata-preserving routing, a base-plus-delta view, and synthetic cascade deletes. In a 12-day measurement window, 95% of sampled events reached the queryable intermediate layer within eight minutes of publication to Kafka. ### Source excerpt Change Data Capture (CDC) is often presented as a straightforward pipeline: read a database transaction log, publish each change, and apply those changes to another system. That description is accurate, but it leaves out many of the decisions that determine whether the resulting data can be trusted. At Doximity, we already had a batch pipeline that periodically copied snapshots of application databases into our data warehouse. Those snapshots were reliable, but their freshness was measured in hours. We introduced CDC to make changes available in minutes so downstream transformations and operational analytics would not have to wait for the next batch snapshot. We continued using the batch pipeline for the consistency and recovery guarantees it already provided. Over a 12-day measurement window, 95% of events from a stratified sample of active tables reached the queryable intermediate layer within eight minutes of being published to Kafka. The most interesting parts of the project were not the connections from a source database to Kafka or from Kafka to Snowflake, but four questions we had to answer around them: How could we reuse our existing, transactionally consistent batch snapshots as an on-demand starting point for CDC, without reprocessing every existing row? How could we onboard new tables and absorb schema changes from many source databases across our products without growing operational overhead for each one? How could new changes become queryable without waiting for the warehouse to merge them into place? How could we handle cascading child deletes that MySQL performs but never emits as individual binary-log events? Our answers are the four design decisions in this article: a trusted batch snapshot, metadata-preserving routing, a base-plus-delta view, and synthetic cascade deletes. Together, they turned a stream of row changes into a system we could bootstrap, scale, validate, and recover. The sections that follow explain the tradeoffs and guardrails so rea ## What's new in clickhousectl v0.4.0 DevFeed: [What's new in clickhousectl v0.4.0]() Original publisher: [Read original article]() Author: Al Brown Published: 2026-07-31T11:44:09Z Content type: release Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Command-line interface](), [autoscaling](), [Cloud](), [Kafka](), [MySQL](), [API keys]() Tags: [api-keys](), [autoscaling](), [cli](), [clickhouse](), [cloud](), [cloud-services](), [kafka](), [mysql](), [telemetry]() ### AI overview clickhousectl v0.4.0 introduces horizontal autoscaling for ClickHouse Cloud services, beta ClickPipe schema discovery for Kafka and Kinesis sources, more controls for object storage and MySQL/Postgres ingestion, global OAuth token storage, safer local server lifecycle behavior, and opt-out anonymous usage telemetry. ### Source excerpt clickhousectl v0.4.0 adds horizontal autoscaling for Cloud services, ClickPipe schema discovery, finer control over ClickPipe ingestion, and opt-out anonymous usage telemetry. ## Announcing Confluent Platform 8.3: Powerful Apache Flink® SQL operations, Easier KRaft Migrations, Expanded Monitoring and more. DevFeed: [Announcing Confluent Platform 8.3: Powerful Apache Flink® SQL operations, Easier KRaft Migrations, Expanded Monitoring and more.]() Original publisher: [Read original article]() Author: Premika Srinivasan Published: 2026-07-29T15:00:10Z Content type: release Language: en Sources: [Confluent: Data in motion]() Topics: [apache-flink](), [Kafka](), [Monitoring](), [data-governance](), [Model Context Protocol](), [Claude Code](), [codex]() Tags: [apache-flink](), [apache-kafka](), [claude-code](), [codex](), [confluent](), [confluent-platform](), [data-governance](), [mcp](), [mcp-server](), [monitoring]() ### AI overview Confluent Platform 8.3.0, built on Apache Kafka 4.3.0, adds simplified Apache Flink SQL operations, expanded monitoring through Unified Stream Manager, easier KRaft migration, and structured governance for data in motion. The release also introduces an MCP server for Confluent Platform for Apache Flink, allowing AI agents such as Claude Code and Codex to inspect and manage Flink resources through existing security controls. ### Source excerpt Announcing Confluent Platform 8.3: Powerful Apache Flink® SQL operations, Easier KRaft Migrations, Expanded Monitoring and more ## How An Incident Agent Would Handle A Port Incident DevFeed: [How An Incident Agent Would Handle A Port Incident]() Original publisher: [Read original article]() Author: Zohar Einy Published: 2026-07-22T11:10:11Z Content type: article Language: en Sources: [Developer Experience & Platform Engineering Blog | Port]() Topics: [incident](), [Artificial Intelligence](), [Automation](), [Kafka](), [DevOps](), [Slack](), [GitHub]() Tags: [agent](), [ai](), [article](), [automation](), [autonomous](), [building](), [config](), [customer](), [devops](), [github](), [incident](), [kafka](), [on-call](), [production](), [scale](), [slack](), [ui](), [work]() ### AI overview The article examines a real incident at Port in which three teams independently investigated the same production problem after a customer generated 1.7 million automation runs in 90 minutes. It describes duplicated investigation, ineffective service restarts, fragmented communication, and slow manual coordination, then considers how an incident agent could connect signals and support autonomous resolution. ### Source excerpt Explore how AI could have handled a real incident at Port, improving response times, automation, and decision-making in production. [Next page]()