# Kafka Published articles for Kafka. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Webhooks to Kafka: How to Receive Events and Publish Them to Topics DevFeed: [Webhooks to Kafka: How to Receive Events and Publish Them to Topics]() Original publisher: [Read original article]() Author: Seán Whelan Published: 2026-11-06T12:27:07Z Content type: article Language: en Sources: [Estuary]() Topics: [Apache Kafka](), [Event-Streaming](), [Messaging](), [Publish-subscribe pattern]() Tags: [aws](), [aws-lambda](), [compare](), [events](), [gateway](), [how-to](), [http](), [kafka](), [managed](), [pipeline](), [retries](), [webhooks]() ### AI overview Webhook providers cannot post directly to Kafka, so an HTTPS receiver must verify each request and publish it. The excerpt compares a custom service or HTTP gateway, API Gateway with Lambda, Kafka Connect, and a managed pipeline; the choice depends on operational control, AWS or Connect infrastructure, and who hosts the endpoint. For a direct receiver, verify signatures over raw bytes, produce with the delivery ID as key, and return success only after Kafka acknowledges; provider retries make delivery at least once, so redeliveries can become duplicate messages. Providers have different deadlines and retry behavior--Shopify allows five seconds, while GitHub does not automatically redeliver. Estuary instead acknowledges after committing requests to a durable collection, then materializes them to Kafka independently, allowing capture to continue during a broker outage. Its excerpted limits include a 64 MiB request cap, message truncation thresholds, and signature-header formats that do not support GitHub's or Stripe's built-in formats through the custom setting. The text also covers Kafka size and authentication requirements, recovery, deduplication, and alternate routes. ### Source excerpt Send webhooks to Kafka with a webhook source connector, an HTTP gateway, AWS Lambda or a managed pipeline. Compare signatures, retries and duplicates. ## S3 to Kafka: How to Turn New S3 Files into Kafka Messages DevFeed: [S3 to Kafka: How to Turn New S3 Files into Kafka Messages]() Original publisher: [Read original article]() Author: Seán Whelan Published: 2026-11-06T12:27:02Z Content type: article Language: en Sources: [Estuary]() Topics: [Apache Kafka](), [Confluent Cloud](), [Amazon MSK](), [parquet]() Tags: [aws](), [aws-lambda](), [compare](), [files](), [how-to](), [kafka](), [managed](), [messages](), [ordering](), [pipeline](), [s3](), [setup](), [source](), [spark]() ### AI overview S3 files can become Kafka messages through Kafka Connect, S3 event notifications with Lambda, Spark or Flink, or a managed pipeline. The main differences are how each discovers and tracks files, supported parsing and transformation, and setup; most approaches deliver at least once, so consumers may need to handle duplicates. Listing tools can miss late-arriving keys when they track only the last key, while re-reading modified files can produce duplicates. The guide details S3 and Kafka permissions, format and message-size constraints, and operational tradeoffs. Estuary's described route lists a bucket prefix, parses new files into a durable collection, then materializes records to Kafka; it runs every five minutes by default, delivers at least once, and has specific Parquet size, field truncation, schema-subject, authentication, and deletion limitations. Event-triggered Lambda is positioned for a modest number of uploads needing quick reaction, and Spark or Flink for files requiring transformation. ### Source excerpt Move S3 data into Kafka with an S3 source connector, AWS Lambda, Spark or a managed pipeline. Compare file ordering, formats, duplicates and setup. ## Kafka to ClickHouse: Streaming Ingestion, Duplicates & Tombstones Guide DevFeed: [Kafka to ClickHouse: Streaming Ingestion, Duplicates & Tombstones Guide]() Original publisher: [Read original article]() Author: Seán Whelan Published: 2026-11-06T12:26:41Z Content type: article Language: en Sources: [Estuary]() Topics: [Apache Kafka](), [Event-Streaming](), [self-hosted databases vs managed services](), [table-partitioning](), [schema-evolution](), [Amazon MSK](), [Confluent Cloud](), [Apache Iceberg]() Tags: [clickhouse](), [compare](), [connect](), [delivery](), [engine](), [guarantees](), [guide](), [kafka](), [stream](), [streaming](), [table]() ### AI overview Kafka ingestion into ClickHouse is generally at least once, so replayed messages can create duplicates; reliable designs need batching and tables suited to event or keyed data. The article compares the Kafka table engine, ClickPipes, the Kafka Connect sink, and managed pipelines: the engine runs consumers on ClickHouse servers, ClickPipes is recommended for ClickHouse Cloud, and the sink fits teams already running Kafka Connect. For event streams, store partition and offset in a ReplacingMergeTree sorting key; keyed topics with tombstones need ReplacingMergeTree keyed by message key and queries with FINAL. Tombstone support differs: ClickPipes can delete rows in exactly-once mode or mark them deleted, while the Connect sink lists deletes as unsupported and the table engine's documentation does not describe null-value messages. Retention limits recovery, and compaction can remove tombstones after the configured retention period. Large, infrequent inserts reduce part pressure; monitor lag and active parts. The article also describes Estuary capture and materialization, which keeps Kafka progress in its own checkpoints and writes to ClickHouse on a configurable schedule. ### Source excerpt Stream Kafka to ClickHouse with the Kafka table engine, ClickPipes or Kafka Connect. Compare 4 methods, delivery guarantees and duplicate handling. ## Graphiant Named Confluent 2027 Data Streaming Startup of the Year for the Americas DevFeed: [Graphiant Named Confluent 2027 Data Streaming Startup of the Year for the Americas]() Original publisher: [Read original article]() Author: Tim Graczewski Published: 2026-11-03T21:29:59Z Content type: news Language: en Sources: [Confluent: Data in motion]() Topics: [Confluent Cloud](), [Apache Kafka](), [streaming data analytics](), [observability](), [Apache Flink]() Tags: [2020](), [2025](), [about](), [adoption](), [amer](), [americas](), [apache](), [apache-kafka](), [confluent](), [data](), [data-streaming](), [kafka](), [network](), [network-observability](), [observability](), [real-time](), [real-time-data](), [real-time-data-streaming](), [startup](), [streaming](), [uses]() ### AI overview Graphiant, named Confluent's 2027 Americas Data Streaming Startup of the Year, uses Apache Kafka on Confluent Cloud to stream telemetry and performance metrics from network edge devices. The data supports network health monitoring, traffic analysis, alerting, and customer dashboards, with analytics flowing to Apache Pinot through StarTree and to TimescaleDB. Confluent Cloud handles variable workloads as the network grows, reducing the need for Graphiant to operate Kafka infrastructure. Graphiant currently manages its own Apache Flink environment and plans to evaluate serverless Apache Flink on Confluent Cloud, as well as adopt Schema Registry to strengthen data management. The excerpt also describes a possible longer-term Streamhouse architecture and potential AI-driven network operations; these are future possibilities, not stated current capabilities. ### Source excerpt Graphiant is Confluent's 2027 Americas Startup of the Year. Discover how it uses Apache Kafka and real-time data streaming for network observability. ## Introducing queue consumption mode for Apache Kafka in AWS Lambda DevFeed: [Introducing queue consumption mode for Apache Kafka in AWS Lambda]() Original publisher: [Read original article]() Author: Vaibhav Jain Published: 2026-11-02T21:23:46Z Content type: release Language: en Sources: [AWS Compute Blog]() Topics: [Apache Kafka](), [AWS Lambda](), [Amazon Web Services](), [Confluent Cloud]() Tags: [amazon-cloudwatch-logs](), [amazon-sns](), [amazon-sqs](), [announcements](), [apache-kafka](), [automatic](), [aws](), [aws-cdk](), [aws-lambda](), [blocking](), [broker](), [cluster](), [event-source](), [intermediate-200](), [kafka](), [parallelism](), [partition](), [queue](), [retries](), [scale](), [share](), [visibility]() ### AI overview AWS Lambda's Queue consumption mode uses Kafka Share Groups on Apache Kafka 4.2 or later to process records from a partition concurrently, removing partition count as the concurrency ceiling. Unlike Stream mode, which preserves per-partition ordering, Queue mode has no ordering guarantees and suits independent records. It uses per-record acknowledgment, visibility locks renewed during processing, and retries after failures; duplicate processing can occur, so functions should be idempotent. Queue mode is available on self-managed Kafka ESMs, including compatible managed Kafka offerings. On-demand scaling reaches five pollers per mapping (up to 5 MB/s or 50 concurrent invocations) and adjusts within three minutes; provisioned mode offers configurable poller capacity for higher throughput or stricter latency needs, subject to broker share-group limits. Queue mode does not support batching windows or starting-position parameters. ### Source excerpt AWS Lambda event source mappings now support queue consumption mode for Apache Kafka. Using Kafka Share Groups, you can scale consumer parallelism independently of partition count, with per-record acknowledgment, visibility timeouts, and automatic retries on self-managed Kafka 4.2 or later clusters. ## How we halved DNS build delay and cut memory use by 85% DevFeed: [How we halved DNS build delay and cut memory use by 85%]() Original publisher: [Read original article]() Author: Alex Fattouche Published: 2026-11-02T14:00:00Z Content type: article Language: en Sources: [Cloudflare Blog]() Topics: [Domain Name System (DNS)](), [scheduling](), [Latency](), [authoritative name server](), [rate-limiting]() Tags: [2](), [2022](), [3](), [about](), [account](), [added](), [aggregate](), [build](), [builds](), [changes](), [deep-dive](), [delay](), [dns](), [full](), [here](), [kafka](), [memory](), [planner](), [quicksilver](), [rebuild](), [records](), [speed-reliability](), [used](), [with]() ### AI overview A targeted build planner and unified scheduler reduced unnecessary full-zone DNS rebuilds while preserving correctness. Events identify changed settings and records; the planner maps them to the smallest safe work--specific names, names plus metadata, apex plus metadata, purge, or full-zone build. Unknown or broad changes still trigger full builds, and large zones avoid escalation to a full build because it cannot finish. Before writing, the builder checks fresh database state, reconstructs shared metadata for zones with the same name, and verifies replica lag. The scheduler merges queued work per zone while retaining Kafka acknowledgments through retries, so uncertain or unfinished work is replayed rather than lost. Equivalence tests and a production-stream mirror compare output before rollout. In a 24-hour measurement, 92.95% of builds avoided full-zone work and full-build share fell about 40%. Peak working-set memory dropped 85% (79 to 12 GiB); the largest p99 delay spike fell from 14.6 to 1.21 minutes. Shorter rebuild intervals and aggregator waits helped halve median completion delay from about 7.7 to 3.3 seconds. This measures writing to Quicksilver, not its subsequent network ### Source excerpt Zone setting changes used to rebuild entire DNS zones, even ones with tens of millions of records. Here is how we built a planner that picks the smallest safe build, cutting full builds by 40%, peak memory by 85%, and halving build delay. ## Milvus Source Connector for Debezium: Enabling Event-Driven Vector Architectures DevFeed: [Milvus Source Connector for Debezium: Enabling Event-Driven Vector Architectures]() Original publisher: [Read original article]() Author: Kartik Angiras Published: 2026-11-02T00:00:00Z Content type: article Language: en Sources: [Debezium]() Topics: [Debezium Connector](), [Apache Kafka](), [Publish-subscribe pattern](), [event-sourcing](), [Embeddings](), [hybrid-search]() Tags: [2026](), [architectures](), [debezium](), [elasticsearch](), [event](), [event-driven](), [kafka](), [open-source](), [postgres](), [redis](), [source](), [stream](), [vector](), [vector-database]() ### AI overview The Debezium 3.7 Milvus source connector reads Milvus 2.5's Kafka write stream, uses etcd checkpoints and Milvus gRPC for snapshots and schemas, and emits inserts and deletes as Debezium events with typed vector fields. Timetick watermarks restore commit order across virtual channels, and Kafka Connect sinks can deliver changes to systems such as Postgres, Elasticsearch, or S3. The connector requires a self-hosted deployment using Kafka as its broker; Milvus Lite, Pulsar, RocksMQ, and Milvus 2.6 are unsupported. It is incubating: its default checkpoint key can miss changes at snapshot handoff, and a crash while events await ordering can lose them. It uses at-least-once snapshot delivery, so idempotent sinks are important; deletes have only primary-key before images, and upserts appear as delete/insert pairs. ### Source excerpt Google Summer of Code 2026 with the Debezium community was the best learning experience I have had as an engineer. Over one summer I built a Kafka Connect source connector from scratch for a database with no documented way to read its changes. Most of the work was taking the database apart to find out where those changes live, writing the design down before building any of it, and carrying each piece through review with the Debezium maintainers. The connector shipped in Debezium 3.7. What came out of that summer is a Debezium source connector that turns Milvus, the open source vector database, into a first-class source of change events. I built it because Milvus has no change stream of its own. If you need its inserts and deletes somewhere else (Elasticsearch for keyword search, Redis as a cache, Postgres for reporting), you either write to both systems from the application and drift apart on the first partial failure, or you run a batch export that cannot see a delete. The connector reads the write stream Milvus already publishes to its message queue, puts the events back in commit order, and emits each insert and delete as a normal Debezium change event with the vector as a typed field. From there any Kafka Connect sink is a Milvus integration, and the service that writes to Milvus never has to know. This post covers how it works and what you can build once Milvus is one. Read more... ## Spring News Roundup: First Release Candidates of Boot, Framework, Data, Security, Modulith, AI DevFeed: [Spring News Roundup: First Release Candidates of Boot, Framework, Data, Security, Modulith, AI]() Original publisher: [Read original article]() Author: Michael Redlich Published: 2026-10-31T13:30:00Z Content type: news Language: en Sources: [InfoQ]() Topics: [Spring Framework](), [Spring Boot](), [Spring AI](), [backend frameworks](), [Apache Kafka](), [Security](), [Redis]() Tags: [2026](), [ai](), [amqp](), [apache](), [apache-kafka](), [architecture-design](), [batch](), [boot](), [cloud](), [data](), [development](), [docs](), [first](), [framework](), [graphql](), [integration](), [java](), [kafka](), [news](), [news-roundup](), [release](), [security](), [spring](), [spring-ai](), [spring-batch](), [spring-boot](), [spring-cloud](), [spring-data](), [spring-framework](), [spring-integration](), [spring-modulith](), [spring-news-roundup-oct19-2026](), [spring-security](), [spring-web-services]() ### AI overview The week of October 19, 2026 brought first release candidates across 17 Spring projects. The excerpt highlights practical changes: Boot 4.2.0 improves Uber-Jar lookup and switches its builder to Ubuntu Resolute; Framework 7.1.0 exposes retry state to listeners; Cloud 2026.0 folds Spring Cloud Bus into Spring Cloud Stream; and Modulith 2.2 distinguishes abandoned from failed event publications and completes publications within the listener transaction by default. Other examples include Spring AI replacing the deprecated `ChatMemory` interface with `SessionService`, Batch adding `@JobParameter` and `PageableItemReader`, and Web Services supporting policy documents on generated WSDL bindings. The roundup also lists bug fixes, documentation and dependency updates, and project-specific changes across Security, GraphQL, Session, Integration, HATEOAS, REST Docs, AMQP, Kafka, and LDAP. These are release candidates; the excerpt does not establish final release status. ### Source excerpt There was a flurry of activity during the week of October 19th, 2026, highlighting: first release candidates of: Spring Boot, Spring Framework, Spring Data, Spring Cloud, Spring Security, Spring for GraphQL, Spring Session, Spring Integration, Spring HATEOS, Spring Modulith, Spring REST Docs, Spring AI, Spring Batch, Spring AMQP, Spring for Apache Kafka, Spring LDAP and Spring Web Services. By Michael Redlich ## Introducing Apache Kafka 4.4.0 DevFeed: [Introducing Apache Kafka 4.4.0]() Original publisher: [Read original article]() Author: Omnia Ibrahim Published: 2026-10-29T18:34:18Z Content type: release Language: en Sources: [Confluent: Data in motion]() Topics: [Apache Kafka](), [project release]() Tags: [announce](), [apache](), [apache-kafka](), [features](), [improvements](), [kafka](), [new-features](), [release]() ### AI overview Apache Kafka 4.4.0 adds features across brokers, clients, Streams, Connect, and command-line tools, with 25 KIPs and 963 commits since 4.3.0. Share groups gain dead-letter queues once share.version is upgraded to 2, and Kafka Streams gains opt-in transactional state stores that preserve committed state on errors while replaying only writes since the last successful commit. Clients can opt into asynchronous bootstrap DNS resolution with bootstrap.resolve.timeout.ms; the default remains synchronous, and Kafka Streams and Connect do not yet support the asynchronous mode. Kafka Connect can disable automatic creation of internal topics, in which case startup fails if required topics are missing; a new tool supports manual creation. Other changes include controller unregistration, broker and client API version inspection through kafka-cluster.sh, additional Linux I/O metrics, and Streams topology descriptions and custom task assignors. A compatibility requirement changes: a broker must be at least as new as the kafka-storage version that formatted its directory. The article points readers to upgrade documentation for detailed steps. ### Source excerpt The Apache Software Foundation is proud to announce the release of Apache Kafka® 4.4.0. This release contains many new features and improvements. ## ClickHouse + dbt: Building a GitOps workflow at Twilio DevFeed: [ClickHouse + dbt: Building a GitOps workflow at Twilio]() Original publisher: [Read original article]() Author: Daniel Poppy Published: 2026-10-27T18:47:00Z Content type: news Language: en Sources: [dbt Labs]() Topics: [GitOps](), [data-engineering](), [CI/CD](), [Apache Kafka](), [GitHub](), [elasticsearch](), [Service fabric]() Tags: [2026](), [clickhouse](), [dbt](), [elasticsearch](), [gitops](), [kafka](), [learn](), [pipeline](), [spark](), [twilio]() ### AI overview Twilio consolidated a Voice Insights pipeline that had used Kafka for ingestion, Spark Streaming for processing, and Elasticsearch for serving into ClickHouse, using materialized views for streaming transformations. Its workflow progressed from shared DDL documents to Git version control and a migration tool, then to dbt with the open-source dbt-clickhouse adapter for transformations, tests, and lineage. Twilio split its monolithic dbt project into product-area projects with shared logic, orchestrated by Argo Workflows. Engineers use real ClickHouse containers for local and CI integration tests and can deploy branch-specific previews against a slice of production data. The change replaced manual production edits with audited, version-controlled schemas and enabled consistent redeployment across regions. The session also reported that the dbt v1 adapter remained supported, v2 was in public beta and nearing feature parity, and ClickHouse support in the dbt platform was in private beta. ### Source excerpt Twilio replaced Kafka, Spark, and Elasticsearch with one ClickHouse pipeline governed by dbt and GitOps. ## Uber Moves Beyond CPU Cores with Performance-Aware Scheduling for Stateful Workloads DevFeed: [Uber Moves Beyond CPU Cores with Performance-Aware Scheduling for Stateful Workloads]() Original publisher: [Read original article]() Author: Craig Risi Published: 2026-10-27T12:00:00Z Content type: article Language: en Sources: [InfoQ]() Topics: [scheduling](), [Stateful](), [Distributed Systems & Parallel Computing](), [resource orchestration](), [Load Balancing]() Tags: [1-4](), [abstraction](), [ai](), [ai-ml-data-engineering](), [apache-kafka](), [arm](), [automation-in-the-cloud-and-management-at-scale](), [aws](), [balancing](), [benchmarks](), [capacity](), [cloud](), [compute](), [core](), [cpu](), [cpu-time](), [data-centres](), [devops](), [fleet](), [kafka](), [kubernetes](), [make](), [ml-data-engineering](), [news](), [performance](), [scheduling](), [server](), [state-machine](), [stateful](), [uber](), [uber-cpu-stateful-workloads](), [units]() ### AI overview Uber's Normalized Compute Units (NCUs) let Odin allocate comparable compute capacity across servers whose CPU performance varies substantially. Odin assigns each server type a performance factor, initially measured during hardware qualification and refined with production workload data, then uses it to translate workload requirements into different numbers of physical cores. This addresses a reliability risk for stateful workloads, which cannot readily shift requests away from a slower server; stateless services can more often use load balancing. Across about 1.4 million automatically scaled cores, Kafka's utilization gap between server types fell from roughly eight to three percentage points, and total Kafka CPU allocation dropped about 8%. Uber contrasts NCUs with Kubernetes CPU requests in cores or millicores, which represent CPU time without expressing relative processor performance. ### Source excerpt Uber has introduced Normalized Compute Units (NCUs) to make CPU allocation more consistent across its highly heterogeneous server fleet. By Craig Risi ## The Hidden Cost of Real-Time Data: Why Faster Pipelines Don't Always Deliver Better Decisions DevFeed: [The Hidden Cost of Real-Time Data: Why Faster Pipelines Don't Always Deliver Better Decisions]() Original publisher: [Read original article]() Author: FNU Aditya Ranjan Published: 2026-10-24T01:40:37Z Content type: article Language: en Sources: [HackerNoon]() Topics: [Event-Streaming](), [data-engineering](), [Data pipelines](), [streaming data analytics](), [software-architecture](), [Apache Kafka](), [data observability](), [Cloud Dataflow]() Tags: [apache-flink](), [apache-kafka](), [api](), [architecture](), [business](), [business-value](), [cdc](), [cost](), [costs](), [data](), [data-architecture](), [data-engineering](), [data-freshness](), [data-pipelines](), [data-quality](), [distributed-systems](), [kafka](), [latency](), [pipelines](), [real-time](), [real-time-data](), [real-time-data-pipelines](), [recovery](), [stream-processing](), [streaming]() ### AI overview Fast event processing does not ensure timely, trustworthy information: end-to-end freshness, completeness, corrections, recovery, and operating cost all matter. Measure freshness from event time to consumer availability, since processing-time metrics can miss upstream and downstream delays. Exactly-once guarantees depend on framework, source, state, sink, and transaction boundaries; external actions may still need stable event IDs, idempotent writes, deduplication, or reconciliation. Late events trade immediate stability for completeness, so distinguish preliminary from finalized results and define correction windows. Stateful processing adds storage and recovery costs, requiring business-driven retention. The article proposes separate availability, completeness, correction, and recovery objectives, and evaluating designs under delayed, repeated, and failed-event scenarios using end-to-end latency, corrections, recovery, and cost. ### Source excerpt Explore the hidden costs of real-time data pipelines and learn why latency, data freshness, correctness, and recovery matter in Kafka and streaming architecture ## Event-Driven CI at Enterprise Scale: Decoupling Webhook Ingestion with Kafka DevFeed: [Event-Driven CI at Enterprise Scale: Decoupling Webhook Ingestion with Kafka]() Original publisher: [Read original article]() Author: Vidhi Shah Published: 2026-10-23T21:08:58Z Content type: article Language: en Sources: [HackerNoon]() Topics: [Apache Kafka](), [CI/CD](), [event driven](), [Messaging](), [Publish-subscribe pattern](), [Chaos Engineering]() Tags: [architecture](), [availability](), [ci](), [cicd](), [decoupling](), [devops](), [distributed-systems](), [enterprise](), [event](), [event-driven](), [event-driven-architecture](), [github](), [kafka](), [partition](), [production](), [queues](), [scale](), [software-architecture](), [stateless](), [webhook]() ### AI overview The article's main finding is that adding runners does not absorb synchronized webhook bursts; separating ingestion from build scheduling does. Its design uses stateless receivers to verify GitHub signatures and wait for durable Kafka acknowledgment, then partitions events by organization, repository, and ref so each ref stays ordered while different refs run in parallel. Consumers schedule builds with idempotent job keys, deduplicate delivery IDs, recheck authorization at execution, and commit offsets only after processing; a dead-letter queue, consumer-lag monitoring, backpressure, and reconciliation address failures and missed deliveries. The authors report 128 partitions for roughly 2,500 scheduling events per second. They caution that rebalances can halt affected partitions, so autoscaling should be gradual with cooldowns. This adds status delay and makes Kafka a critical dependency; the article says synchronous CI remains simpler and more economical for modest workloads. ### Source excerpt Decouple CI webhook ingestion with Kafka: stateless receivers, per-ref partition keys, idempotent consumers, dead-letter queues, and production lessons. ## Extending Your Quarkus Application with Apache Camel DevFeed: [Extending Your Quarkus Application with Apache Camel]() Original publisher: [Read original article]() Author: Gaelle Fournier Published: 2026-10-23T00:00:00Z Content type: tutorial Language: en Sources: [Quarkus - Supersonic Subatomic Java]() Topics: [Quarkus](), [Apache Kafka](), [REST API](), [Publish-subscribe pattern]() Tags: [apache-camel](), [api](), [csv](), [file](), [integration](), [kafka](), [quarkus](), [rest-api]() ### AI overview The excerpt shows how to extend a Quarkus order API with Camel Quarkus routes for three integration tasks: publish orders as JSON to Kafka, notify a loyalty API over HTTP with up to three retries using exponential backoff, and ingest CSV files into the existing repository while moving processed and failed files to separate directories. A `ProducerTemplate` call connects the existing REST endpoint to the Kafka and HTTP route; CSV ingestion runs in a separate file route. The example notes the retry policy applies to Kafka steps too, and exhausted retries are logged and handled while the order remains persisted. It presents Camel's shared route model as useful as integrations grow, while stating that Quarkus Reactive Messaging is enough for a single Kafka topic. ### Source excerpt Introduction You built a standard Quarkus application: REST endpoints, a service layer, a database. Then you are given a new requirement: "When an order is placed, send an event to Kafka so the analytics team can track it." A week later another one: "Also notify the partner loyalty system via their REST API and make sure to handle it properly when their API is down." And another one: "The partner is going to start sending us CSV order files."... Each of these requirements is an integration task, and Quarkus already gives you building blocks for all of them: Reactive Messaging for Kafka, the REST client with fault tolerance for the loyalty API, a scheduled job for file polling. But you could also hand all three to Apache Camel, which covers them with a single model. Retry with backoff, error policies, JSON and CSV conversion, file polling that moves handled files aside: each one is a built-in step you declare on a route, a line at a time, in the same vocabulary whichever system you are talking to. And it runs inside the application you already have. Camel Quarkus packages over 300 of Camel's components as Quarkus extensions, as natural to use as any other CDI bean. In this post we start from a working Quarkus REST application and progressively extend it with three integration capabilities using Camel. You'll see exactly what changes in your existing code and what stays untouched. The starting point Here's what we're starting with: a simple order management REST API. @Path("/api/orders") @Produces(MediaType.APPLICATION_JSON) @Consumes(MediaType.APPLICATION_JSON) public class OrderResource { @Inject OrderRepository repository; @GET public List list() { return repository.listAll(); } @POST public Response create(Order order) { repository.add(order); return Response.status(Response.Status.CREATED).entity(order).build(); } } Order is a POJO with four fields (id, customer, type, amount), a no-arg constructor and an all-args one. OrderRepository is an @ApplicationScop ## Native Support for Debezium in Apache Beam DevFeed: [Native Support for Debezium in Apache Beam]() Original publisher: [Read original article]() Author: Ilyas Ahsan Published: 2026-10-21T00:00:00Z Content type: tutorial Language: en Sources: [Debezium]() Topics: [Debezium Connector](), [Cloud Dataflow](), [BigQuery](), [Apache Kafka]() Tags: [apache-kafka](), [beam](), [bigquery](), [change-data-capture](), [debezium](), [kafka](), [mysql]() ### AI overview Apache Beam's DebeziumIO runs Debezium change capture inside a Beam pipeline, without requiring Kafka or Kafka Connect. The excerpt demonstrates a Java pipeline that reads MySQL changes, converts each event to a BigQuery row, and appends it using either the Direct runner or Dataflow. It requires Java 17, Gradle, Docker, and a Google Cloud project with BigQuery enabled; the destination table must exist beforehand. DebeziumIO is available in Beam 2.76.0, supports Debezium 3.5, and the excerpt lists MySQL, PostgreSQL, SQL Server, and Db2 as supported databases. The feature is described as experimental. A noted limitation is that its metadata omits binlog position and row number, which the example says are needed for reliable deduplication, as well as the database commit timestamp useful for debugging. ### Source excerpt Apache Beam has a built-in connector for Debezium called DebeziumIO. It lets you capture changes from your database and process them directly inside a Beam pipeline. You don't need to run Apache Kafka or Kafka Connect. In this post, we'll build an example pipeline that streams changes from MySQL into Google BigQuery. The complete example is available in the debezium-examples repository. Read more... ## How We Deploy Reliably at Razorpay DevFeed: [How We Deploy Reliably at Razorpay]() Original publisher: [Read original article]() Author: Anusha Musthaphak Published: 2026-10-20T08:20:57Z Content type: article Language: en Sources: [Razorpay]() Topics: [progressive delivery](), [Deployment Strategies](), [canary release](), [Production Engineering](), [CI/CD]() Tags: [ai](), [alerts](), [blast-radius](), [canary](), [canary-deployment](), [deploy](), [kafka](), [logs](), [on-call](), [payments](), [process](), [production](), [progressive-delivery](), [razorpay](), [reliability](), [risk](), [services](), [traffic](), [transactions](), [version]() ### AI overview Razorpay reduces deployment risk through progressive canaries that limit exposure, compare new and current versions, and automatically gate or roll back unhealthy changes. Web services use a default five-step traffic ladder from 1% to 15%, with two-hour analysis per step (longer for thin or bursty traffic); Kafka consumers get a parallel canary process that compares processing errors, lag, and latency. Rollouts are ordered by merchant criticality, keeping P0-P2 merchants on the current version until less critical traffic completes canary cycles. Canary checks draw eligible signals from live Prometheus alert rules, while Canary Sentinel compares logs during rollout and sends ambiguous results to a human. Deploy Sentinel performs six follow-up checks through 16 hours after deployment, combining logs, metrics, infrastructure health, and change-specific analysis; its weakest score determines the verdict. The article says overnight and multi-day bake periods and peak-aware rollout timing remain planned work because short observation windows can miss low-frequency or high-volume regressions. ### Source excerpt Most production incidents do not begin with a server dying. They begin with a deployment: dozens a day, across hundreds of services, each the moment risk enters the system. We run payments infrastructure at Razorpay. When a payments system degrades, transactions fail: merchants lose revenue, customers lose trust, and someone's checkout page spins at the worst possible moment. Reliability is not a feature we ship; it is the thing everything else stands on. A system in steady state rarely breaks on its own. So if you are serious about reliability, you invest where the risk actually enters: not in the servers, but in the process that changes them. This is what that investment looks like for us today: progressive canaries with more and smaller steps, canaries for Kafka consumers, rollouts ordered by merchant criticality, canary analysis derived from the same alerts that page on-call, and AI judges that read logs during and after the rollout. What reliability means at deployment time You cannot ship software with zero risk. Any process that promises that is either lying or shipping nothing. So the goal is not to prevent every bad deployment; it is to control three quantities: Blast radius. How much traffic, and whose traffic, a bad change touches before anyone notices. Time to detect. How quickly you find out the change is bad, and whether a machine finds out before a customer does. Time to recover. How quickly you get back to a known-good state once you know. Progressive delivery is the discipline built on those three levers: expose a change to a small slice of reality, measure it honestly against the current version, expand only while it stays healthy, and revert automatically the moment it does not. In practice, that is a canary deployment. Everything below is one of those three levers, pulled harder. Smaller steps, longer looks Every production deployment spins up the new version as a canary next to a baseline: a fresh copy of the currently running version, started a ## Production logging practices for traceability, incident response, and secret protection DevFeed: [Production logging practices for traceability, incident response, and secret protection]() Original publisher: [Read original article]() Author: Shalini Goyal Published: 2026-10-19T16:19:16Z Content type: tutorial Language: en Sources: [Shalini Goyal]() Topics: [Production Engineering](), [observability](), [tracing](), [OpenTelemetry](), [CI/CD]() Tags: [code](), [data-leak](), [incident](), [kafka](), [observability]() ### AI overview The article presents six common production logging mistakes and recommended practices for addressing them. It covers structured JSON logs, consistent fields, trace IDs across services and Kafka, appropriate severity levels, business milestones, secret masking, and periodic log review. It also recommends using metrics for counts and notes that more logging can increase costs. ### Source excerpt This is Part 1 of a short series: one developer, one bad night, six mistakes, and a production-ready setup at the end. ## Pinterest Uses Event Time Watermarks to Finalize Petabyte Scale Data Partitions DevFeed: [Pinterest Uses Event Time Watermarks to Finalize Petabyte Scale Data Partitions]() Original publisher: [Read original article]() Author: Leela Kumili Published: 2026-10-19T14:37:00Z Content type: news Language: en Sources: [InfoQ]() Topics: [Apache Flink](), [data-engineering](), [Apache Kafka](), [deltalake](), [Event-Streaming](), [Temporal data](), [Retrieval Augmented Generation (RAG)](), [Amazon Redshift]() Tags: [apache-flink](), [apache-iceberg](), [apache-kafka](), [apache-spark](), [architecture-design](), [cdc](), [change-data-capture](), [data](), [data-lake](), [data-partitioning](), [development](), [distributed-systems](), [event-driven-architecture](), [iceberg](), [kafka](), [news](), [partitioning](), [pinterest-event-time-partition](), [published](), [streaming]() ### AI overview Pinterest's ingestion framework combines Apache Flink event-time progress with Apache Iceberg metadata to publish a monotonic partition-finalization watermark for continuously ingested change data. The signal helps downstream jobs determine when data is sufficiently complete, while percentile-based policies can trade completeness for freshness. ### Source excerpt Pinterest has introduced event time-based partition finalization for its database ingestion framework. The mechanism uses Apache Flink to track event time and Apache Iceberg metadata to publish a non-regressing watermark, helping downstream jobs determine when continuously ingested CDC data is sufficiently complete for processing. By Leela Kumili ## The Kafka Optimization That Saved Us $350K a Year DevFeed: [The Kafka Optimization That Saved Us $350K a Year]() Original publisher: [Read original article]() Author: Mayank Published: 2026-10-19T07:37:31Z Content type: article Language: en Sources: [JioHotstar]() Topics: [Apache Kafka](), [data-engineering](), [Event-Streaming](), [Data serialization](), [zero downtime deployment]() Tags: [data](), [go](), [kafka](), [protobuf](), [real-time](), [scale](), [smt]() ### AI overview JioHotstar describes a stateless Kafka-to-Kafka service that scans protobuf wire bytes to retain only fields needed by downstream consumers. The article details offset tracking, delivery acknowledgments, backpressure, rebalancing, and shutdown behavior, reporting an average message-size reduction of 84.9% and close to $350,000 in annual savings. ### Source excerpt The JioHotstar platform receives in excess of 40TB data every day. At 2.1 million messages per second. At this scale, every superflous byte is costly, like real money costly. In this blog, we cover strategies that helped us optimise our ingest workflow which ultimately saved real money. The Waste Our video streaming platform emits a telemetry heartbeat for every active playback session -- a rich, 7.5 KB protobuf event carrying 300+ fields of session context. Six independent downstream consumers subscribe to it. Most of them need about 1.1 KB. That is 6.4 KB of waste, multiplied by six consumers, multiplied by 2.1 million times a second. The annual cost ran into hundreds of thousands of dollars in Kafka egress alone -- before counting the CPU and GC pressure of fully deserializing messages just to throw away 85% of the fields. The Strategy We needed a way to strip the fat after ingestion but before fan-out, without losing a single message. The obvious tool -- Kafka Connect SMTs -- could not do it without reintroducing the full deserialization cost. So we built a stateless Kafka-to-Kafka micro service that scans raw protobuf wire bytes, extracts only the fields consumers actually need, and publishes a minimized event -- with a zero-loss offset tracking chain that guarantees at-least-once delivery through crashes and rebalances. This post walks through the architecture, the ack chain that makes zero loss possible, the wire-level protobuf trick that avoids proto.Unmarshal, and the rebalancing and shutdown sequences that keep it safe at scale. What Is a Heartbeat? A heartbeat is a periodic telemetry event emitted by every client on our video streaming platform during playback. It is the signal that tells our systems whether a session is healthy -- whether playback started, stalled, buffered, dropped quality, or failed. Our real-time alerting system, QoS dashboards, churn models, and several other systems all depend on it to understand the real-time health of every active sessi ## Vercel Daemons is now in public beta DevFeed: [Vercel Daemons is now in public beta]() Original publisher: [Read original article]() Author: Florentin Eckl Published: 2026-10-15T07:00:00Z Content type: release Language: en Sources: [Vercel News]() Topics: [AWS Fargate](), [Serverless](), [container](), [CI/CD]() Tags: [configuration](), [containers](), [deploy](), [dockerfile](), [environment-variables](), [jobs](), [kafka](), [memory](), [metrics](), [observability](), [public-beta](), [queues](), [rabbitmq](), [self-hosted](), [vercel]() ### AI overview Vercel Daemons entered public beta, enabling long-running worker processes to poll queues, databases, and third-party systems without an HTTP server. Daemons run as containers alongside an app; Vercel builds the image, maintains configured regional instance counts, and restarts instances that exit or crash. Setup uses a Dockerfile or Containerfile and a daemons configuration entry. ### Source excerpt Vercel Daemons is now in public beta, allowing you to run long-running processes that pull and execute work from Vercel Queues, databases, or third-party systems without requiring an HTTP server: Consumers using Vercel Queues poll mode, Celery, BullMQ, RabbitMQ, or Kafka. Temporal or Cadence workers that long-poll their task queues. Self-hosted CI runners for GitHub Actions, GitLab, or CircleCI that poll for jobs without requiring inbound connections. Daemons run containers on Fluid compute and deploy alongside your app in the same project. Vercel builds your image and maintains the instance count you configure in each region, restarting instances that exit or crash. Get started by packaging your application and dependencies in a Dockerfile or Containerfile, then adding a daemons entry to one configuration file at your project root. This example uses worker/Dockerfile to run one Celery worker in iad1 (Washington, D.C.): Set any connection details your worker needs in your project's production environment variables, then deploy to production. Vercel starts the configured instances, and their output appears in your project's Logs. CPU and memory metrics are available with Observability Plus. Learn more in the Vercel Daemons documentation. Read more ## TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads DevFeed: [TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads]() Original publisher: [Read original article]() Author: Netflix Technology Blog Published: 2026-10-14T19:56:08Z Content type: article Language: en Sources: [Netflix]() Topics: [Apache Cassandra](), [cost-optimization](), [performance-engineering](), [big-data](), [Apache Flink](), [Apache Kafka](), [Distributed Systems & Parallel Computing](), [Event-Streaming](), [parquet](), [Data integrity in software engineering](), [Amazon Elastic Block Store (Amazon EBS)]() Tags: [2](), [abstraction](), [analysis](), [apache](), [apache-kafka](), [api](), [audits](), [backups](), [based](), [but](), [can](), [cases](), [cassandra](), [challenges](), [flink](), [journey](), [kafka](), [native](), [storage](), [streams]() ### AI overview Netflix describes two ways to move older, immutable TimeSeries data from Cassandra hot storage to S3 while retaining the same query interface. The first streams data through Kafka and Flink, then compacts it into sorted Parquet files with small row groups and cached Bloom filters; it reports roughly 150 ms p50 and 500 ms p99 reads, but adds streaming infrastructure and operational cost. The later design reads directly from existing Cassandra backups. A ReadOnlyCompactionStrategy reduces SSTable overlap before slices become immutable; Cassandra's pluggable Memtables then use backup metadata and locally cached index components to plan parallel, cached S3 chunk reads. Retries, integrity checks, initialization isolation, and cluster-wide checks for completed compaction and uploaded backups protect the transition. Because S3 reads use more CPU, they can run on a separate, smaller Cassandra cluster. The excerpt reports over 15 PB stored, thousands of cold reads per second per region, about 30% better p90 latency than the earlier cold tier, and p99 below 500 ms. ### Source excerpt By Oleksii Tkachuk, Rajiv Shringi, Chris Lohfink, Abe Ratnofsky, Kartik Sathyanarayanan Introduction In an earlier post, we introduced the TimeSeries Abstraction, our high-throughput, low-latency platform designed to store and query temporal event data at scale. It has since become the operational backbone for Netflix's largest datasets, which now span multiple petabytes. However, managing such large datasets in hot storage, tuned for low-millisecond latency access, is not cheap. At Netflix, cost-efficiency is a continuous discipline rather than a temporary initiative. This post shares two solutions that significantly reduce storage costs by continuously moving older, less frequently accessed data into cheaper object storage: Solution 1: Uses Apache Kafka and Flink to stream data directly to S3 Solution 2: Leverages Cassandra backups in S3 and pluggable Memtables to read data directly from S3 We'll start with the Kafka/Flink-based design, then describe how we evolved to a native Cassandra-based cold tier. Challenges As Netflix's TimeSeries datasets have expanded, we've encountered several compounding challenges: Growing retention demands: Users often push for increasing data retention to serve a broader range of use cases, which in turn increases the storage costs. Escalating costs: Maintaining massive, dense clusters for infrequently accessed historical data is prohibitively expensive. Sparse historical access: Most queries target recent events, but older data must remain readily available for occasional audits and investigations. Operational strain: High storage density on hot nodes complicates scaling, deployment, and cluster stability. Latency requirements: Even historical data sometimes requires sub-second query performance for internal tooling and analysis. This creates a clear opportunity: keep the most recent data on fast storage and migrate older, rarely read data to cheaper storage while preserving a consistent API. Done well, this can meet user and operat ## Vast Data bets stitched-together AI pipelines can't keep sensitive data in sync DevFeed: [Vast Data bets stitched-together AI pipelines can't keep sensitive data in sync]() Original publisher: [Read original article]() Author: Jonathan Anthony Published: 2026-10-14T11:15:06Z Content type: news Language: en Sources: [SiliconANGLE]() Topics: [digital sovereignty](), [AI-Ready Data](), [AI Infrastructure](), [sensitive data](), [CI/CD secrets management](), [Embedded Replicas](), [Cloud]() Tags: [access-control](), [access-controls](), [ai](), [ai-data-pipeline-forum-2026](), [ai-inference](), [ai-infrastructure](), [ai-networking](), [aidatapipelineforum](), [chiplets](), [cohere](), [computing](), [cube-event-coverage](), [dase](), [data-centers](), [data-sovereignty](), [dataenclave](), [dell](), [hadoop](), [hyperscale-computing](), [infrastructure](), [john-furrier](), [kafka](), [liquid-cooling](), [neocloud](), [news](), [nvidia](), [open-compute-project](), [open-hardware](), [open-standards](), [power-management](), [scaleflux](), [sensitive-data](), [source](), [thecube](), [vast-data](), [vaughn-stewart]() ### AI overview Vast Data argues that assembling AI pipelines from separate tools can leave embeddings, source data, and access permissions out of sync, limiting the use of sensitive datasets. Its proposed DataEnclave runtime uses secured virtual machines, cryptographic attestation, and access logging to support inference on regulated data. ### Source excerpt Data sovereignty is becoming an architectural problem as enterprises push artificial intelligence inference toward their most sensitive information. Pipelines assembled from separate tools struggle to keep embeddings, permissions and source data aligned, leaving many valuable data sets out of AI's reach. The shift from model training to inference is changing what AI infrastructure has to [...] The post Vast Data bets stitched-together AI pipelines can't keep sensitive data in sync appeared first on SiliconANGLE. ## AWS Reworks EventBridge Custom Buses for Enterprise-Scale Event-Driven Architectures DevFeed: [AWS Reworks EventBridge Custom Buses for Enterprise-Scale Event-Driven Architectures]() Original publisher: [Read original article]() Author: Craig Risi Published: 2026-10-13T12:00:00Z Content type: news Language: en Sources: [InfoQ]() Topics: [Amazon EventBridge](), [AWS Lambda](), [Resource Access Manager (RAM)](), [Apache Kafka]() Tags: [amazon-eventbridge](), [architecture](), [aws](), [aws-eventbridge-scaling](), [devops](), [event-driven](), [event-driven-architecture](), [governance](), [kafka](), [news](), [scaling]() ### AI overview AWS introduced an enhanced Amazon EventBridge Custom Event Bus that can be shared across accounts, with subscriber management, retention and replay, ordering, deduplication, and revised pricing. The model supports a central event backbone with teams managing their own subscriptions; AWS describes it as complementary to Kafka rather than a direct replacement for every streaming use case. ### Source excerpt AWS has introduced an enhanced Custom Event Bus for Amazon EventBridge, targeting a problem that often emerges when event-driven architectures move beyond a single team or AWS account: the event infrastructure itself becomes fragmented. By Craig Risi ## Article: Building a Session-Ordered Kafka Pipeline in Go DevFeed: [Article: Building a Session-Ordered Kafka Pipeline in Go]() Original publisher: [Read original article]() Author: Joshua Oluikpe Published: 2026-10-07T09:00:00Z Content type: tutorial Language: en Sources: [InfoQ]() Topics: [Apache Kafka](), [Publish-subscribe pattern](), [sharding]() Tags: [ai](), [ai-ml-data-engineering](), [apache-kafka](), [apache-kafka-golang-session-ordered-pipeline](), [architecture](), [architecture-design](), [article](), [consistent-hashing](), [database](), [event-driven-architecture](), [event-stream-processing](), [go](), [kafka](), [key-value-store](), [ml-data-engineering](), [nosql](), [partitions](), [redis](), [streaming](), [testing]() ### AI overview The article explains a Go implementation for preserving message order within Kafka sessions while processing separate sessions in parallel. It describes consistent-hash routing to per-session goroutines, in-place retries, contiguous offset commits for recovery, and operational safeguards such as backpressure and rebalance handling. Production monitoring observed no ordering violations across more than 40 million messages, while the article notes that full-scale sequence-level validation remains outstanding. ### Source excerpt The article describes a custom implementation that provides a session-level ordering on top of Apache Kafka partitions, supporting strict message ordering across 1000s of independent channels. The solution required application-level routing, consistent hashing, retries, and contiguous watermark commits. Engineers conducted operational hardening supported by extensive performance testing. By Joshua Oluikpe [Next page]()