# Apache Kafka

Apache Kafka is an open-source distributed event streaming platform used for data pipelines, streaming analytics, data integration, and mission-critical applications.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## A modern dilemma: Consume data from a stream or a lake

DevFeed: [A modern dilemma: Consume data from a stream or a lake](<https://devfeed.tech/articles/a-modern-dilemma-consume-data-from-a-stream-or-a-lake-141161.md>)

Original publisher: [Read original article](<https://medium.com/capital-one-tech/a-modern-dilemma-consume-data-from-a-stream-or-a-lake-296bbb354094?source=rss----3db3a67cb648---4>)

Author: Capital One Tech

Published: 2026-11-18T17:35:14Z

Content type: article

Language: en

Sources: [Capital One Tech](<https://devfeed.tech/sources/capital-one-tech.md>)

Topics: [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [etl](<https://devfeed.tech/topics/etl.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [apis](<https://devfeed.tech/tags/apis.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [batch](<https://devfeed.tech/tags/batch.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [business-processes](<https://devfeed.tech/tags/business-processes.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-computing](<https://devfeed.tech/tags/cloud-computing.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-lakes](<https://devfeed.tech/tags/data-lakes.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>), [enrichment](<https://devfeed.tech/tags/enrichment.md>), [etl](<https://devfeed.tech/tags/etl.md>), [event](<https://devfeed.tech/tags/event.md>), [flink](<https://devfeed.tech/tags/flink.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [latency](<https://devfeed.tech/tags/latency.md>), [legacy](<https://devfeed.tech/tags/legacy.md>), [log](<https://devfeed.tech/tags/log.md>), [mapping](<https://devfeed.tech/tags/mapping.md>), [model-context-protocol](<https://devfeed.tech/tags/model-context-protocol.md>), [software-architecture](<https://devfeed.tech/tags/software-architecture.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [stream](<https://devfeed.tech/tags/stream.md>)

### AI overview

Choosing a data consumption pattern early can avoid costly architectural changes later. APIs support synchronous, record-by-record requests and paginated multirecord results, making them suitable for low-latency needs; Model Context Protocol can provide similar access for agents and is often an API wrapper. Event streams, commonly built with Kafka, Flink or Spark Streaming, suit sub-second latency and enable per-event calculations, transformations and enrichment. The excerpt also identifies Delta Change Data Feed (CDF) as a record-by-record pattern and traditional batch ETL as a multirecord pattern, but ends before explaining CDF's mechanics or comparing these approaches further.

### Source excerpt

Data is a critical component of software architecture, providing the foundation for business processes, including those powered by artificial intelligence (AI). Quality, volume, latency, consistency, ordering and duplicates are essential considerations in software design. Modern cloud architectures consume data in various ways, such as through application programming interfaces (APIs), streams and data lakes. You can consume data at the individual event or record level as it arrives or in larger batches. Legacy architectures often process data in scheduled batches, forcing surrounding architecture and business processes to depend on that approach. Getting the data consumption approach right up front is vital, because rearchitecting later is costly. General patterns to consume data There are several general data consumption patterns. These include APIs (record-by-record or multirecord results), event stream consumption (record-by-record), Delta Change Data Feed (CDF) consumption (record-by-record) or traditional batch ETL-extract, transform, load (multirecord results). APIs are ideal for low-latency, record-by-record requirements. They also support multirecord results, typically through pagination. APIs usually follow a synchronous approach, where the client makes a call and waits for a response before moving to the next step. Model Context Protocol is an industry specification for agents to consume data and can support record-by-record or batch results via pagination, similar to APIs. They are often a wrapper on top of APIs. The event stream pattern works well for sub-second latency requirements and typically runs on a tech stack like Kafka, Flink or Spark Streaming. These technologies excel at record-level calculations (like on-the-fly calculation), transformations (like column mapping) or enrichment (like adding data to an existing event). Delta Lake is often used in data lakes and serves as a transaction log to standard Apache Parquet data files (source). CDF is

## Evaluating AST-Derived Knowledge Graph Retrieval for AIOps Incident Diagnosis in Apache Kafka

DevFeed: [Evaluating AST-Derived Knowledge Graph Retrieval for AIOps Incident Diagnosis in Apache Kafka](<https://devfeed.tech/articles/evaluating-ast-derived-knowledge-graph-retrieval-for-aiops-incident-diagnosis-in-apache-kafka-139584.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21625>)

Author: Shashank Mishra, Shweta Mishra

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [AIOps](<https://devfeed.tech/topics/aiops.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>), [graph-database](<https://devfeed.tech/topics/graph-database.md>), [Parsing](<https://devfeed.tech/topics/parsing.md>)

Tags: [aiops](<https://devfeed.tech/tags/aiops.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [ast](<https://devfeed.tech/tags/ast.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [diagnosis](<https://devfeed.tech/tags/diagnosis.md>), [graph](<https://devfeed.tech/tags/graph.md>), [incident](<https://devfeed.tech/tags/incident.md>), [knowledge-graph](<https://devfeed.tech/tags/knowledge-graph.md>), [log](<https://devfeed.tech/tags/log.md>), [queries](<https://devfeed.tech/tags/queries.md>), [reranking](<https://devfeed.tech/tags/reranking.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [source](<https://devfeed.tech/tags/source.md>)

### AI overview

In Apache Kafka incident diagnosis, vector-only retrieval achieved Recall@20 of 0.956, and adding static or confidence-guided staged AST ownership-graph expansion did not significantly improve recall. Staging did reduce cross-encoder pair workload and measured reranking time by about 52% compared with static expansion. As an external control, HippoRAG scored 0.308 versus 0.803 for its ColBERTv2-only backbone; a post-hoc document ensemble raised its score to 0.648 but did not close the gap. In a manually evaluated 100-query study, AST-derived structural evidence showed no significant advantage in root-cause correctness. The findings are based on 1,600 source-linked log records and a 315-query benchmark covering 103 targets.

### Source excerpt

arXiv:2611.21625v1 Announce Type: new Abstract: Incident diagnosis requires retrieving source-linked operational evidence from large codebases. We evaluate graph-augmented retrieval for Apache Kafka using an abstract-syntax-tree (AST)-derived ownership graph over 1,600 source-linked log records and an audited benchmark of 315 incident-style queries covering 103 unique targets. Vector-only retrieval is a strong baseline at Recall@20 of 0.956, and neither static nor confidence-guided staged AST expansion significantly improves Recall@20 over it. As an external control we evaluate HippoRAG, a published text-derived graph-retrieval system, against its own ColBERTv2 retrieval backbone. Against ColBERTv2-only (Recall@20 0.803), HippoRAG attained 0.308; a post-hoc document-ensemble sensitivity raised Recall@20 to 0.648 but did not eliminate the deficit. Relative to static expansion, staging reduces cross-encoder pair workload and measured reranking time by about 52%. A manually evaluated 100-query downstream study finds no significant root-cause-correctness advantage for AST-derived structural evidence.

## Gateway-Free Kafka Streaming from FPGA-Based Detectors: A Bare-Metal Pipelined Producer on Zynq-7000

DevFeed: [Gateway-Free Kafka Streaming from FPGA-Based Detectors: A Bare-Metal Pipelined Producer on Zynq-7000](<https://devfeed.tech/articles/gateway-free-kafka-streaming-from-fpga-based-detectors-a-bare-metal-pipelined-producer-on-zynq-7000-140403.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22674>)

Author: Fernanda Zapata Bascu\\~n\\'an, Mat\\'ias Hampel

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Networking](<https://devfeed.tech/sources/arxiv-networking.md>)

Topics: [Xilinx Zynq](<https://devfeed.tech/topics/xilinx-zynq.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [fpga](<https://devfeed.tech/topics/fpga.md>), [lwIP](<https://devfeed.tech/topics/lwip.md>), [IO](<https://devfeed.tech/topics/io.md>)

Tags: [bare-metal](<https://devfeed.tech/tags/bare-metal.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-ni](<https://devfeed.tech/tags/cs-ni.md>), [dma](<https://devfeed.tech/tags/dma.md>), [errors](<https://devfeed.tech/tags/errors.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [fpga](<https://devfeed.tech/tags/fpga.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [lwip](<https://devfeed.tech/tags/lwip.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [topic](<https://devfeed.tech/tags/topic.md>)

### AI overview

A bare-metal Kafka producer on a Zynq-7000 streams detector events directly from programmable logic to a standard broker, without an intermediate gateway. Events move over AXI4-Stream and AXI DMA to the processing system, which builds ProduceRequest messages and sends them through lwIP's raw TCP/IP API. A configurable limit on outstanding requests was tested at pipeline depths 1, 2, 4, and 8 using two DMA sources across 80 runs totaling 400,000 events. Mean throughput rose from about 5.3 kevents/s at depth 1 to 16.2 kevents/s at depth 4 and 16.8-17.5 kevents/s at depth 8. Median publish-to-acknowledgement time was lowest at depth 4 (0.66 ms), versus 0.83-0.86 ms at depth 8; depth 4's P95 was 0.81-0.83 ms. All runs had the expected frame sequence in exported metrics and no recorded Kafka errors. The excerpt reports these measurements but does not establish performance beyond the tested setup.

### Source excerpt

arXiv:2611.22674v1 Announce Type: new Abstract: We present a bare-metal Apache Kafka producer implemented on a Zynq-7000 System-on-Chip (SoC) for event-driven data acquisition. Events are generated in Programmable Logic (PL), transferred to the Processing System (PS) through AXI4-Stream and AXI DMA, and published to a standard Kafka broker without an intermediate gateway computer. The PS constructs ProduceRequest messages and transmits them through the lwIP raw TCP/IP API. A configurable pipeline limits the number of outstanding Kafka requests. Pipeline depths of D = 1, 2, 4, and 8 were evaluated with two independent DMA sources. For each depth, 20 independent runs of 5,000 events were acquired, for 400,000 event records in total. Each event carries a 32-bit frame counter and a 32-bit PL timestamp. PS-side instrumentation records DMA start and completion, publication, successful local tcp_write() acceptance, and reception of the corresponding Kafka acknowledgement. Measurement records are exported to a separate Kafka topic after the timed event stream has completed. Each of the 80 runs contained 5,000 records with frame values from 0 to 4,999, with no sequence discontinuities in the exported metric records and no recorded Kafka errors. Mean operational throughput was approximately 5.3 kevents/s at D = 1, 9.4-9.6 kevents/s at D = 2, 16.2 kevents/s at D = 4, and 16.8-17.5 kevents/s at D = 8. The median publish-to-acknowledgement interval was 1.62-1.64 ms at D = 1, 0.98 ms at D = 2, 0.66 ms at D = 4, and 0.83-0.86 ms at D = 8. At D = 4, the corresponding P95 values were 0.81-0.83 ms.

## Deliver petabytes of Apache Kafka data to Amazon S3 at up to 60% lower cost with Amazon MSK Express brokers

DevFeed: [Deliver petabytes of Apache Kafka data to Amazon S3 at up to 60% lower cost with Amazon MSK Express brokers](<https://devfeed.tech/articles/deliver-petabytes-of-apache-kafka-data-to-amazon-s3-at-up-to-60-lower-cost-with-amazon-msk-express-brokers-136179.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/deliver-petabytes-of-apache-kafka-data-to-amazon-s3-at-up-to-60-lower-cost-with-amazon-msk-express-brokers/>)

Author: Diksha Dubey

Published: 2026-11-16T20:42:44Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>)

Tags: [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-managed-streaming-for-apache-kafka-amazon-msk](<https://devfeed.tech/tags/amazon-managed-streaming-for-apache-kafka-amazon-msk.md>), [amazon-msk](<https://devfeed.tech/tags/amazon-msk.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-simple-storage-service-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-s3.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [channel](<https://devfeed.tech/tags/channel.md>), [connect](<https://devfeed.tech/tags/connect.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [delivered](<https://devfeed.tech/tags/delivered.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [express](<https://devfeed.tech/tags/express.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [one](<https://devfeed.tech/tags/one.md>), [post](<https://devfeed.tech/tags/post.md>), [run](<https://devfeed.tech/tags/run.md>), [s3](<https://devfeed.tech/tags/s3.md>), [up](<https://devfeed.tech/tags/up.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Amazon MSK Express brokers deliver Kafka topic records to general purpose Amazon S3 buckets through a channel, without Kafka Connect. The article says delivery is billed per byte, adds no broker egress load, and can cost up to 60% less than self-managed alternatives. Brokers automatically scale storage and rebalance partitions; MSK manages recovery and updates. Channels batch records into S3 objects, preserving producer formats, with configurable compression, storage class, and object key layout. JSON and String records are validated; failures go to an S3 dead-letter queue while delivery continues. The article also describes CloudWatch metrics and logs, setup through the console, APIs, CLI, or an Agent Skill, and recommends overlapping a new channel with an existing connector until delivered data is confirmed. Review supported configurations and prerequisites before production use.

### Source excerpt

Amazon MSK Express brokers now deliver Apache Kafka topic data to Amazon S3 at up to 60% lower cost, with no Kafka Connect to run. You create a channel on a topic and pay only for the data delivered. In this post, we cover the two challenges of delivering Kafka data to Amazon S3 today and how Express brokers and data delivery to Amazon S3 address each one.

## How Infinitas Learning uses ClickHouse Managed Postgres to process billions of events for 1.6M unique learners

DevFeed: [How Infinitas Learning uses ClickHouse Managed Postgres to process billions of events for 1.6M unique learners](<https://devfeed.tech/articles/how-infinitas-learning-uses-clickhouse-managed-postgres-to-process-billions-of-events-for-1-6m-unique-learners-136653.md>)

Original publisher: [Read original article](<https://clickhouse.com/blog/infinitas-learning-clickhouse-managed-postgres>)

Author: ClickHouse

Published: 2026-11-16T10:54:45Z

Content type: news

Language: en

Sources: [ClickHouse Blog](<https://devfeed.tech/sources/clickhouse-blog.md>)

Topics: [clickhouse](<https://devfeed.tech/topics/clickhouse.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [self-hosted databases vs managed services](<https://devfeed.tech/topics/self-hosted-databases-vs-managed-services.md>)

Tags: [1-6](<https://devfeed.tech/tags/1-6.md>), [about](<https://devfeed.tech/tags/about.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [cache](<https://devfeed.tech/tags/cache.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-cache](<https://devfeed.tech/tags/distributed-cache.md>), [events](<https://devfeed.tech/tags/events.md>), [latency](<https://devfeed.tech/tags/latency.md>), [learning](<https://devfeed.tech/tags/learning.md>), [managed](<https://devfeed.tech/tags/managed.md>), [p99](<https://devfeed.tech/tags/p99.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [process](<https://devfeed.tech/tags/process.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [time](<https://devfeed.tech/tags/time.md>), [transactions](<https://devfeed.tech/tags/transactions.md>), [unique](<https://devfeed.tech/tags/unique.md>), [uses](<https://devfeed.tech/tags/uses.md>)

### AI overview

Infinitas Learning separates learner-result transactions from customer-facing analytics: Postgres processes assignment events and updates learner progress, while ClickHouse Cloud serves teacher dashboards. After Neon, the team moved 750 GB to ClickHouse Managed Postgres in 16 hours and completed a no-downtime cutover in 20 minutes; reported P50 transaction latency fell from milliseconds to double-digit microseconds. For analytics, ClickHouse compressed a nearly 2-billion-row table from 300 GB to 27 GB, but cache misses caused P99 spikes up to 13 seconds against a 500 ms target. Self-hosting brought P99 to 300 ms but left the team responsible for operations and running a single replica. With ClickHouse support optimizing data and settings, then enabling distributed cache in Cloud, load-test P99 reached 410 ms. The team plans to connect Postgres and ClickHouse through pg_clickhouse, but that integration is described as a future vision.

### Source excerpt

Infinitas Learning runs transactions for 1.6 million learners on ClickHouse Managed Postgres and serves real-time analytics to teachers from ClickHouse Cloud, where distributed cache cut P99 latency from about 13 seconds to 410 milliseconds.

## Kafka Listeners, Mail and Live Streams in Your Backend, With Records as Entities

DevFeed: [Kafka Listeners, Mail and Live Streams in Your Backend, With Records as Entities](<https://devfeed.tech/articles/kafka-listeners-mail-and-live-streams-in-your-backend-with-records-as-entities-135299.md>)

Original publisher: [Read original article](<https://www.codenameone.com/blog/kafka-mail-and-records-in-your-java-backend/>)

Author: Shai Almog

Published: 2026-11-16T00:00:00Z

Content type: release

Language: en

Sources: [CodeName One](<https://devfeed.tech/sources/codename-one.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [email](<https://devfeed.tech/topics/email.md>), [RabbitMQ](<https://devfeed.tech/topics/rabbitmq.md>), [jpa](<https://devfeed.tech/topics/jpa.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>)

Tags: [17](<https://devfeed.tech/tags/17.md>), [accessors](<https://devfeed.tech/tags/accessors.md>), [amazon-ses](<https://devfeed.tech/tags/amazon-ses.md>), [amqp](<https://devfeed.tech/tags/amqp.md>), [annotations](<https://devfeed.tech/tags/annotations.md>), [api](<https://devfeed.tech/tags/api.md>), [backend](<https://devfeed.tech/tags/backend.md>), [events](<https://devfeed.tech/tags/events.md>), [java](<https://devfeed.tech/tags/java.md>), [jpa](<https://devfeed.tech/tags/jpa.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [live](<https://devfeed.tech/tags/live.md>), [mail](<https://devfeed.tech/tags/mail.md>), [rabbitmq](<https://devfeed.tech/tags/rabbitmq.md>), [records](<https://devfeed.tech/tags/records.md>), [redis](<https://devfeed.tech/tags/redis.md>), [server](<https://devfeed.tech/tags/server.md>), [server-sent-events](<https://devfeed.tech/tags/server-sent-events.md>), [spring](<https://devfeed.tech/tags/spring.md>), [streams](<https://devfeed.tech/tags/streams.md>)

### AI overview

The backend compiles Java to native code and generates wiring at build time, without runtime proxies or reflection. Its ORM maps immutable Java records as entities on devices and servers: saves return managed copies with generated keys and versions, while `Ref<T>` handles relationships. Record sources must use Java 17; self-referential entity records, `Map` relationships, lazy `Optional` relationships, and relationships inside embedded records are unsupported. Messaging uses Spring-style listeners across seven transports, including Kafka, RabbitMQ and Redis Streams; transactional sends go through an outbox, with at-least-once delivery, retries and dead-letter destinations. Kafka currently lacks snappy, lz4 and zstd compression, transactions and SASL. Mail is staged until commit, supports SMTP, SMTPS and Amazon SES, and retries transient failures up to six times; receiving mail and DKIM signing are unsupported. Server-sent events support reconnect resumption through `lastEventId()`, and `HttpProxy` can relay event streams without buffering.

### Source excerpt

Kafka, RabbitMQ and Redis with Spring's annotations, transactional mail, server-sent events, and Java records as entities, which JPA still doesn't allow.

## Apache Spark vs Apache Flink: which should you use?

DevFeed: [Apache Spark vs Apache Flink: which should you use?](<https://devfeed.tech/articles/apache-spark-vs-apache-flink-which-should-you-use-133225.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/spark-vs-flink>)

Author: Databricks Staff

Published: 2026-11-13T20:44:47.516244Z

Content type: comparison

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [batch](<https://devfeed.tech/tags/batch.md>), [better](<https://devfeed.tech/tags/better.md>), [choice](<https://devfeed.tech/tags/choice.md>), [data-plus-ai-foundations](<https://devfeed.tech/tags/data-plus-ai-foundations.md>), [etl](<https://devfeed.tech/tags/etl.md>), [flink](<https://devfeed.tech/tags/flink.md>), [spark](<https://devfeed.tech/tags/spark.md>), [sql](<https://devfeed.tech/tags/sql.md>), [teams](<https://devfeed.tech/tags/teams.md>), [vs](<https://devfeed.tech/tags/vs.md>)

### AI overview

Spark and Flink both support exactly-once processing, stateful operators and SQL APIs, but Spark spans batch, streaming, SQL and AI while Flink is stream-native. Spark Real-Time Mode uses a trigger on an existing Structured Streaming query to process continuously at millisecond latency, so teams can move between latency tiers without rewriting query code. The excerpt says Databricks' benchmark found Spark matched Flink on stateless work and had lower p99 latency for stateful aggregation and stream-static joins; results came from different providers and the comparison used Spark's worst run versus Flink's best, so teams should benchmark their own workloads. Flink may fit specialized cases such as very large keyed state, complex event processing, or an existing Flink platform. Real-Time Mode requires update output mode and Kafka or a row-level foreach sink; stateful operators are available on Databricks and are slated for open-source Spark 4.3, according to the excerpt.

### Source excerpt

For most teams, Apache Spark™ is the better choice. Spark runs batch ETL, SQL analytics,...

## What's new in Postman: Kafka, MCP tests, RAML

DevFeed: [What's new in Postman: Kafka, MCP tests, RAML](<https://devfeed.tech/articles/what-s-new-in-postman-kafka-mcp-tests-raml-133540.md>)

Original publisher: [Read original article](<https://blog.postman.com/whats-new-in-postman-kafka-mcp-tests-raml/>)

Author: Quinton Wall

Published: 2026-11-13T16:20:26Z

Content type: release

Language: en

Sources: [Postman](<https://devfeed.tech/sources/postman-blog.md>)

Topics: [Postman](<https://devfeed.tech/topics/postman.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [OpenAPI Specification](<https://devfeed.tech/topics/openapi.md>), [Postman Agent Mode](<https://devfeed.tech/topics/postman-agent-mode.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [advanced-security](<https://devfeed.tech/tags/advanced-security.md>), [agent-mode](<https://devfeed.tech/tags/agent-mode.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [api](<https://devfeed.tech/tags/api.md>), [apis](<https://devfeed.tech/tags/apis.md>), [asyncapi](<https://devfeed.tech/tags/asyncapi.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-iam](<https://devfeed.tech/tags/aws-iam.md>), [blog](<https://devfeed.tech/tags/blog.md>), [connect](<https://devfeed.tech/tags/connect.md>), [general](<https://devfeed.tech/tags/general.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [mcp](<https://devfeed.tech/tags/mcp.md>), [openapi](<https://devfeed.tech/tags/openapi.md>), [post](<https://devfeed.tech/tags/post.md>), [postman](<https://devfeed.tech/tags/postman.md>), [runner](<https://devfeed.tech/tags/runner.md>), [scripts](<https://devfeed.tech/tags/scripts.md>), [spec-hub](<https://devfeed.tech/tags/spec-hub.md>), [tests](<https://devfeed.tech/tags/tests.md>), [what-s-new](<https://devfeed.tech/tags/what-s-new.md>)

### AI overview

Postman brings Kafka and MCP requests into multi-protocol collections and adds RAML 1.0 and OpenAPI 3.2 support in Spec Hub. Kafka requests can produce and consume topic messages, use SASL, AWS IAM or mTLS, and run scripts and tests alongside HTTP steps; bounded consumption lets reads finish without waiting indefinitely. Kafka requests are free, while running them in the Collection Runner or Postman CLI requires Solo, Team or Enterprise. MCP scripts and tests are available on every plan, including Free and Consumption Pricing, with hooks for connection setup, server checks, per-call results and disconnect; test results are grouped by phase. Spec Hub can import RAML projects, resolve includes and libraries, generate synced collections, and author or validate OpenAPI 3.2 specs with two-way collection sync and governance linting. The post also reports cross-region domain capture for Enterprise customers with Advanced Security Administration; it routes users to allowed teams without moving data.

### Source excerpt

Postman now supports Kafka requests, MCP scripts and tests, RAML 1.0, and OpenAPI 3.2. See what shipped and try it in the Collection Runner. The post What's new in Postman: Kafka, MCP tests, RAML appeared first on Postman Blog.

## PostgreSQL + ClickHouse Recovery: Test the Whole Data Service

DevFeed: [PostgreSQL + ClickHouse Recovery: Test the Whole Data Service](<https://devfeed.tech/articles/postgresql-clickhouse-recovery-test-the-whole-data-service-132583.md>)

Original publisher: [Read original article](<https://severalnines.com/blog/postgresql-clickhouse-recovery-test-the-whole-data-service/>)

Author: Jessie Baynas

Published: 2026-11-13T11:47:17Z

Content type: tutorial

Language: en

Sources: [SeveralNines](<https://devfeed.tech/sources/severalnines.md>)

Topics: [Disaster Recovery](<https://devfeed.tech/topics/disaster-recovery.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Synthetic Data Generation](<https://devfeed.tech/topics/synthetic-data-generation.md>)

Tags: [acceptance](<https://devfeed.tech/tags/acceptance.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [application](<https://devfeed.tech/tags/application.md>), [backup](<https://devfeed.tech/tags/backup.md>), [backup-management](<https://devfeed.tech/tags/backup-management.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [clustercontrol](<https://devfeed.tech/tags/clustercontrol.md>), [data](<https://devfeed.tech/tags/data.md>), [debezium](<https://devfeed.tech/tags/debezium.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [modern-data-stack-operations](<https://devfeed.tech/tags/modern-data-stack-operations.md>), [offset](<https://devfeed.tech/tags/offset.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [reconciliation](<https://devfeed.tech/tags/reconciliation.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [restoration](<https://devfeed.tech/tags/restoration.md>), [restore](<https://devfeed.tech/tags/restore.md>), [service](<https://devfeed.tech/tags/service.md>), [test](<https://devfeed.tech/tags/test.md>), [updates](<https://devfeed.tech/tags/updates.md>)

### AI overview

A green restore and zero Kafka lag do not prove that ClickHouse analytics are correct. In this PostgreSQL-Debezium-Kafka-ClickHouse exercise, the team records the backup's last applied Kafka offset, rewinds the consumer group before restoring, then waits for a source-backed, stable recovery boundary. It reconciles PostgreSQL and ClickHouse against independently generated expected results, checking keys, values, updates, deletes, outage-window inserts, and the dashboard query. ReplacingMergeTree checks use FINAL or version-based selection and filter delete markers; LSN versions make replayed events idempotent. The walkthrough's negative test skips the rewind: ingestion resumes and lag reaches zero, yet pre-outage updates and a delete are missing. PostgreSQL can rebuild ClickHouse only while it retains all needed rows; purged history depends on ClickHouse backups and retained Kafka events. The lab timings are not production targets, and PostgreSQL loss, Kafka loss, node loss with on-node backups, and retained-history recovery are outside the tested sequence.

### Source excerpt

A successful restore does not prove that an analytics service is delivering correct data. This PostgreSQL + ClickHouse walkthrough uses Debezium and Kafka to test backup restoration, offset replay, updates, deletes, and application-level reconciliation--including a negative test that catches a seemingly healthy but incomplete recovery. The post PostgreSQL + ClickHouse Recovery: Test the Whole Data Service appeared first on Severalnines.

## What is Real-Time Mode in Apache Spark?

DevFeed: [What is Real-Time Mode in Apache Spark?](<https://devfeed.tech/articles/what-is-real-time-mode-in-apache-spark-136421.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/what-is-real-time-mode>)

Author: Databricks Staff

Published: 2026-11-13T07:49:12Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>)

Tags: [17](<https://devfeed.tech/tags/17.md>), [2026](<https://devfeed.tech/tags/2026.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [api](<https://devfeed.tech/tags/api.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [autoscaling](<https://devfeed.tech/tags/autoscaling.md>), [availability](<https://devfeed.tech/tags/availability.md>), [batch](<https://devfeed.tech/tags/batch.md>), [batches](<https://devfeed.tech/tags/batches.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [blocking](<https://devfeed.tech/tags/blocking.md>), [data-plus-ai-foundations](<https://devfeed.tech/tags/data-plus-ai-foundations.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [spark](<https://devfeed.tech/tags/spark.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [structured](<https://devfeed.tech/tags/structured.md>), [trigger](<https://devfeed.tech/tags/trigger.md>)

### AI overview

Apache Spark Structured Streaming Real-Time Mode processes records continuously through long-running tasks, reducing latency from seconds to milliseconds without changing the query's core code. It is suited to workloads that need sub-second responses; micro-batch or availableNow costs less when seconds or minutes suffice. On Databricks, classic compute requires Runtime 16.4 LTS or later, Photon, autoscaling and spot instances disabled, plus an enablement setting; serverless use is limited to continuous Lakeflow Spark Declarative Pipelines. Capacity must cover concurrently running tasks. Kafka sources and sinks, foreach sinks, and most operators including stateful ones are supported. Delta output requires Kafka followed by a separate micro-batch query. Monitor Real-Time Mode latency metrics rather than batch duration; the end-to-end metric excludes sink write time. Latency varies with workload and capacity. The text says Databricks has GA support, while open-source availability is rolling out; it also notes stateful support arrives in Spark 4.3.

### Source excerpt

Real-Time Mode is a trigger for Apache Spark™ Structured Streaming that processes...

## Allocate shared Amazon MSK costs to Kafka topics for cost visibility

DevFeed: [Allocate shared Amazon MSK costs to Kafka topics for cost visibility](<https://devfeed.tech/articles/allocate-shared-amazon-msk-costs-to-kafka-topics-for-cost-visibility-129557.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/allocate-shared-amazon-msk-costs-to-kafka-topics-for-cost-visibility/>)

Author: Durai Krishnan

Published: 2026-11-11T19:46:04Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [cloud cost management](<https://devfeed.tech/topics/cloud-cost-management.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [Amazon Managed Service for Prometheus](<https://devfeed.tech/topics/amazon-managed-service-for-prometheus.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Prometheus](<https://devfeed.tech/topics/prometheus.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [aws-sns](<https://devfeed.tech/topics/aws-sns.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-eventbridge](<https://devfeed.tech/tags/amazon-eventbridge.md>), [amazon-managed-service-for-prometheus](<https://devfeed.tech/tags/amazon-managed-service-for-prometheus.md>), [amazon-managed-streaming-for-apache-kafka-amazon-msk](<https://devfeed.tech/tags/amazon-managed-streaming-for-apache-kafka-amazon-msk.md>), [amazon-msk](<https://devfeed.tech/tags/amazon-msk.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-simple-storage-service-amazon-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-amazon-s3.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cost-and-usage-report](<https://devfeed.tech/tags/aws-cost-and-usage-report.md>), [billing](<https://devfeed.tech/tags/billing.md>), [cloudwatch](<https://devfeed.tech/tags/cloudwatch.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cost-and-usage-report](<https://devfeed.tech/tags/cost-and-usage-report.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [prometheus](<https://devfeed.tech/tags/prometheus.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [topic](<https://devfeed.tech/tags/topic.md>), [visibility](<https://devfeed.tech/tags/visibility.md>)

### AI overview

The pipeline estimates each Kafka topic's share of an Amazon MSK Provisioned cluster's daily cost by joining cluster-level CUR charges with per-topic ingress metrics, mapping cluster names to ARNs, then allocating cost in proportion to bytes in. A daily Lambda writes Parquet results to S3 for Athena and Quick Sight. Enhanced Monitoring publishes per-topic CloudWatch metrics with little setup, but metric charges grow with topic-broker pairs. Open Monitoring exposes broker metrics at no additional charge; a Fargate scraper sends them to Amazon Managed Service for Prometheus, whose ingestion costs grow with samples. Both options use the same allocation logic. Bytes in is only a proxy: it misses retention, replication, compression, and consumer fan-out. Idle topics receive no row, so active topics may absorb the full cost; if no topics report traffic, the function splits cost evenly across configured topics. CUR can lag up to 24 hours, and metrics exist only after they are enabled. The method applies to Provisioned clusters; Serverless is out of scope.

### Source excerpt

Amazon MSK bills at the cluster level, with no per-topic breakdown. This post shows how to allocate shared MSK cluster costs to individual Kafka topics by combining AWS Cost and Usage Report billing with per-topic CloudWatch or Prometheus metrics, giving each team a usage-based view of its spend.

## How Intuit migrated from Apache Hive Metastore to AWS Glue Data Catalog

DevFeed: [How Intuit migrated from Apache Hive Metastore to AWS Glue Data Catalog](<https://devfeed.tech/articles/how-intuit-migrated-from-apache-hive-metastore-to-aws-glue-data-catalog-129561.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/how-intuit-migrated-from-apache-hive-metastore-to-aws-glue-data-catalog/>)

Author: Srinivasan Krishnasamy

Published: 2026-11-11T15:43:26Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [AWS Lake Formation](<https://devfeed.tech/topics/aws-lake-formation.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>)

Tags: [access-control](<https://devfeed.tech/tags/access-control.md>), [access-controls](<https://devfeed.tech/tags/access-controls.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-services](<https://devfeed.tech/tags/ai-services.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-aurora](<https://devfeed.tech/tags/amazon-aurora.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-dynamodb](<https://devfeed.tech/tags/amazon-dynamodb.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-hive](<https://devfeed.tech/tags/apache-hive.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data](<https://devfeed.tech/tags/data.md>), [data-catalog](<https://devfeed.tech/tags/data-catalog.md>), [data-discovery](<https://devfeed.tech/tags/data-discovery.md>), [hive](<https://devfeed.tech/tags/hive.md>), [integration](<https://devfeed.tech/tags/integration.md>), [intuit](<https://devfeed.tech/tags/intuit.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>)

### AI overview

Intuit migrated databases, tables, and partitions from Apache Hive Metastore (HMS) to AWS Glue Data Catalog (GDC) using a phased approach: a one-time migration, ongoing synchronization in both directions, hourly parity checks, and targeted repair syncs. A DynamoDB source-of-truth flag per table prevents sync loops. HMS changes flow through Debezium CDC and Kafka, with partition events batched over 10-second windows; GDC changes use EventBridge, Lambda, Kafka, and Flink to order events before applying them to HMS. Before migrating, Intuit recommends auditing unused objects, estimating and load-testing Glue API quotas, batching writes, and using retries with backoff to manage throttling. The migration covered databases, tables, and partitions for S3-backed data; column statistics, privileges, roles, functions, and transactions were out of scope. GDC also does not support database or table renames.

### Source excerpt

How Intuit migrated its data catalog from Apache Hive Metastore to AWS Glue Data Catalog on its journey to an open, secure lakehouse. This post covers the challenges, the phased bidirectional-sync solution, and the benefits: reduced operational overhead, improved data discovery, and tighter integration with AWS analytics services.

## What Actually Happens When a Kafka Region Fails?

DevFeed: [What Actually Happens When a Kafka Region Fails?](<https://devfeed.tech/articles/what-actually-happens-when-a-kafka-region-fails-126633.md>)

Original publisher: [Read original article](<https://www.sitepoint.com/what-actually-happens-when-a-kafka-region-fails/>)

Author: SitePoint Team

Published: 2026-11-10T00:42:14Z

Content type: article

Language: en

Sources: [SitePoint](<https://devfeed.tech/sources/sitepoint.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [CAP theorem](<https://devfeed.tech/topics/cap-theorem.md>), [consistency](<https://devfeed.tech/topics/consistency.md>), [downtime](<https://devfeed.tech/topics/downtime.md>), [Data integrity in software engineering](<https://devfeed.tech/topics/data-integrity-in-software-engineering.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [availability](<https://devfeed.tech/tags/availability.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [computer-science](<https://devfeed.tech/tags/computer-science.md>), [computing](<https://devfeed.tech/tags/computing.md>), [durability](<https://devfeed.tech/tags/durability.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [performance-scaling](<https://devfeed.tech/tags/performance-scaling.md>), [region](<https://devfeed.tech/tags/region.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [technology](<https://devfeed.tech/tags/technology.md>)

### AI overview

A three-region Kafka design with two brokers per region and RF=6 can retain four replicas after a region fails. With `min.insync.replicas=3` and producer `acks=all`, writes can continue while at least three replicas remain in the ISR, including during a broker restart; Kafka stops writes below that threshold. Five KRaft controllers split 2+2+1 preserve quorum after any one region loss, but may not leave room to restart a controller. More controllers add headroom and coordination overhead; dynamic quorum reconfiguration (Kafka 3.9/KIP-853) can reduce voting membership during an outage, but requires successful committed changes. WAN degradation can shrink ISR or break quorum, while cross-region replication adds latency and resource costs. The surviving regions must be sized for extra traffic, and throttled replica catch-up can limit recovery impact at the cost of longer recovery.

### Source excerpt

null Continue reading What Actually Happens When a Kafka Region Fails? on SitePoint.

## Webhooks to Kafka: How to Receive Events and Publish Them to Topics

DevFeed: [Webhooks to Kafka: How to Receive Events and Publish Them to Topics](<https://devfeed.tech/articles/webhooks-to-kafka-how-to-receive-events-and-publish-them-to-topics-121710.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/webhooks-to-kafka>)

Author: Seán Whelan

Published: 2026-11-06T12:27:07Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [compare](<https://devfeed.tech/tags/compare.md>), [events](<https://devfeed.tech/tags/events.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [http](<https://devfeed.tech/tags/http.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [managed](<https://devfeed.tech/tags/managed.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [retries](<https://devfeed.tech/tags/retries.md>), [webhooks](<https://devfeed.tech/tags/webhooks.md>)

### AI overview

Webhook providers cannot post directly to Kafka, so an HTTPS receiver must verify each request and publish it. The excerpt compares a custom service or HTTP gateway, API Gateway with Lambda, Kafka Connect, and a managed pipeline; the choice depends on operational control, AWS or Connect infrastructure, and who hosts the endpoint. For a direct receiver, verify signatures over raw bytes, produce with the delivery ID as key, and return success only after Kafka acknowledges; provider retries make delivery at least once, so redeliveries can become duplicate messages. Providers have different deadlines and retry behavior--Shopify allows five seconds, while GitHub does not automatically redeliver. Estuary instead acknowledges after committing requests to a durable collection, then materializes them to Kafka independently, allowing capture to continue during a broker outage. Its excerpted limits include a 64 MiB request cap, message truncation thresholds, and signature-header formats that do not support GitHub's or Stripe's built-in formats through the custom setting. The text also covers Kafka size and authentication requirements, recovery, deduplication, and alternate routes.

### Source excerpt

Send webhooks to Kafka with a webhook source connector, an HTTP gateway, AWS Lambda or a managed pipeline. Compare signatures, retries and duplicates.

## S3 to Kafka: How to Turn New S3 Files into Kafka Messages

DevFeed: [S3 to Kafka: How to Turn New S3 Files into Kafka Messages](<https://devfeed.tech/articles/s3-to-kafka-how-to-turn-new-s3-files-into-kafka-messages-121707.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/s3-to-kafka>)

Author: Seán Whelan

Published: 2026-11-06T12:27:02Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [parquet](<https://devfeed.tech/topics/parquet.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [compare](<https://devfeed.tech/tags/compare.md>), [files](<https://devfeed.tech/tags/files.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [managed](<https://devfeed.tech/tags/managed.md>), [messages](<https://devfeed.tech/tags/messages.md>), [ordering](<https://devfeed.tech/tags/ordering.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [s3](<https://devfeed.tech/tags/s3.md>), [setup](<https://devfeed.tech/tags/setup.md>), [source](<https://devfeed.tech/tags/source.md>), [spark](<https://devfeed.tech/tags/spark.md>)

### AI overview

S3 files can become Kafka messages through Kafka Connect, S3 event notifications with Lambda, Spark or Flink, or a managed pipeline. The main differences are how each discovers and tracks files, supported parsing and transformation, and setup; most approaches deliver at least once, so consumers may need to handle duplicates. Listing tools can miss late-arriving keys when they track only the last key, while re-reading modified files can produce duplicates. The guide details S3 and Kafka permissions, format and message-size constraints, and operational tradeoffs. Estuary's described route lists a bucket prefix, parses new files into a durable collection, then materializes records to Kafka; it runs every five minutes by default, delivers at least once, and has specific Parquet size, field truncation, schema-subject, authentication, and deletion limitations. Event-triggered Lambda is positioned for a modest number of uploads needing quick reaction, and Spark or Flink for files requiring transformation.

### Source excerpt

Move S3 data into Kafka with an S3 source connector, AWS Lambda, Spark or a managed pipeline. Compare file ordering, formats, duplicates and setup.

## Kafka to ClickHouse: Streaming Ingestion, Duplicates & Tombstones Guide

DevFeed: [Kafka to ClickHouse: Streaming Ingestion, Duplicates & Tombstones Guide](<https://devfeed.tech/articles/kafka-to-clickhouse-streaming-ingestion-duplicates-tombstones-guide-121703.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/kafka-to-clickhouse>)

Author: Seán Whelan

Published: 2026-11-06T12:26:41Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [self-hosted databases vs managed services](<https://devfeed.tech/topics/self-hosted-databases-vs-managed-services.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>)

Tags: [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [compare](<https://devfeed.tech/tags/compare.md>), [connect](<https://devfeed.tech/tags/connect.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [engine](<https://devfeed.tech/tags/engine.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [guide](<https://devfeed.tech/tags/guide.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [stream](<https://devfeed.tech/tags/stream.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [table](<https://devfeed.tech/tags/table.md>)

### AI overview

Kafka ingestion into ClickHouse is generally at least once, so replayed messages can create duplicates; reliable designs need batching and tables suited to event or keyed data. The article compares the Kafka table engine, ClickPipes, the Kafka Connect sink, and managed pipelines: the engine runs consumers on ClickHouse servers, ClickPipes is recommended for ClickHouse Cloud, and the sink fits teams already running Kafka Connect. For event streams, store partition and offset in a ReplacingMergeTree sorting key; keyed topics with tombstones need ReplacingMergeTree keyed by message key and queries with FINAL. Tombstone support differs: ClickPipes can delete rows in exactly-once mode or mark them deleted, while the Connect sink lists deletes as unsupported and the table engine's documentation does not describe null-value messages. Retention limits recovery, and compaction can remove tombstones after the configured retention period. Large, infrequent inserts reduce part pressure; monitor lag and active parts. The article also describes Estuary capture and materialization, which keeps Kafka progress in its own checkpoints and writes to ClickHouse on a configurable schedule.

### Source excerpt

Stream Kafka to ClickHouse with the Kafka table engine, ClickPipes or Kafka Connect. Compare 4 methods, delivery guarantees and duplicate handling.

## What's New in Q4'26 Confluent Data Streaming Platform: The Most Complete, Performant, and Open Foundation for Streamhouse

DevFeed: [What's New in Q4'26 Confluent Data Streaming Platform: The Most Complete, Performant, and Open Foundation for Streamhouse](<https://devfeed.tech/articles/what-s-new-in-q4-26-confluent-data-streaming-platform-the-most-complete-performant-and-open-foundation-for-streamhouse-117672.md>)

Original publisher: [Read original article](<https://www.confluent.io/blog/2026-q4-confluent-bundle-launch/>)

Author: Jason Lauritzen

Published: 2026-11-04T15:55:22Z

Content type: release

Language: en

Sources: [Confluent: Data in motion](<https://devfeed.tech/sources/confluent-data-in-motion.md>)

Topics: [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Confluent Platform](<https://devfeed.tech/topics/confluent-platform.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [private cloud](<https://devfeed.tech/topics/private-cloud.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [cloud](<https://devfeed.tech/tags/cloud.md>), [complete](<https://devfeed.tech/tags/complete.md>), [confluent](<https://devfeed.tech/tags/confluent.md>), [confluent-cloud](<https://devfeed.tech/tags/confluent-cloud.md>), [confluent-platform](<https://devfeed.tech/tags/confluent-platform.md>), [cpc](<https://devfeed.tech/tags/cpc.md>), [data](<https://devfeed.tech/tags/data.md>), [data-streaming](<https://devfeed.tech/tags/data-streaming.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [google](<https://devfeed.tech/tags/google.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [inference](<https://devfeed.tech/tags/inference.md>), [lightning](<https://devfeed.tech/tags/lightning.md>), [managed](<https://devfeed.tech/tags/managed.md>), [open](<https://devfeed.tech/tags/open.md>), [performant](<https://devfeed.tech/tags/performant.md>), [platform](<https://devfeed.tech/tags/platform.md>), [q4](<https://devfeed.tech/tags/q4.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [what-s-new](<https://devfeed.tech/tags/what-s-new.md>)

### AI overview

Confluent's Q4'26 updates position its Data Streaming Platform as a Streamhouse foundation that connects live Kafka data to applications, AI agents and analytics. Lightning Tables, generally available on AWS, provide read-only SQL and API access to continuously updated tables, with stated freshness within hundreds of milliseconds and query latency of 10-100 ms at 1,000+ queries per second. Managed Model Inference is generally available for supported models in Amazon Bedrock, Vertex AI and Azure OpenAI, callable from Flink SQL without configuring credentials or serving infrastructure. Other updates include Tableflow support for Google Cloud and BigQuery with BigLake Metastore synchronization; isolated logical Kafka clusters on shared Confluent Private Cloud infrastructure; and Schema Registry checks intended to catch incompatible changes across Kafka, Flink and lakehouse consumers. The excerpt also describes Cloud disaster recovery and topic-based migration, private-cloud monitoring and security controls, and WarpStream improvements. Availability varies: some features are generally available, while others are early access or open preview; the source cautions that product direction,.

### Source excerpt

Learn about Lightning Tables, Managed Model Inference, multi-tenancy for CPC, Tableflow for Google Cloud, and other Confluent Cloud, Confluent Platform, and WarpStream updates.

## A Tale of Two Houses: A Comparison Between Streamhouse and Lakehouse

DevFeed: [A Tale of Two Houses: A Comparison Between Streamhouse and Lakehouse](<https://devfeed.tech/articles/a-tale-of-two-houses-a-comparison-between-streamhouse-and-lakehouse-117673.md>)

Original publisher: [Read original article](<https://www.confluent.io/blog/a-tale-of-two-houses-streamhouse-lakehouse/>)

Author: Alex Stuart

Published: 2026-11-04T15:53:00Z

Content type: comparison

Language: en

Sources: [Confluent: Data in motion](<https://devfeed.tech/sources/confluent-data-in-motion.md>)

Topics: [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [access-control](<https://devfeed.tech/tags/access-control.md>), [account](<https://devfeed.tech/tags/account.md>), [acid](<https://devfeed.tech/tags/acid.md>), [act](<https://devfeed.tech/tags/act.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [compare](<https://devfeed.tech/tags/compare.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [confluent-cloud](<https://devfeed.tech/tags/confluent-cloud.md>), [data](<https://devfeed.tech/tags/data.md>), [houses](<https://devfeed.tech/tags/houses.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [learn](<https://devfeed.tech/tags/learn.md>), [one](<https://devfeed.tech/tags/one.md>), [other](<https://devfeed.tech/tags/other.md>)

### AI overview

A Streamhouse captures, validates, enriches, and governs each change as it happens, then serves current data to operational applications and agents in the formats they need. A lakehouse stores historical data in object storage with open table formats and supports large batch queries, business intelligence, and model training. Their latency and reliability needs differ: Streamhouse outages can stop production workflows, while lakehouse delays typically make reports or models stale. For customer 360 use cases, the text assigns long-range analysis to the lakehouse and time-sensitive questions--such as fraud checks or live personalization--to the Streamhouse. The two can work together: the Streamhouse shares curated Iceberg or Delta tables for lakehouse use, avoiding a separate ingestion job. The text presents them as complementary architectures, with the Streamhouse for current operational context and the lakehouse for historical analysis.

### Source excerpt

Compare streamhouse and lakehouse data architectures, learn which workloads each one fits and how they complement each other.

## Your Business, Always Current. Streamhouse - The Data Architecture for the Age of AI.

DevFeed: [Your Business, Always Current. Streamhouse - The Data Architecture for the Age of AI.](<https://devfeed.tech/articles/your-business-always-current-streamhouse-the-data-architecture-for-the-age-of-ai-117675.md>)

Original publisher: [Read original article](<https://www.confluent.io/blog/streamhouse-your-business-always-current/>)

Author: Yashwanth Dasari

Published: 2026-11-04T15:52:47Z

Content type: article

Language: en

Sources: [Confluent: Data in motion](<https://devfeed.tech/sources/confluent-data-in-motion.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>), [data-governance](<https://devfeed.tech/topics/data-governance.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [business logic](<https://devfeed.tech/topics/business-logic.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [about](<https://devfeed.tech/tags/about.md>), [account](<https://devfeed.tech/tags/account.md>), [age-of-ai](<https://devfeed.tech/tags/age-of-ai.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [applications](<https://devfeed.tech/tags/applications.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [business](<https://devfeed.tech/tags/business.md>), [confluent-cloud](<https://devfeed.tech/tags/confluent-cloud.md>), [context](<https://devfeed.tech/tags/context.md>), [data](<https://devfeed.tech/tags/data.md>), [data-architecture](<https://devfeed.tech/tags/data-architecture.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [making](<https://devfeed.tech/tags/making.md>), [processes](<https://devfeed.tech/tags/processes.md>), [state](<https://devfeed.tech/tags/state.md>), [systems](<https://devfeed.tech/tags/systems.md>)

### AI overview

A Streamhouse is presented as an architecture for keeping reusable business state and context current across distributed systems so applications, AI agents, and analytics can act on it. It combines historical data with ongoing operational changes, maintains state incrementally, and supports replay, correction, and recomputation. Its three requirements are freshness with history, decentralized sharing without consolidating onto one platform, and mission-critical correctness, availability, security, recovery, and traceability. The described components include connectors, a durable event backbone, stateful processing, shared semantics and governance, and serving through streams, queries, APIs, or tables. Operational systems still authorize transactions; warehouses and lakehouses still support historical analysis and training. The architecture does not guarantee a globally synchronized "now," and implementations trade off latency, consistency, replay, cost, and operational complexity. The text gives examples including Siemens Healthineers, Neuron Systems, and Yelp, and presents Confluent's platform as one implementation; its claims about platform completeness are vendor claims.

### Source excerpt

A Streamhouse is a data architecture that continuously captures, processes, governs and serves current business state and context across distributed systems, making them available to applications, AI agents, and analytics.

## Graphiant Named Confluent 2027 Data Streaming Startup of the Year for the Americas

DevFeed: [Graphiant Named Confluent 2027 Data Streaming Startup of the Year for the Americas](<https://devfeed.tech/articles/graphiant-named-confluent-2027-data-streaming-startup-of-the-year-for-the-americas-117674.md>)

Original publisher: [Read original article](<https://www.confluent.io/blog/graphiant-named-confluent-2027-data-streaming-startup-of-the-year-for-the-americas/>)

Author: Tim Graczewski

Published: 2026-11-03T21:29:59Z

Content type: news

Language: en

Sources: [Confluent: Data in motion](<https://devfeed.tech/sources/confluent-data-in-motion.md>)

Topics: [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>)

Tags: [2020](<https://devfeed.tech/tags/2020.md>), [2025](<https://devfeed.tech/tags/2025.md>), [about](<https://devfeed.tech/tags/about.md>), [adoption](<https://devfeed.tech/tags/adoption.md>), [amer](<https://devfeed.tech/tags/amer.md>), [americas](<https://devfeed.tech/tags/americas.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [confluent](<https://devfeed.tech/tags/confluent.md>), [data](<https://devfeed.tech/tags/data.md>), [data-streaming](<https://devfeed.tech/tags/data-streaming.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [network](<https://devfeed.tech/tags/network.md>), [network-observability](<https://devfeed.tech/tags/network-observability.md>), [observability](<https://devfeed.tech/tags/observability.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [real-time-data](<https://devfeed.tech/tags/real-time-data.md>), [real-time-data-streaming](<https://devfeed.tech/tags/real-time-data-streaming.md>), [startup](<https://devfeed.tech/tags/startup.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [uses](<https://devfeed.tech/tags/uses.md>)

### AI overview

Graphiant, named Confluent's 2027 Americas Data Streaming Startup of the Year, uses Apache Kafka on Confluent Cloud to stream telemetry and performance metrics from network edge devices. The data supports network health monitoring, traffic analysis, alerting, and customer dashboards, with analytics flowing to Apache Pinot through StarTree and to TimescaleDB. Confluent Cloud handles variable workloads as the network grows, reducing the need for Graphiant to operate Kafka infrastructure. Graphiant currently manages its own Apache Flink environment and plans to evaluate serverless Apache Flink on Confluent Cloud, as well as adopt Schema Registry to strengthen data management. The excerpt also describes a possible longer-term Streamhouse architecture and potential AI-driven network operations; these are future possibilities, not stated current capabilities.

### Source excerpt

Graphiant is Confluent's 2027 Americas Startup of the Year. Discover how it uses Apache Kafka and real-time data streaming for network observability.

## Introducing queue consumption mode for Apache Kafka in AWS Lambda

DevFeed: [Introducing queue consumption mode for Apache Kafka in AWS Lambda](<https://devfeed.tech/articles/introducing-queue-consumption-mode-for-apache-kafka-in-aws-lambda-114289.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/compute/introducing-queue-consumption-mode-for-apache-kafka-in-aws-lambda/>)

Author: Vaibhav Jain

Published: 2026-11-02T21:23:46Z

Content type: release

Language: en

Sources: [AWS Compute Blog](<https://devfeed.tech/sources/aws-compute-blog.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [AWS Lambda](<https://devfeed.tech/topics/aws-lambda.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>)

Tags: [amazon-cloudwatch-logs](<https://devfeed.tech/tags/amazon-cloudwatch-logs.md>), [amazon-sns](<https://devfeed.tech/tags/amazon-sns.md>), [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [automatic](<https://devfeed.tech/tags/automatic.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cdk](<https://devfeed.tech/tags/aws-cdk.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [blocking](<https://devfeed.tech/tags/blocking.md>), [broker](<https://devfeed.tech/tags/broker.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [event-source](<https://devfeed.tech/tags/event-source.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>), [partition](<https://devfeed.tech/tags/partition.md>), [queue](<https://devfeed.tech/tags/queue.md>), [retries](<https://devfeed.tech/tags/retries.md>), [scale](<https://devfeed.tech/tags/scale.md>), [share](<https://devfeed.tech/tags/share.md>), [visibility](<https://devfeed.tech/tags/visibility.md>)

### AI overview

AWS Lambda's Queue consumption mode uses Kafka Share Groups on Apache Kafka 4.2 or later to process records from a partition concurrently, removing partition count as the concurrency ceiling. Unlike Stream mode, which preserves per-partition ordering, Queue mode has no ordering guarantees and suits independent records. It uses per-record acknowledgment, visibility locks renewed during processing, and retries after failures; duplicate processing can occur, so functions should be idempotent. Queue mode is available on self-managed Kafka ESMs, including compatible managed Kafka offerings. On-demand scaling reaches five pollers per mapping (up to 5 MB/s or 50 concurrent invocations) and adjusts within three minutes; provisioned mode offers configurable poller capacity for higher throughput or stricter latency needs, subject to broker share-group limits. Queue mode does not support batching windows or starting-position parameters.

### Source excerpt

AWS Lambda event source mappings now support queue consumption mode for Apache Kafka. Using Kafka Share Groups, you can scale consumer parallelism independently of partition count, with per-record acknowledgment, visibility timeouts, and automatic retries on self-managed Kafka 4.2 or later clusters.

## Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg

DevFeed: [Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg](<https://devfeed.tech/articles/migrate-google-bigquery-to-an-open-lakehouse-on-aws-with-apache-iceberg-114202.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/migrate-google-bigquery-to-an-open-lakehouse-on-aws-with-apache-iceberg/>)

Author: Saman Irfan

Published: 2026-11-02T17:21:17Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [AWS Lake Formation](<https://devfeed.tech/topics/aws-lake-formation.md>), [Amazon Bedrock Knowledge Bases](<https://devfeed.tech/topics/amazon-bedrock-knowledge-bases.md>), [AWS Step Functions](<https://devfeed.tech/topics/aws-step-functions.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [data](<https://devfeed.tech/tags/data.md>), [design](<https://devfeed.tech/tags/design.md>), [future](<https://devfeed.tech/tags/future.md>), [google](<https://devfeed.tech/tags/google.md>), [google-bigquery](<https://devfeed.tech/tags/google-bigquery.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [migrate](<https://devfeed.tech/tags/migrate.md>), [migration](<https://devfeed.tech/tags/migration.md>), [open](<https://devfeed.tech/tags/open.md>), [optimize](<https://devfeed.tech/tags/optimize.md>), [over](<https://devfeed.tech/tags/over.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

The article presents a phased path from BigQuery to an AWS lakehouse using Apache Iceberg tables on Amazon S3, with governance through AWS Glue Data Catalog and Lake Formation and engines selected by workload. It recommends assessing tables, query complexity, consumers, volumes, and access controls; designing storage, partitioning, ingestion, and governance; then converting a representative slice and validating results, performance, and cost. For bulk transfer, export BigQuery tables to Parquet in Google Cloud Storage, move them to S3 with AWS DataSync, create or register Iceberg tables, and synchronize incremental changes during transition. Cutover includes parallel validation and deciding whether each consumer will reconnect, be rebuilt, or relay data. The article distinguishes managed S3 Tables, which handle maintenance automatically, from self-managed Iceberg on general-purpose S3, which requires tuning. It also cautions that BigQuery scan-based and Redshift compute-based pricing are not directly comparable, recommending workload-based benchmarks and monthly TCO estimates. Post-migration steps include query and cost optimization.

### Source excerpt

Learn how to migrate Google BigQuery workloads to an open lakehouse on AWS built on Apache Iceberg and Amazon S3. This post explains why an open lakehouse future-proofs your data, presents the target architecture on AWS, and walks through a step-by-step migration: assess, design, convert, migrate, cut over, and optimize.

## Message Queues

DevFeed: [Message Queues](<https://devfeed.tech/articles/message-queues-111142.md>)

Original publisher: [Read original article](<https://blog.alexewerlof.com/p/message-queues>)

Author: Alex Ewerlöf

Published: 2026-11-02T05:30:53Z

Content type: article

Language: en

Sources: [Alex Ewerlof Notes](<https://devfeed.tech/sources/alex-ewerlof-notes.md>)

Topics: [Messaging](<https://devfeed.tech/topics/messaging.md>), [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [decoupling](<https://devfeed.tech/topics/decoupling.md>), [backpressure](<https://devfeed.tech/topics/backpressure.md>), [scalable-architecture](<https://devfeed.tech/topics/scalable-architecture.md>), [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [about](<https://devfeed.tech/tags/about.md>), [admin](<https://devfeed.tech/tags/admin.md>), [ads](<https://devfeed.tech/tags/ads.md>), [ai](<https://devfeed.tech/tags/ai.md>), [alerting](<https://devfeed.tech/tags/alerting.md>), [alerts](<https://devfeed.tech/tags/alerts.md>), [all](<https://devfeed.tech/tags/all.md>), [allows](<https://devfeed.tech/tags/allows.md>), [also](<https://devfeed.tech/tags/also.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [message](<https://devfeed.tech/tags/message.md>), [message-queues](<https://devfeed.tech/tags/message-queues.md>), [queues](<https://devfeed.tech/tags/queues.md>), [resilient-architecture](<https://devfeed.tech/tags/resilient-architecture.md>)

### AI overview

Message queues let producers hand work to separate consumers, shortening user-facing waits, isolating worker failures, and buffering traffic spikes. Consumers acknowledge completed messages; if a worker fails before acknowledging, messages can be redelivered, so at-least-once delivery may cause duplicates and works best with idempotent processing. The text recommends queues for asynchronous work, bursty traffic, decoupled scaling, and reliability, while warning they add latency to time-sensitive synchronous workloads. It explains partitioning, ordering versus even workload distribution, back pressure, retry limits and dead letter queues, and durable storage with replay. Kafka retains messages for a configured period and scales through partitions and consumer groups; SQS is managed and offers Standard queues with best-effort ordering or FIFO queues with strict ordering and lower throughput; RabbitMQ uses exchanges and bindings for complex routing.

### Source excerpt

Part of resilient architecture patterns series

## Milvus Source Connector for Debezium: Enabling Event-Driven Vector Architectures

DevFeed: [Milvus Source Connector for Debezium: Enabling Event-Driven Vector Architectures](<https://devfeed.tech/articles/milvus-source-connector-for-debezium-enabling-event-driven-vector-architectures-111256.md>)

Original publisher: [Read original article](<https://debezium.io/blog/2026/11/02/milvus-source-connector-for-debezium/>)

Author: Kartik Angiras

Published: 2026-11-02T00:00:00Z

Content type: article

Language: en

Sources: [Debezium](<https://devfeed.tech/sources/debezium-io.md>)

Topics: [Debezium Connector](<https://devfeed.tech/topics/debezium-connector.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [event-sourcing](<https://devfeed.tech/topics/event-sourcing.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [hybrid-search](<https://devfeed.tech/topics/hybrid-search.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [debezium](<https://devfeed.tech/tags/debezium.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [event](<https://devfeed.tech/tags/event.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [redis](<https://devfeed.tech/tags/redis.md>), [source](<https://devfeed.tech/tags/source.md>), [stream](<https://devfeed.tech/tags/stream.md>), [vector](<https://devfeed.tech/tags/vector.md>), [vector-database](<https://devfeed.tech/tags/vector-database.md>)

### AI overview

The Debezium 3.7 Milvus source connector reads Milvus 2.5's Kafka write stream, uses etcd checkpoints and Milvus gRPC for snapshots and schemas, and emits inserts and deletes as Debezium events with typed vector fields. Timetick watermarks restore commit order across virtual channels, and Kafka Connect sinks can deliver changes to systems such as Postgres, Elasticsearch, or S3. The connector requires a self-hosted deployment using Kafka as its broker; Milvus Lite, Pulsar, RocksMQ, and Milvus 2.6 are unsupported. It is incubating: its default checkpoint key can miss changes at snapshot handoff, and a crash while events await ordering can lose them. It uses at-least-once snapshot delivery, so idempotent sinks are important; deletes have only primary-key before images, and upserts appear as delete/insert pairs.

### Source excerpt

Google Summer of Code 2026 with the Debezium community was the best learning experience I have had as an engineer. Over one summer I built a Kafka Connect source connector from scratch for a database with no documented way to read its changes. Most of the work was taking the database apart to find out where those changes live, writing the design down before building any of it, and carrying each piece through review with the Debezium maintainers. The connector shipped in Debezium 3.7. What came out of that summer is a Debezium source connector that turns Milvus, the open source vector database, into a first-class source of change events. I built it because Milvus has no change stream of its own. If you need its inserts and deletes somewhere else (Elasticsearch for keyword search, Redis as a cache, Postgres for reporting), you either write to both systems from the application and drift apart on the first partial failure, or you run a batch export that cannot see a delete. The connector reads the write stream Milvus already publishes to its message queue, puts the events back in commit order, and emits each insert and delete as a normal Debezium change event with the vector as a typed field. From there any Kafka Connect sink is a Milvus integration, and the service that writes to Milvus never has to know. This post covers how it works and what you can build once Milvus is one. Read more...

[Next page](<https://devfeed.tech/topics/apache-kafka.md?cursor=WyIyMDI2LTExLTAyVDAwOjAwOjAwKzAwOjAwIiwgImNlMThmNjc3LTExMGMtNDE5Yy1hY2U4LTdlOWQzM2FhN2U0OSJd>)