# Apache-Kafka

Apache Kafka is an open-source distributed event streaming platform used for data pipelines, streaming analytics, data integration, and mission-critical applications.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Why Kafka retention.ms can delay message deletion

DevFeed: [Why Kafka retention.ms can delay message deletion](<https://devfeed.tech/articles/why-your-kafka-topic-ignores-retention-ms-and-how-to-fix-it-31403.md>)

Original publisher: [Read original article](<https://developers.redhat.com/articles/2026/09/16/why-your-kafka-topic-ignores-retentionms-and-how-fix-it>)

Author: Rogerio Santos

Published: 2026-09-16T13:05:06Z

Content type: tutorial

Language: en

Sources: [Red Hat](<https://devfeed.tech/sources/red-hat.md>), [Red Hat Developer](<https://devfeed.tech/sources/red-hat-developer.md>)

Topics: [Kafka](<https://devfeed.tech/topics/kafka.md>), [Apache-Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [configuration](<https://devfeed.tech/topics/configuration.md>), [file](<https://devfeed.tech/topics/file.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>)

Tags: [broker](<https://devfeed.tech/tags/broker.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [diagnose](<https://devfeed.tech/tags/diagnose.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [lifecycle](<https://devfeed.tech/tags/lifecycle.md>), [retention](<https://devfeed.tech/tags/retention.md>), [storage](<https://devfeed.tech/tags/storage.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>)

### AI overview

This guide explains why Kafka messages can remain readable beyond a topic's retention.ms setting. Kafka deletes closed log segments rather than individual messages, and the active segment and continuous writes can delay removal of older records.

### Source excerpt

A customer opened a support case with a deceptively simple complaint: a Kafka topic was configured with a 12-hour retention (retention.ms), yet messages produced on July 24 were still readable 4 days later, on July 28. Nothing was broken. The broker logged no errors. The retention policy was working as designed, but the segment layout and continuous write pattern delayed when the old records could actually be removed. The post Why your Kafka topic ignores retention.ms (and how to fix it) appeared first on Red Hat Developer.

## Kafka's log compaction corrupts data. Here's how we fixed it

DevFeed: [Kafka's log compaction corrupts data. Here's how we fixed it](<https://devfeed.tech/articles/kafka-s-log-compaction-corrupts-data-here-s-how-we-fixed-it-12713.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/kafka-log-compaction-bug-fix-streaming>)

Author: Alexey Bashtanov

Published: 2026-06-25T00:00:00Z

Content type: article

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [Kafka](<https://devfeed.tech/topics/kafka.md>), [Apache-Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [bug](<https://devfeed.tech/topics/bug.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Testing](<https://devfeed.tech/topics/testing.md>)

Tags: [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [bug](<https://devfeed.tech/tags/bug.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

The article examines a bug in Apache Kafka log compaction that can cause replicas to permanently disagree about retained records. It describes how tombstones and transaction control batches may be removed independently on different brokers, allowing deleted data to reappear or aborted data to be served as committed, and presents coordinated compaction in Redpanda Streaming as the solution.

### Source excerpt

There's a problem with Apache Kafka's log compaction. Here's what we found, how to reproduce it, and how we solved it in Redpanda.

## Event-Driven Architecture with Apache Kafka

DevFeed: [Event-Driven Architecture with Apache Kafka](<https://devfeed.tech/articles/mastering-event-driven-architecture-with-apache-kafka-39559.md>)

Original publisher: [Read original article](<https://ankit-rana.com/logs/07-kafka-event-driven-architecture/>)

Author: hello@ankit-rana.com

Published: 2026-03-16T00:00:00Z

Content type: tutorial

Language: en

Sources: [Ankit Rana | Mechanical Sympathy](<https://devfeed.tech/sources/ankit-rana-mechanical-sympathy.md>)

Topics: [event driven](<https://devfeed.tech/topics/event-driven.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Apache-Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Scalability](<https://devfeed.tech/topics/scalability.md>), [Resilience](<https://devfeed.tech/topics/resilience.md>), [streaming-data-processing](<https://devfeed.tech/topics/streaming-data-processing.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [systems](<https://devfeed.tech/topics/systems.md>)

Tags: [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [consumer](<https://devfeed.tech/tags/consumer.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [event-driven-architecture](<https://devfeed.tech/tags/event-driven-architecture.md>), [event-sourcing](<https://devfeed.tech/tags/event-sourcing.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [microservices](<https://devfeed.tech/tags/microservices.md>), [partitions](<https://devfeed.tech/tags/partitions.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [resilience](<https://devfeed.tech/tags/resilience.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [schemas](<https://devfeed.tech/tags/schemas.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>)

### AI overview

This tutorial explains event-driven architecture and how Apache Kafka supports asynchronous, real-time data processing. It covers producers, consumers, immutable events, event sourcing, scalability, resilience, stream-processing pipelines, and challenges such as ordering, debugging, and eventual consistency.

### Source excerpt

Event-driven architecture replaces synchronous point-to-point calls with immutable events on a durable log, so producers and consumers scale and fail independently. Kafka provides that log: topics sharded into ordered append-only partitions, replicated across brokers, with consumer groups sharing partitions to scale read throughput.

## FIPS-ing the Un-FIPS-able: Apache Kafka

DevFeed: [FIPS-ing the Un-FIPS-able: Apache Kafka](<https://devfeed.tech/articles/fips-ing-the-un-fips-able-apache-kafka-13044.md>)

Original publisher: [Read original article](<https://www.chainguard.dev/unchained/fips-ing-the-un-fips-able-apache-kafka>)

Published: 2026-03-05T00:00:00Z

Content type: release

Language: en

Sources: [Chainguard: Unchained](<https://devfeed.tech/sources/chainguard-unchained.md>)

Topics: [Apache-Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [chainguard](<https://devfeed.tech/topics/chainguard.md>), [Cryptography](<https://devfeed.tech/topics/cryptography.md>), [Security](<https://devfeed.tech/topics/security.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [TLS (Transport Layer Security)](<https://devfeed.tech/topics/tls.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [apache](<https://devfeed.tech/tags/apache.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [apache-kafka-fips](<https://devfeed.tech/tags/apache-kafka-fips.md>), [chainguard](<https://devfeed.tech/tags/chainguard.md>), [chainguard-containers](<https://devfeed.tech/tags/chainguard-containers.md>), [chainguard-images](<https://devfeed.tech/tags/chainguard-images.md>), [chainguard-kafka-container](<https://devfeed.tech/tags/chainguard-kafka-container.md>), [chainguard-kafka-image](<https://devfeed.tech/tags/chainguard-kafka-image.md>), [cryptography](<https://devfeed.tech/tags/cryptography.md>), [event-streaming](<https://devfeed.tech/tags/event-streaming.md>), [fips](<https://devfeed.tech/tags/fips.md>), [fips-140-3](<https://devfeed.tech/tags/fips-140-3.md>), [fips-containers](<https://devfeed.tech/tags/fips-containers.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [tls](<https://devfeed.tech/tags/tls.md>)

### AI overview

Chainguard announces FIPS-validated images for Apache Kafka, describing targeted source changes, validated cryptographic providers, TLS and keystore requirements, testing, and image maintenance for regulated environments.

### Source excerpt

Chainguard delivers the FIPS-validated Apache Kafka images, enabling secure event streaming for FedRAMP and regulated environments.

## Integrating Grafbase and Redpanda for seamless real-time features

DevFeed: [Integrating Grafbase and Redpanda for seamless real-time features](<https://devfeed.tech/articles/integrating-grafbase-and-redpanda-for-seamless-real-time-features-12707.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/integrating-grafbase-real-time-features>)

Author: Chandler Mayo

Published: 2025-09-23T00:00:00Z

Content type: tutorial

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [Kafka](<https://devfeed.tech/topics/kafka.md>), [GraphQL](<https://devfeed.tech/topics/graphql.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Apache-Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [Extension](<https://devfeed.tech/topics/extension.md>), [configuration](<https://devfeed.tech/topics/configuration.md>), [API](<https://devfeed.tech/topics/api.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [Command-line interface](<https://devfeed.tech/topics/cli.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [apis](<https://devfeed.tech/tags/apis.md>), [cli](<https://devfeed.tech/tags/cli.md>), [config](<https://devfeed.tech/tags/config.md>), [extensions](<https://devfeed.tech/tags/extensions.md>), [graphql](<https://devfeed.tech/tags/graphql.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>)

### AI overview

This tutorial explains how to integrate Apache Kafka with a federated GraphQL API using Grafbase Extensions and Redpanda. It covers declarative configuration for publishing and subscribing to event streams without deploying additional subgraph servers or writing glue code.

### Source excerpt

Grafbase launched native support for Kafka through Grafbase Extensions and Redpanda is the ideal streaming engine to power it. Here's how to integrate them.

## A Deep Dive Into Ingesting Debezium Events From Kafka With Flink SQL

DevFeed: [A Deep Dive Into Ingesting Debezium Events From Kafka With Flink SQL](<https://devfeed.tech/articles/a-deep-dive-into-ingesting-debezium-events-from-kafka-with-flink-sql-18833.md>)

Original publisher: [Read original article](<https://www.morling.dev/blog/ingesting-debezium-events-from-kafka-with-flink-sql/>)

Published: 2025-04-16T09:25:00Z

Content type: tutorial

Language: en

Sources: [Gunnar Morling](<https://devfeed.tech/sources/gunnar-morling.md>)

Topics: [apache-flink](<https://devfeed.tech/topics/apache-flink.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Apache-Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [changelog](<https://devfeed.tech/tags/changelog.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [data](<https://devfeed.tech/tags/data.md>), [debezium](<https://devfeed.tech/tags/debezium.md>), [flink](<https://devfeed.tech/tags/flink.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [schema](<https://devfeed.tech/tags/schema.md>), [streams](<https://devfeed.tech/tags/streams.md>)

### AI overview

This tutorial explains how to ingest Debezium change events from Kafka into Apache Flink using Flink SQL. It compares the Apache Kafka SQL Connector and Upsert Kafka SQL Connector, discusses JSON and Avro formats, and explains how connector and format choices determine append-only or changelog semantics.

### Source excerpt

Table of Contents Flink SQL Connectors for Apache Kafka The Apache Kafka SQL Connector in Append-Only Mode The Apache Kafka SQL Connector As a Changelog Source The Upsert Kafka SQL Connector Summary Over the years, I've spoken quite a bit about the use cases for processing Debezium data change events with Apache Flink, such as metadata enrichment, building denormalized data views, and creating data contracts for your CDC streams. One detail I haven't covered in depth so far is how to actually ingest Debezium change events from a Kafka topic into Flink, in particular via Flink SQL. Several connectors and data formats exist for this, which can make things somewhat confusing at first. So let's dive into the different options and the considerations around them!

## Kafka for System Design Interviews

DevFeed: [Kafka for System Design Interviews](<https://devfeed.tech/articles/kafka-for-system-design-interviews-32314.md>)

Original publisher: [Read original article](<https://evanking1.medium.com/kafka-for-system-design-interviews-f031ac05d186?source=rss-9736778727ef------2>)

Author: Evan King

Published: 2024-07-01T03:18:09Z

Content type: tutorial

Language: en

Sources: [Evan King](<https://devfeed.tech/sources/evan-king.md>)

Topics: [Kafka](<https://devfeed.tech/topics/kafka.md>), [Apache-Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Scalability](<https://devfeed.tech/topics/scalability.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [durability](<https://devfeed.tech/tags/durability.md>), [google](<https://devfeed.tech/tags/google.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [meta](<https://devfeed.tech/tags/meta.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>), [system-design-interview](<https://devfeed.tech/tags/system-design-interview.md>)

### AI overview

A tutorial on Apache Kafka for system design interviews. It explains Kafka as an open-source distributed event streaming platform that can function as a message queue or stream processing system, then uses a real-time sports statistics example to discuss producers, consumers, scaling, and event ordering.

### Source excerpt

By the Co-founder of www.hellointerview.comIntro There is a good chance you've heard of Kafka. It's popular. In fact, according to their website, it's used by 80% of the Fortune 100. If it's good enough to help scale the largest companies in the world, it's probably good enough for your next system design interview. In fact, it's one of the top 5 technologies we see used in design interviews. Apache Kafka is an open-source distributed event streaming platform that can be used either as a message queue or as a stream processing system. Kafka excels in delivering high performance, scalability, and durability. It's engineered to handle vast volumes of data in real-time, ensuring that no message is ever lost and that each piece of data is processed as swiftly as possible. In this deep dive, we're going to take a top down approach. Starting with a zoomed out view of Kafka and progressing into more and more detail. If you know the basics, feel free to skip ahead to the more advanced sections. A Motivating Example It's the World Cup (my personal favorite competition). And we run a website that provides real-time statistics on the matches. Each time a goal is scored, a player is booked, or a substitution is made, we want to update our website with the latest information. Events are placed on a queue when they occur. We call the server or process responsible for putting these events on the queue the producer. Downstream, we have a server that reads events off the queue and updates the website. We call this the consumer. Now, imagine the World Cup expanded from just the top 48 teams to a hypothetical 1,000-team tournament, and all the games are now played at the same time. The number of events has increased significantly, and our single server hosting the queue is struggling to keep up. Similarly, our consumer feels like it has its mouth under a firehose and is crashing under the load. We need to scale the system by adding more servers to distribute our queue. But how do we e

## Testing Kafka Connectors

DevFeed: [Testing Kafka Connectors](<https://devfeed.tech/articles/testing-kafka-connectors-18876.md>)

Original publisher: [Read original article](<https://www.morling.dev/blog/testing-kafka-connectors/>)

Published: 2022-08-25T08:20:00Z

Content type: tutorial

Language: en

Sources: [Gunnar Morling](<https://devfeed.tech/sources/gunnar-morling.md>)

Topics: [Kafka](<https://devfeed.tech/topics/kafka.md>), [Apache-Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Testing](<https://devfeed.tech/topics/testing.md>), [Unit testing](<https://devfeed.tech/topics/unit-testing.md>)

Tags: [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [testing](<https://devfeed.tech/tags/testing.md>), [unit-test](<https://devfeed.tech/tags/unit-test.md>), [unit-testing](<https://devfeed.tech/tags/unit-testing.md>)

### AI overview

This article explains how to test custom Kafka Connect source connectors. It covers unit tests for connector classes and polling logic, then describes a testing harness that uses an external etcd system; similar approaches can be applied to sink connectors.

### Source excerpt

Table of Contents Unit Tests Integration Tests Wrap-Up Kafka Connect is a key factor for the wide-spread adoption of Apache Kafka: a framework and runtime environment for connectors, it makes the task of getting data either into Kafka or out of Kafka solely a matter of configuration, rather than a bespoke programming job. There's dozens, if not hundreds, of readymade source and sink connectors, allowing you to create no-code data pipelines between all kinds of databases, APIs, and other systems. There may be situations though where there is no existing connector matching your requirements, in which case you can implement your own custom connector using the Kafka Connect framework. Naturally, this raises the question of how to test such a Kafka connector, making sure it propagates the data between the connected external system and Kafka correctly and completely. In this blog post I'd like to focus on testing approaches for Kafka Connect source connectors, i.e. connectors like Debezium, which ingest data from an external system into Kafka. Very similar strategies can be employed for testing sink connectors, though.

## Tiny Letter from Kafka

DevFeed: [Tiny Letter from Kafka](<https://devfeed.tech/articles/tiny-letter-from-kafka-2169.md>)

Original publisher: [Read original article](<https://developers.soundcloud.com/blog//tiny-letter-from-kafka>)

Published: 2021-10-13T00:00:00Z

Content type: article

Language: en

Sources: [SoundCloud Backstage Blog](<https://devfeed.tech/sources/soundcloud-backstage-blog.md>)

Topics: [Apache-Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [article](<https://devfeed.tech/tags/article.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [event](<https://devfeed.tech/tags/event.md>), [event-streaming](<https://devfeed.tech/tags/event-streaming.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [messaging](<https://devfeed.tech/tags/messaging.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

This article introduces Apache Kafka as an open-source distributed event streaming and messaging platform. It explains the publish-subscribe messaging pattern and uses the story behind Kafka's name to illustrate the system's core flow, with a brief reference to how SoundCloud teams consume event data.

### Source excerpt

This article discusses the powerful design choice of Apache Kafka, "an open-source distributed event streaming platform," and gives a sneak...