# debezium Published articles for debezium. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## How INTEGER and INT Produced Different Schemas in Debezium DevFeed: [How INTEGER and INT Produced Different Schemas in Debezium]() Original publisher: [Read original article]() Author: Prathit Malik Published: 2026-09-02T07:02:02Z Content type: article Language: en Sources: [Grofers]() Topics: [MySQL](), [migration](), [Streaming](), [Kafka](), [SQL](), [Databases]() Tags: [big-data](), [blinkit](), [change](), [database](), [debezium](), [jdbc](), [kafka](), [migration](), [mysql](), [open-source](), [schema](), [sql](), [streaming]() ### AI overview This article investigates a Debezium CDC pipeline failure caused by MySQL INT and INTEGER synonyms being treated as different types. The resulting schema mismatch produced an Integer where a downstream consumer expected a Long, causing a ClassCastException on every batch. It explains how streaming and snapshot schema handling differ and how routine migrations exposed the problem. ### Source excerpt A Debezium investigation: how MySQL synonyms INT and INTEGER were treated as different types. One of our CDC pipelines started failing with a ClassCastException on every batch. java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Long The pipeline was producing an Integer, but the downstream consumer expected a Long. Every run failed in the same way, which pointed us toward a schema mismatch rather than an issue with individual records. Background: how CDC works To see why a mismatch like that can hide for years, it helps to know how CDC actually works. Most companies replicate their transactional database (MySQL, Postgres, something similar) into a separate data lake for analytics, rather than querying the source directly, and Change Data Capture (CDC) is what keeps that copy in sync: it tails the database's transaction log and replays every insert, update, and delete downstream. Debezium is the most widely used open-source CDC tool for MySQL, and it builds a table's schema in one of two ways that are supposed to agree but do not always. Streaming mode: the first time it sees a CREATE TABLE or ALTER TABLE in the binlog, it parses the raw SQL text and writes the result to its own internal Kafka topic, database.history.kafka.topic. Every restart after that rebuilds the in-memory schema by replaying that topic, not by re-reading the binlog. Snapshot mode: reads the table definition fresh through MySQL's JDBC metadata interface, which normalizes types, every time it runs. Schema Registry sits downstream of both: each connector writes whatever schema it built into the registry, but neither connector reads its own schema back from it. Keep that in mind; it matters later. The trigger: a routine migration With that in mind, here's what actually happened to us. Rewind two years: one of our upstream service teams added a few columns to a source table as part of a standard schema change. ALTER TABLE ADD COLUMN length double NU ## Oracle CDC now available in Redpanda Connect DevFeed: [Oracle CDC now available in Redpanda Connect]() Original publisher: [Read original article]() Author: Prakhar Garg Published: 2026-04-09T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Databases](), [Streaming](), [Structured-data](), [Transactions](), [data](), [systems]() Tags: [announcement](), [data](), [debezium](), [event](), [jvm](), [oracle](), [redpanda-connect](), [schema](), [streaming](), [transactions]() ### AI overview Redpanda Connect v4.83.0 introduces a native Oracle CDC connector that captures inserts, updates, and deletes without a JVM, Kafka Connect cluster, or separate workers. It can snapshot existing tables, stream subsequent changes from Oracle redo logs, resume from checkpoints, and route schema-tracked Avro events to per-table Redpanda topics. ### Source excerpt Native CDC for Oracle enables real-time access without rearchitecting. No JVM, middleware, or operational overhead. ## Postgres Replication Slots: Confirmed Flush LSN vs. Restart LSN DevFeed: [Postgres Replication Slots: Confirmed Flush LSN vs. Restart LSN]() Original publisher: [Read original article]() Published: 2025-08-05T11:55:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [Replication](), [Databases](), [Transactions]() Tags: [capture](), [concurrent](), [database](), [debezium](), [postgres](), [recovery](), [replication](), [retention](), [stream](), [streaming](), [transactions](), [troubleshooting]() ### AI overview This tutorial explains the difference between PostgreSQL replication slots' confirmed_flush_lsn and restart_lsn attributes. It describes how consumers resume from replication slots, how concurrent transactions affect the oldest WAL that may still be required, and why understanding both LSNs helps with troubleshooting and WAL retention. ### Source excerpt Table of Contents confirmed_flush_sn: Tracking Consumer Progress restart_lsn: Handling Concurrent Transactions Mid-Transaction Recovery Looking Forward: Streaming In-Progress Transactions Replication slots in Postgres keep track of how far consumers have read a replication stream. After a restart, consumers--either Postgres read replicas or external tools for change data capture (CDC), like Debezium--resume reading from the last confirmed log sequence number (LSN) of their replication slot. The slot prevents the database from disposing of required log segments, allowing safe resumption after downtime. In this post, we are going to take a look at why Postgres replication slots don't have one but two LSN-related attributes: restart_lsn and confirmed_flush_lsn. Understanding the difference between the two is crucial for troubleshooting replication issues, optimizing WAL retention, and avoiding common pitfalls in production environments. ## Mastering Postgres Replication Slots: Preventing WAL Bloat and Other Production Issues DevFeed: [Mastering Postgres Replication Slots: Preventing WAL Bloat and Other Production Issues]() Original publisher: [Read original article]() Published: 2025-07-08T11:55:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [Replication](), [Databases](), [Monitoring]() Tags: [best-practices](), [database](), [debezium](), [failover](), [monitoring](), [postgres](), [production](), [replication]() ### AI overview This practical guide explains how to manage Postgres replication slots for Change Data Capture pipelines and reduce excessive write-ahead log retention. It covers decoding plug-ins, heartbeats, failover, publications, filtering, replica identity, monitoring, and removing unused slots, with examples based primarily on Debezium's Postgres connector. ### Source excerpt Table of Contents Use the pgoutput Logical Decoding Output Plug-in Define a Maximum Replication Slot Size Enable Heartbeats Use Table-level Publications Use Column and Row Filters Enable Fail-Over Slots Consider Using Replica Identity FULL Monitor, Monitor, Monitor! Drop Unused Replication Slots Summary Over the last couple of years, I've helped dozens of users and organizations to build Change Data Capture (CDC) pipelines for their Postgres databases. A key concern in that process is setting up and managing replication slots, which are Postgres' mechanism for making sure that any segments of the write-ahead log (WAL) of the database are kept around until they have been processed by registered replication consumers. When not being careful, a replication slot may cause unduly large amounts of WAL segments to be retained by the database. This post describes best practices helping to prevent this and other issues, discussing aspects like heartbeats, replication slot failover, monitoring, the management of Postgres publications, and more. While this is primarily based on my experience of using replication slots via Debezium's Postgres connector, the principles are generally applicable and are worth considering also when using other CDC tools for Postgres based on logical replication. ## Backfilling Postgres TOAST Columns in Debezium Data Change Events DevFeed: [Backfilling Postgres TOAST Columns in Debezium Data Change Events]() Original publisher: [Read original article]() Published: 2025-05-26T14:40:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Databases](), [Replication](), [apache-flink](), [stream-processing](), [SQL](), [API]() Tags: [apache-flink](), [data](), [debezium](), [debezium-connector](), [flink](), [postgres](), [replication](), [sql](), [stateful](), [stream-processing]() ### AI overview This article explains how Postgres logical replication handles TOAST columns in Debezium change events. It discusses Debezium's reselect postprocessor and explores Apache Flink solutions using stateful stream processing, Flink SQL, and process table functions. ### Source excerpt Table of Contents Debezium Reselect Postprocessor Flink DataStream API Flink SQL With OVER Aggregation Flink Process Table Functions Summary and Discussion Postgres logical replication, while powerful for capturing real-time data changes, presents challenges with TOAST columns, whose values can be absent from data change events in specific situations. This post discusses how Debezium addresses this through its built-in reselect post processor, then explores more robust solutions leveraging Apache Flink's capabilities for stateful stream processing, including Flink SQL and the brand-new process table functions (PTFs) in Flink 2.1. ## A Deep Dive Into Ingesting Debezium Events From Kafka With Flink SQL DevFeed: [A Deep Dive Into Ingesting Debezium Events From Kafka With Flink SQL]() Original publisher: [Read original article]() Published: 2025-04-16T09:25:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [apache-flink](), [Kafka](), [Apache-Kafka]() Tags: [changelog](), [connectors](), [data](), [debezium](), [flink](), [how-to](), [kafka](), [schema](), [streams]() ### AI overview This tutorial explains how to ingest Debezium change events from Kafka into Apache Flink using Flink SQL. It compares the Apache Kafka SQL Connector and Upsert Kafka SQL Connector, discusses JSON and Avro formats, and explains how connector and format choices determine append-only or changelog semantics. ### Source excerpt Table of Contents Flink SQL Connectors for Apache Kafka The Apache Kafka SQL Connector in Append-Only Mode The Apache Kafka SQL Connector As a Changelog Source The Upsert Kafka SQL Connector Summary Over the years, I've spoken quite a bit about the use cases for processing Debezium data change events with Apache Flink, such as metadata enrichment, building denormalized data views, and creating data contracts for your CDC streams. One detail I haven't covered in depth so far is how to actually ingest Debezium change events from a Kafka topic into Flink, in particular via Flink SQL. Several connectors and data formats exist for this, which can make things somewhat confusing at first. So let's dive into the different options and the considerations around them! ## Failover Replication Slots with Postgres 17 DevFeed: [Failover Replication Slots with Postgres 17]() Original publisher: [Read original article]() Published: 2024-12-03T00:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Replication](), [Availability](), [Database](), [data-platforms]() Tags: [availability](), [data](), [data-platforms](), [database](), [debezium](), [failover](), [read-replica](), [replication]() ### AI overview This article explains how Postgres 17 introduces built-in failover slots. Replication slots created on a primary are automatically propagated to read replicas, allowing consumers such as Debezium to continue after failover without missing change events or creating a new initial snapshot. ### Source excerpt Table of Contents Hello, Failover Slots! Failover Slots in Decodable Wrapping Up This post originally appeared on the Decodable blog. All rights reserved. Postgres read replicas are commonly used not only to distribute query load amongst multiple nodes, but also to ensure high availability (HA) of the database. If the primary node of a Postgres cluster fails, a read replica can be promoted to be the new primary, processing write (and read) requests from thereon. ## Thoughts On Moving Debezium to the Commonhaus Foundation DevFeed: [Thoughts On Moving Debezium to the Commonhaus Foundation]() Original publisher: [Read original article]() Published: 2024-11-27T16:25:00Z Content type: opinion Language: en Sources: [Gunnar Morling]() Topics: [Open Source](), [Databases](), [data]() Tags: [databases](), [debezium](), [open-source](), [thoughts]() ### AI overview An opinion article discusses Debezium's move to the Commonhaus Foundation, explaining its significance for the project, its community, and open-source change data capture. It also describes Debezium's community growth and Red Hat's historical role in controlling project assets. ### Source excerpt If you are following the news around Debezium--an open-source platform for Change Data Capture (CDC) for a variety of databases--you may have seen the announcement that the project is in the process of moving to the Commonhaus Foundation. I think this is excellent news for the Debezium project, its community, and open-source CDC at large. In this post I'd like to share some more context on why I am so excited about this development. ## CDC Is a Feature Not a Product DevFeed: [CDC Is a Feature Not a Product]() Original publisher: [Read original article]() Published: 2024-10-18T13:55:00Z Content type: opinion Language: en Sources: [Gunnar Morling]() Topics: [data](), [Kafka](), [Database](), [Microservice](), [observability]() Tags: [connectors](), [data](), [database](), [debezium](), [governance](), [kafka](), [observability](), [open-source](), [opinion](), [real-time](), [replication](), [schema]() ### AI overview The author argues that Change Data Capture, including Debezium, is an enabling feature rather than a standalone product. CDC becomes more valuable as part of a broader data platform with connectors, processing, governance, schema management, observability, and quality management. The article also discusses integrating CDC closely with databases and notes acquisitions of CDC-focused startups by Databricks and ClickHouse. ### Source excerpt During and after my time as the lead of Debezium, a widely used open-source platform for Change Data Capture (CDC) for a variety of database, I got repeatedly asked whether I'd be interested in creating a company around CDC. VCs, including wellknown household names, did and do reach out to me, pitching this idea. ## Rustic Witcher: Reimagining data anonymization DevFeed: [Rustic Witcher: Reimagining data anonymization]() Original publisher: [Read original article]() Author: Pavlos Petros Tournaris Published: 2024-07-01T11:10:44Z Content type: article Language: en Sources: [Blueground Engineering blog]() Topics: [data](), [pii](), [AWS Database Migration Service](), [Rust](), [datasets](), [Amazon S3](), [Kafka](), [Microservices](), [Testing]() Tags: [aws-database-migration-service](), [data](), [datasets](), [debezium](), [kafka](), [microservices](), [pii](), [rust](), [s3](), [testing]() ### AI overview Blueground describes replacing an increasingly slow production-data de-identification pipeline with Rustic Witcher, an internal Rust tool designed to support growing data volumes and database counts. The existing process used Debezium, Kafka, temporary Postgres storage, and S3, while the proposed approach considers AWS Database Migration Service exports as its input. ### Source excerpt At Blueground, the majority of our services require data for testing end-to-end user journeys, whether client-facing or operational. Consequently, we need to de-identify PII from production data to provide our engineers with realistic datasets. This approach ensures our pre-production systems mirror the behavior of production ## A Taxonomy Of Data Change Events DevFeed: [A Taxonomy Of Data Change Events]() Original publisher: [Read original article]() Published: 2024-03-13T00:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [data](), [Databases](), [real-time](), [Microservice]() Tags: [comparison](), [data](), [database](), [debezium](), [etl](), [event](), [logging](), [microservices](), [real-time]() ### AI overview This article develops a taxonomy of data change events used in Change Data Capture systems such as Debezium. It explains full events, delta events, and ID-only events, along with their contents and use cases including real-time ETL, microservices data exchange, and audit logging. ### Source excerpt Table of Contents Full Events Delta Events Id-only Events Change Event Metadata Comparison This post originally appeared on the Decodable blog. All rights reserved. Data change events are at the core of Change Data Capture (CDC) solutions such as Debezium. They describe the changes made to a specific record in a database and allow event consumers to take action based on this information, enabling a wide range of use cases, such as real-time ETL (by propagating the updated data into downstream data stores such as data warehouses, analytics databases, or fulltext search indexes), microservices data exchange, or audit logging. ## Logical Replication From Postgres 16 Stand-By Servers--Debezium and Failover Slots DevFeed: [Logical Replication From Postgres 16 Stand-By Servers--Debezium and Failover Slots]() Original publisher: [Read original article]() Published: 2023-12-19T00:00:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [Replication](), [Databases](), [Kafka](), [Streaming](), [Open Source](), [Amazon DocumentDB]() Tags: [amazon-rds](), [apache-kafka](), [debezium](), [failover](), [open-source](), [postgres](), [replication]() ### AI overview A tutorial on using Debezium to stream logical replication changes from PostgreSQL 16 standby servers, including failover scenarios and replication slots. It demonstrates a Docker Compose environment with Kafka and Kafka Connect, and discusses PostgreSQL deployments on Amazon RDS. ### Source excerpt Table of Contents Stand-By Logical Replication With Debezium Towards Fail-Over Slots Wrap-Up This post originally appeared on the Decodable blog. All rights reserved. Welcome back to this series about logical replication from Postgres 16 stand-by servers, in which we'll discuss how to use this feature with Debezium--a popular open-source platform for Change Data Capture (CDC) for a wide range of databases--as well as how to manage logical replication in case of failover scenarios, i.e. a situation where your primary Postgres server becomes unavailable and a stand-by server needs to take over. ## When Change Data Capture Can Break Application Encapsulation DevFeed: [When Change Data Capture Can Break Application Encapsulation]() Original publisher: [Read original article]() Published: 2023-11-21T00:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Architecture & Design](), [Software Engineering](), [streaming-data-processing](), [Kafka](), [Database]() Tags: [apache-kafka](), [data](), [debezium](), [schema](), [stream-processing](), [streaming]() ### AI overview This article examines whether exposing database change event feeds through Change Data Capture (CDC) breaks application encapsulation. It explains that CDC can expose internal table models as event APIs and create schema and downstream-consumer risks, then discusses ways to address those risks. ### Source excerpt Table of Contents CDC--A Quick Primer Does CDC Break Encapsulation? Entering Data Contracts Implementation Approaches For Data Contracts The Outbox Pattern Stream Processing Streaming Data Contracts--Beyond the Basics Handling Schema Changes Summary This post originally appeared on the Decodable blog. All rights reserved. Having worked on Debezium--an open-source platform for Change Data Capture (CDC)--for several years, one concern I've heard repeatedly is this: aren't you breaking the encapsulation of your application when you expose change event feeds directly from your database? After all, CDC exposes your internal persistent data model to the outside world, which may have unintended consequences, e.g. in terms of data exposure but also when it comes to changes to the schema of your data, which may break downstream consumers. ## CDC Use Cases: 7 Ways to Put CDC to Work DevFeed: [CDC Use Cases: 7 Ways to Put CDC to Work]() Original publisher: [Read original article]() Published: 2023-11-02T00:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [data-engineering](), [streaming-data-processing](), [Kafka](), [apache-flink](), [MySQL](), [Microservice](), [migration]() Tags: [apache-flink](), [data](), [data-engineering](), [debezium](), [engineering](), [kafka](), [migration](), [mysql](), [real-time]() ### AI overview This article explains change data capture (CDC), focusing on log-based CDC and its low-latency, resource-efficient handling of database changes. It introduces Debezium and describes how CDC works with data streaming and stream-processing tools such as Apache Kafka and Apache Flink, before outlining seven common use cases. ### Source excerpt Table of Contents What is CDC? CDC Tools Analytics Data Platforms Application Caches Full-Text Search Audit Logs Continuous Queries Microservices Data Exchange Monolith-to-Microservices Migration Summary This post originally appeared on the Decodable blog. All rights reserved. Change Data Capture (CDC) is a powerful tool in data engineering and has seen a tremendous uptake in organizations of all kinds over the last few years. This is because it enables the tight integration of transactional databases into many other systems in your business at a very low latency. ## Postgres 15: Logical Decoding Row Filters With Debezium DevFeed: [Postgres 15: Logical Decoding Row Filters With Debezium]() Original publisher: [Read original article]() Published: 2022-12-15T00:00:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [Database](), [Replication](), [Open Source]() Tags: [configuration](), [connectors](), [data](), [database](), [debezium](), [decoding](), [network](), [pii](), [postgres](), [privacy](), [replication](), [server]() ### AI overview This tutorial explains how Postgres 15 row filters for logical decoding can be used with Debezium to produce more targeted change data streams. It contrasts server-side filtering with Debezium-side configuration and transformations, including use cases involving bandwidth reduction, privacy, and tenant-specific streams. It also notes a Postgres bug affecting UPDATE and DELETE events when column lists are used. ### Source excerpt Table of Contents Using Logical Decoding Row Filters With Debezium Observing Filtered Change Events Wrap-Up This post originally appeared on the Decodable blog. All rights reserved. Since logical decoding was added to Postgres in version 9.4, this powerful feature for capturing changes from the write-ahead log of the database has been continuously improved. Postgres 15, released in October this year, added support for fine-grained control over which columns (by means of column lists) and rows (via row filters) should be exported from captured tables. This means, in relational terminology, projections and filters are now natively supported by Postgres change event publications. ## Investigating WAL Growth from an Inactive Postgres Replication Slot on Amazon RDS DevFeed: [Investigating WAL Growth from an Inactive Postgres Replication Slot on Amazon RDS]() Original publisher: [Read original article]() Published: 2022-11-30T13:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Replication](), [Amazon RDS](), [apache-flink](), [Database]() Tags: [amazon](), [amazon-rds](), [apache-flink](), [database](), [debezium](), [disk-space](), [postgres](), [replication](), [storage]() ### AI overview This article investigates how an inactive Postgres replication slot on Amazon RDS can retain WAL segments and consume disk space even when the database appears idle. It reproduces the behavior and compares replication-slot LSN differences with RDS storage and transaction-log metrics. ### Source excerpt Table of Contents The Observation The Solution Take Away While working on a demo for processing change events from Postgres with Apache Flink, I noticed an interesting phenomenon: A Postgres database which I had set up for that demo on Amazon RDS, ran out of disk space. The machine had a disk size of 200 GiB which was fully used up in the course of less than two weeks. Now a common cause for this kind of issue are replication slots which are not advanced: in that case, Postgres will hold on to all WAL segments after the latest log sequence number (LSN) which was confirmed for that slot. Indeed I had set up a replication slot (via the Decodable CDC source connector for Postgres, which is based on Debezium). I then had stopped that connector, causing the slot to become inactive. The problem was though that I was really sure that there was no traffic in that database whatsoever! What could cause a WAL growth of ~18 GB/day then? ## ByteBuffer and the Dreaded NoSuchMethodError DevFeed: [ByteBuffer and the Dreaded NoSuchMethodError]() Original publisher: [Read original article]() Published: 2020-12-21T16:45:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Java](), [Compiler](), [Apache Pulsar](), [Eclipse Vert.x]() Tags: [apache](), [compatibility](), [compiler](), [debezium](), [issue](), [java](), [java-9]() ### AI overview This article explains why Java code compiled with JDK 9 or newer can trigger NoSuchMethodError when run on Java 8. Covariant return types cause the compiler to record a method signature that is unavailable in Java 8, creating a compatibility problem for projects that target older Java runtimes. ### Source excerpt Table of Contents How to Prevent This Situation? The other day, a user in the Debezium community reported an interesting issue; They were using Debezium with Java 1.8 and got an odd NoSuchMethodError: ## Single Message Transformations - The Swiss Army Knife of Kafka Connect DevFeed: [Single Message Transformations - The Swiss Army Knife of Kafka Connect]() Original publisher: [Read original article]() Published: 2020-05-14T13:30:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [Kafka](), [data]() Tags: [connectors](), [data](), [debezium](), [filter](), [format](), [kafka](), [open-source](), [routing]() ### AI overview This tutorial explains how Kafka Connect single message transformations (SMTs) modify Kafka records in source and sink connector pipelines. It covers use cases including format conversion, timestamp handling, value masking, filtering, routing, and other transformations, with examples informed by Debezium change data capture. ### Source excerpt Table of Contents Format Conversions Ensuring Backwards Compatibility Filtering and Routing Tombstone Handling Externalizing Large Payloads Limitations Learning More Do you remember Angus "Mac" MacGyver? The always creative protagonist of the popular 80ies/90ies TV show, who could solve about any problem with nothing more than a Swiss Army knife, duct tape, shoe strings and a paper clip? The single message transformations (SMTs) of Kafka Connect are almost as versatile as MacGyver's Swiss Army knife: How to change the timezone or format of date/time message fields? How to change the topic a specific message gets sent to? How to filter out specific records? SMTs can be the answer to these and many other questions that come up in the context of Kafka Connect. Applied to source or sink connectors, SMTs allow to modify Kafka records before they are sent to Kafka, or after they are consumed from a topic, respectively.