# Gunnar Morling Recent content in Blogs on Gunnar Morling This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Parquet File Write Support, Bloom Filters, Improved Performance: Hardwood 1.1.0.Beta1 Is Out DevFeed: [Parquet File Write Support, Bloom Filters, Improved Performance: Hardwood 1.1.0.Beta1 Is Out]() Original publisher: [Read original article]() Published: 2026-08-31T19:36:00Z Content type: release Language: en Sources: [Gunnar Morling]() Topics: [parquet](), [Library](), [Parser](), [Command-line interface](), [Compression]() Tags: [apache-parquet](), [api](), [cli](), [compression](), [library](), [parquet](), [performance](), [release]() ### AI overview The first beta of Hardwood 1.1 introduces initial Parquet file-writing support through record-based and batch-oriented APIs. The release also adds Bloom filters, dictionary-based row-group pruning, performance improvements, and CLI enhancements. ### Source excerpt Table of Contents Write Support Query Evaluation: Bloom Filters and Dictionary-Based Row-Group Pruning Performance Improvements Hardwood CLI Closing Thoughts "When is write support gonna land in Hardwood?" That's probably the most common question I got over the last few months. As of today, I am very happy to share that the answer has changed from "It's coming soon" to "A first cut is there, give it a try" -- the first Beta of Hardwood 1.1 is out! This is a major milestone for the project, marking the first step in evolving Hardwood from being solely a Parquet parser to a complete library for this widely used columnar file format. But there's more. This release also comes with significant enhancements to the query layer (Bloom filters, dictionary-based row-group pruning), many performance improvements such as a fast path for effectively fixed-length list columns, an even snappier CLI, and much more. Let's dig into some of the new features and changes! ## A Fast Path for Fixed-Length Lists in Parquet DevFeed: [A Fast Path for Fixed-Length Lists in Parquet]() Original publisher: [Read original article]() Published: 2026-07-22T05:30:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [parquet](), [Embeddings](), [Optimization](), [data]() Tags: [embedding](), [embeddings](), [encoding](), [optimization](), [parquet](), [performance](), [retrieval](), [vector]() ### AI overview The article explains why Apache Parquet handles fixed-length lists inefficiently through Dremel encoding and describes a Hardwood optimization that detects effectively fixed-length data pages and bypasses regular record reconstruction. Benchmarks show speed-ups of up to 3.7x for 768-element lists, reaching performance comparable to a flat column. ### Source excerpt Table of Contents Parquet's Dremel Encoding Reading Effectively-Fixed-Length Lists Faster Performance Gains Summary In its current form Apache Parquet isn't a great fit for storing fixed-length lists, such as coordinates, RGB(A) colors, or--an increasingly common case--vector embeddings driving search and retrieval workloads. A 768-dimensional embedding is just a list of floats that always has the same length, yet Parquet's Dremel machinery encodes it as if that length could vary from row to row, spelling out and reconstructing each vector's structure on read. That costs roughly 3x more than a purely flat columnar representation of the same data (see apache/arrow#34510). ## Hardwood 1.0: A Fast, Lightweight Apache Parquet Reader for the JVM DevFeed: [Hardwood 1.0: A Fast, Lightweight Apache Parquet Reader for the JVM]() Original publisher: [Read original article]() Published: 2026-06-25T05:30:00Z Content type: release Language: en Sources: [Gunnar Morling]() Topics: [parquet](), [releases](), [Java](), [Library](), [Open Source](), [Maven Central]() Tags: [apache-parquet](), [compatibility](), [compression](), [cpu](), [java](), [jvm](), [library](), [maven-central](), [open-source](), [parquet](), [performance](), [releases]() ### AI overview The article announces Hardwood 1.0, an open-source Apache Parquet library for Java 21 or newer and the JVM. It focuses on fast, multi-threaded reading with no mandatory dependencies, supports local and remote files, and is available from Maven Central. ### Source excerpt Table of Contents Why Hardwood What's in Hardwood 1.0 Performance The Hardwood CLI Building Open-Source With AI A Big Thank You What's Ahead Hardwood is a new Parquet library for the JVM, written from scratch to do one thing well: read (and soon, write) Apache Parquet files fast, with no mandatory dependencies. It is performance-focused and multi-threaded at its core, fanning page decoding out across all your CPU cores by default. Today, Hardwood reaches 1.0. After five preview releases since the start of the year (Alpha1, Beta1, Beta2, CR1, CR2), we now consider Hardwood ready for production, and its public API will evolve with a strong focus on backwards compatibility going forward. Hardwood targets Java 21 or newer, is open-source (Apache License 2.0), and is available from Maven Central. ## Improved Column Reader API, First Cut of Geospatial Support: Hardwood 1.0.0.CR1 Is Available DevFeed: [Improved Column Reader API, First Cut of Geospatial Support: Hardwood 1.0.0.CR1 Is Available]() Original publisher: [Read original article]() Published: 2026-05-31T20:36:00Z Content type: release Language: en Sources: [Gunnar Morling]() Topics: [parquet](), [Command-line interface](), [API]() Tags: [apache-parquet](), [cli](), [documentation](), [geospatial](), [improvements](), [parquet](), [release]() ### AI overview Hardwood 1.0.0.CR1 introduces a substantially improved column reader API for Apache Parquet files, initial support for GEOMETRY and GEOGRAPHY column types, and other improvements to the core library and CLI. The columnar API remains experimental. ### Source excerpt Table of Contents Reworked ColumnReader API Geospatial Support Documentation Overhaul Further Fixes and Improvements I am happy to announce the release of Hardwood 1.0.0.CR1! This first candidate release of Hardwood 1.0 brings a substantially improved API for columnar access to Apache Parquet files, initial support for Parquet's GEOMETRY/GEOGRAPHY column types, and many other improvements to the core library as well as the Hardwood CLI. ## VARIANT Support, Interactive Parquet File TUI: Hardwood 1.0.0.Beta2 Is Out DevFeed: [VARIANT Support, Interactive Parquet File TUI: Hardwood 1.0.0.Beta2 Is Out]() Original publisher: [Read original article]() Published: 2026-04-29T17:45:00Z Content type: release Language: en Sources: [Gunnar Morling]() Topics: [parquet](), [Text-based user interface](), [Command-line interface](), [Parser](), [API](), [Amazon S3]() Tags: [apache-parquet](), [cli](), [compression](), [parquet](), [performance](), [release](), [s3](), [screen](), [storage]() ### AI overview Hardwood 1.0.0.Beta2 adds support for VARIANT columns, an interactive text-based UI for examining Parquet files, improved performance, and more efficient object-storage reads. ### Source excerpt Table of Contents VARIANT Support Hardwood CLI TUI Unified Reader API Performance Improvements Wrapping Up I am happy to announce the release of Hardwood 1.0.0.Beta2! The latest version of this new parser for Apache Parquet comes with support for VARIANT columns, an interactive text-based UI (TUI) for examining and analysing the structure of Parquet files, significantly improved performance, more efficient reading of files from object storage, and much more. ## Hardwood Reaches Beta: S3, Predicate Push-Down, CLI, and More DevFeed: [Hardwood Reaches Beta: S3, Predicate Push-Down, CLI, and More]() Original publisher: [Read original article]() Published: 2026-04-02T18:20:00Z Content type: release Language: en Sources: [Gunnar Morling]() Topics: [parquet](), [Amazon S3](), [Command-line interface]() Tags: [apache-parquet](), [cli](), [parquet](), [rest-api](), [s3]() ### AI overview Hardwood 1.0.0.Beta1 is a new Apache Parquet parser with an S3 backend, predicate push-down for local and remote files, Avro bindings, and a CLI for inspecting Parquet files. Its S3 support uses Java's built-in HTTP client and adds no mandatory dependencies. ### Source excerpt Table of Contents S3 Backend Predicate Push-Down Avro Bindings hardwood-cli Wrapping Up I am pleased to announce the release of Hardwood 1.0.0.Beta1! Hardwood is a new parser for Apache Parquet, optimized for minimal dependencies and great performance. Since the project's initial release just a few weeks back, a small yet very active community has come together and evolved Hardwood significantly. Today, we are shipping an S3 backend, allowing to parse files directly from object storage, predicate pushdown for both local and remote files, Avro bindings, a CLI for inspecting Parquet files, and much more. We're also excited to launch a website for the project, hardwood.dev, which contains the documentation and API reference. Let's dig in. ## Hardwood: A New Parser for Apache Parquet DevFeed: [Hardwood: A New Parser for Apache Parquet]() Original publisher: [Read original article]() Published: 2026-02-26T12:30:00Z Content type: release Language: en Sources: [Gunnar Morling]() Topics: [parquet](), [Parser](), [Java](), [Open Source](), [Compression](), [DuckDB]() Tags: [apache-parquet](), [compression](), [dependencies](), [java](), [maven-central](), [open-source](), [parquet](), [parsing](), [performance](), [release]() ### AI overview The article announces Hardwood, an open-source Java 21+ parser for Apache Parquet. It is designed to minimize dependencies and improve parsing performance through a multithreaded decoding pipeline, with optional libraries for Parquet compression algorithms. The article also introduces Maven setup and row-oriented and columnar APIs. ### Source excerpt Table of Contents Why Hardwood? Hello, Hardwood! Parsing Performance Built With AI, Not By AI What's Next? Today, it's my great pleasure to announce the first public release of Hardwood, a new parser for the Apache Parquet file format, optimized for minimal dependencies and great performance. Hardwood is open-source (Apache License 2.0) and supports Java 21 or newer. You can grab it from Maven Central and start parsing your Parquet files with ease and efficiency. ## Using Materialized Views and Derived Datasets to Optimize Data Queries DevFeed: [Using Materialized Views and Derived Datasets to Optimize Data Queries]() Original publisher: [Read original article]() Published: 2025-12-07T09:05:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [data-architecture](), [Data Management](), [Database](), [datasets](), [Latency](), [elasticsearch](), [parquet]() Tags: [data](), [data-lake](), [data-management](), [database](), [datasets](), [elasticsearch](), [latency](), [parquet](), [performance]() ### AI overview The article explains how pull-based queries retrieve matching records at query time and why this can create performance, data-format, data-shape, and data-location challenges. It presents materialized views and derived datasets as a way to precompute query results and store them in an optimized format, shape, and location. ### Source excerpt Table of Contents Materialized Views Embracing Data Duplication Streams for machines, tables for humans Historically, data management systems have been built around the notion of pull queries: users query data which, for instance, is stored in tables in an RDBMS, Parquet files in a data lake, or a full-text index in Elasticsearch. When a user issues a query, the engine will produce the result set at that point in time by churning through the data set and finding all matching records (oftentimes sped up by utilizing indexes). ## On Idempotency Keys DevFeed: [On Idempotency Keys]() Original publisher: [Read original article]() Published: 2025-11-25T12:10:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [distributed-systems](), [Database]() Tags: [database](), [distributed](), [distributed-systems](), [systems]() ### AI overview The article explains how idempotency keys enable exactly-once processing despite duplicate message delivery. Consumers compare keys, atomically persist the key with the processed message or derived view, and skip recognized duplicates. It also discusses UUIDv4, UUIDv7, and ULID trade-offs for retaining and aging keys. ### Source excerpt Table of Contents UUIDs Monotonically Increasing Sequences Deriving Idempotency Keys From the Transaction Log Discussion In distributed systems, there's a common understanding that it is not possible to guarantee exactly-once delivery of messages. What is possible though is exactly-once processing. By adding a unique idempotency key to each message, you can enable consumers to recognize and ignore duplicate messages, i.e. messages which they have received and successfully processed before. ## Building a Durable Execution Engine With SQLite DevFeed: [Building a Durable Execution Engine With SQLite]() Original publisher: [Read original article]() Published: 2025-11-20T12:10:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [SQLite](), [Programming](), [Code](), [Transactions]() Tags: [code](), [programming](), [sqlite](), [transactions](), [workflows]() ### AI overview This article introduces durable execution for long-running, multi-step workflows and discusses building a durable execution engine with SQLite. It explains how persistent execution state and checkpoints allow interrupted flows to resume without repeating completed steps, with applications including business processes and systems that interact with LLMs. ### Source excerpt Table of Contents Hello Persistasaurus! Capturing Execution State Delayed Executions Human Interaction Managing State Wrapping Up Lately, there has been a lot of excitement around Durable Execution (DE) engines. The basic idea of DE is to take (potentially long-running) multi-step workflows, such as processing a purchase order or a user sign-up, and make their individual steps persistent. If a flow gets interrupted while running, for instance due to a machine failure, the DE engine can resume it from the last successfully executed step and drive it to completion. ## Why Kafka and Postgres Serve Different Purposes DevFeed: [Why Kafka and Postgres Serve Different Purposes]() Original publisher: [Read original article]() Published: 2025-11-03T17:02:00Z Content type: opinion Language: en Sources: [Gunnar Morling]() Topics: [Kafka](), [Database]() Tags: [argument](), [kafka](), [postgres](), [queue](), [rdbms]() ### AI overview The article argues that Postgres and Kafka serve different purposes and that choosing between them should depend on the problem being solved. It challenges advice to use Postgres instead of Kafka, noting that Kafka is not primarily a queue and that building a robust queue on Postgres can introduce operational difficulties. ### Source excerpt Looking to make it to the front page of HackerNews? Then writing a post arguing that "Postgres is enough", or why "you don't need Kafka at your scale" is a pretty failsafe way of achieving exactly that. No matter how often it has been discussed before, this topic is always doing well. And sure, what's not to love about that? I mean, it has it all: Postgres, everybody's most favorite RDBMS--check! Keeping things lean and easy--sure, count me in! A somewhat spicy take--bring it on! ## Comparing Java ZGC and G1 for Tail Latency in a Quarkus Microservice DevFeed: [Comparing Java ZGC and G1 for Tail Latency in a Quarkus Microservice]() Original publisher: [Read original article]() Published: 2025-09-17T15:09:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Java](), [benchmarking](), [Quarkus](), [Databases]() Tags: [benchmarking](), [gc](), [java](), [lts](), [performance](), [postgres](), [quarkus](), [sample]() ### AI overview This article compares Java's ZGC and G1 garbage collectors using default settings in a sample Quarkus microservice that reads data from a Postgres database. The benchmark applies 1,000 requests per second and measures request latencies on a four-core, 4 GB RAM instance. The supplied excerpt does not include the comparison results. ### Source excerpt Table of Contents ZGC Allocation Stalls Summary In the "Let's Take a Look at...!" blog series I am exploring interesting projects, developments and technologies in the data and streaming space. This can be KIPs and FLIPs, open-source projects, services, relevant improvements to Java and the JVM, and more. The idea is to get some hands-on experience, learn about potential use cases and applications, and understand the trade-offs involved. If you think there's a specific subject I should take a look at, let me know in the comments below. Java 25 was released earlier this week, and it is the first Java release with long-term support (LTS) which ships with Generational ZGC as the one (and only) flavor of the ZGC garbage collector. ZGC itself is a relatively new concurrent collector, originally added in Java 11. ## Postgres Replication Slots: Confirmed Flush LSN vs. Restart LSN DevFeed: [Postgres Replication Slots: Confirmed Flush LSN vs. Restart LSN]() Original publisher: [Read original article]() Published: 2025-08-05T11:55:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [Replication](), [Databases](), [Transactions]() Tags: [capture](), [concurrent](), [database](), [debezium](), [postgres](), [recovery](), [replication](), [retention](), [stream](), [streaming](), [transactions](), [troubleshooting]() ### AI overview This tutorial explains the difference between PostgreSQL replication slots' confirmed_flush_lsn and restart_lsn attributes. It describes how consumers resume from replication slots, how concurrent transactions affect the oldest WAL that may still be required, and why understanding both LSNs helps with troubleshooting and WAL retention. ### Source excerpt Table of Contents confirmed_flush_sn: Tracking Consumer Progress restart_lsn: Handling Concurrent Transactions Mid-Transaction Recovery Looking Forward: Streaming In-Progress Transactions Replication slots in Postgres keep track of how far consumers have read a replication stream. After a restart, consumers--either Postgres read replicas or external tools for change data capture (CDC), like Debezium--resume reading from the last confirmed log sequence number (LSN) of their replication slot. The slot prevents the database from disposing of required log segments, allowing safe resumption after downtime. In this post, we are going to take a look at why Postgres replication slots don't have one but two LSN-related attributes: restart_lsn and confirmed_flush_lsn. Understanding the difference between the two is crucial for troubleshooting replication issues, optimizing WAL retention, and avoiding common pitfalls in production environments. ## Converting Future to CompletableFuture With Java Virtual Threads DevFeed: [Converting Future to CompletableFuture With Java Virtual Threads]() Original publisher: [Read original article]() Published: 2025-07-17T08:25:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [Java](), [Concurrency](), [Concurrent Programming]() Tags: [api](), [asynchronous](), [blocking](), [java](), [java-8](), [threads]() ### AI overview This tutorial explains how Java 21+ virtual threads can help convert legacy Future objects into CompletableFuture instances. It contrasts blocking, polling, and asynchronous approaches, noting that virtual threads make blocking inexpensive by unmounting blocked threads from their underlying platform threads. ### Source excerpt This post explores how virtual threads in Java 21+ provide an elegant solution for converting legacy Future objects into CompletableFuture instances. Since Java 8, the CompletableFuture API provides a convenient way for performing asynchronous operations in a functional, composable way. This makes it very simple to call some long-running methods--for instance involving external I/O--asynchronously and process each result as soon as it is available, without blocking on any threads: ## Mastering Postgres Replication Slots: Preventing WAL Bloat and Other Production Issues DevFeed: [Mastering Postgres Replication Slots: Preventing WAL Bloat and Other Production Issues]() Original publisher: [Read original article]() Published: 2025-07-08T11:55:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [Replication](), [Databases](), [Monitoring]() Tags: [best-practices](), [database](), [debezium](), [failover](), [monitoring](), [postgres](), [production](), [replication]() ### AI overview This practical guide explains how to manage Postgres replication slots for Change Data Capture pipelines and reduce excessive write-ahead log retention. It covers decoding plug-ins, heartbeats, failover, publications, filtering, replica identity, monitoring, and removing unused slots, with examples based primarily on Debezium's Postgres connector. ### Source excerpt Table of Contents Use the pgoutput Logical Decoding Output Plug-in Define a Maximum Replication Slot Size Enable Heartbeats Use Table-level Publications Use Column and Row Filters Enable Fail-Over Slots Consider Using Replica Identity FULL Monitor, Monitor, Monitor! Drop Unused Replication Slots Summary Over the last couple of years, I've helped dozens of users and organizations to build Change Data Capture (CDC) pipelines for their Postgres databases. A key concern in that process is setting up and managing replication slots, which are Postgres' mechanism for making sure that any segments of the write-ahead log (WAL) of the database are kept around until they have been processed by registered replication consumers. When not being careful, a replication slot may cause unduly large amounts of WAL segments to be retained by the database. This post describes best practices helping to prevent this and other issues, discussing aspects like heartbeats, replication slot failover, monitoring, the management of Postgres publications, and more. While this is primarily based on my experience of using replication slots via Debezium's Postgres connector, the principles are generally applicable and are worth considering also when using other CDC tools for Postgres based on logical replication. ## Building AI agents as streaming SQL queries with Apache Flink DevFeed: [Building AI agents as streaming SQL queries with Apache Flink]() Original publisher: [Read original article]() Published: 2025-06-18T13:30:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [AI Agent](), [SQL](), [Streaming](), [apache-flink](), [event driven](), [Scalability](), [Large Language Model](), [Developer experience]() Tags: [ai](), [ai-agent](), [apache-flink](), [developer-experience](), [event-driven](), [llms](), [scalability](), [sql](), [streaming]() ### AI overview This article explores whether AI agents can be built as streaming SQL queries. It presents Apache Flink and stream-processing concepts as potential foundations for event-driven, scalable, consistent, low-latency agentic systems, while noting that many deployed agents are defined AI-assisted workflows. ### Source excerpt Table of Contents Agents Need to Interact With LLMs Agents Should Be Event-Driven Agents Need Context Agents Require Memory When SQL Is Not Enough Parting Thoughts AI Agents have improved in leaps and bounds in recent times, moving beyond simple chatbots to sophisticated, autonomous systems. This post explores a novel approach to building agentic systems: using the power of streaming SQL queries. Discover how platforms like Apache Flink can transform the development of AI Agents, offering benefits in consistency, scalability, and developer experience. ## Backfilling Postgres TOAST Columns in Debezium Data Change Events DevFeed: [Backfilling Postgres TOAST Columns in Debezium Data Change Events]() Original publisher: [Read original article]() Published: 2025-05-26T14:40:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Databases](), [Replication](), [apache-flink](), [stream-processing](), [SQL](), [API]() Tags: [apache-flink](), [data](), [debezium](), [debezium-connector](), [flink](), [postgres](), [replication](), [sql](), [stateful](), [stream-processing]() ### AI overview This article explains how Postgres logical replication handles TOAST columns in Debezium change events. It discusses Debezium's reselect postprocessor and explores Apache Flink solutions using stateful stream processing, Flink SQL, and process table functions. ### Source excerpt Table of Contents Debezium Reselect Postprocessor Flink DataStream API Flink SQL With OVER Aggregation Flink Process Table Functions Summary and Discussion Postgres logical replication, while powerful for capturing real-time data changes, presents challenges with TOAST columns, whose values can be absent from data change events in specific situations. This post discusses how Debezium addresses this through its built-in reselect post processor, then explores more robust solutions leveraging Apache Flink's capabilities for stateful stream processing, including Flink SQL and the brand-new process table functions (PTFs) in Flink 2.1. ## Streaming and Batch Processing Are Complementary; Pull Versus Push Is the Key Distinction DevFeed: [Streaming and Batch Processing Are Complementary; Pull Versus Push Is the Key Distinction]() Original publisher: [Read original article]() Published: 2025-05-14T08:10:00Z Content type: opinion Language: en Sources: [Gunnar Morling]() Topics: [Streaming](), [stream-processing](), [streaming-data-processing](), [real-time](), [systems](), [data]() Tags: [kafka](), [latency](), [low-latency](), [performance](), [real-time](), [stream-processing](), [streaming](), [systems]() ### AI overview The article argues that streaming systems commonly use batching to improve throughput, so streaming and batch processing are not opposites. It proposes pull versus push semantics as the more meaningful distinction and explains that push-based streaming can provide timely updates, while adding complexity around state, joins, and out-of-order data. ### Source excerpt Often times, "Stream vs. Batch" is discussed as if it's one or the other, but to me this does not make that much sense really. ## Designing a Cloud-Native Event Log Beyond Kafka's Partition Model DevFeed: [Designing a Cloud-Native Event Log Beyond Kafka's Partition Model]() Original publisher: [Read original article]() Published: 2025-04-24T14:25:00Z Content type: opinion Language: en Sources: [Gunnar Morling]() Topics: [Kafka](), [Amazon S3](), [cloud-infrastructure](), [event driven](), [systems]() Tags: [agentic](), [cloud-native](), [event-driven](), [event-sourcing](), [kafka](), [partition](), [pipelines](), [s3]() ### AI overview The article proposes a personal wishlist for a durable cloud-native event log designed beyond Kafka's traditional architecture. It discusses separating storage and compute, using object storage such as S3, removing partitions, and providing key-centric streams with ordering and dynamically scalable consumers. ### Source excerpt The last few days I spent some time digging into the recently announced KIP-1150 ("Diskless Kafka"), as well AutoMQ's Kafka fork, tightly integrating Apache Kafka and object storage, such as S3. Following the example set by WarpStream, these projects aim to substantially improve the experience of using Kafka in cloud environments, providing better elasticity, drastically reducing cost, and paving the way towards native lakehouse integration. This got me thinking, if we were to start all over and develop a durable cloud-native event log from scratch--Kafka.next if you will--which traits and characteristics would be desirable for this to have? Separating storage and compute and object store support would be table stakes, but what else should be there? Having used Kafka for many years for building event-driven applications as well as for running realtime ETL and change data capture pipelines, here's my personal wishlist: ## A Deep Dive Into Ingesting Debezium Events From Kafka With Flink SQL DevFeed: [A Deep Dive Into Ingesting Debezium Events From Kafka With Flink SQL]() Original publisher: [Read original article]() Published: 2025-04-16T09:25:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [apache-flink](), [Kafka](), [Apache-Kafka]() Tags: [changelog](), [connectors](), [data](), [debezium](), [flink](), [how-to](), [kafka](), [schema](), [streams]() ### AI overview This tutorial explains how to ingest Debezium change events from Kafka into Apache Flink using Flink SQL. It compares the Apache Kafka SQL Connector and Upsert Kafka SQL Connector, discusses JSON and Avro formats, and explains how connector and format choices determine append-only or changelog semantics. ### Source excerpt Table of Contents Flink SQL Connectors for Apache Kafka The Apache Kafka SQL Connector in Append-Only Mode The Apache Kafka SQL Connector As a Changelog Source The Upsert Kafka SQL Connector Summary Over the years, I've spoken quite a bit about the use cases for processing Debezium data change events with Apache Flink, such as metadata enrichment, building denormalized data views, and creating data contracts for your CDC streams. One detail I haven't covered in depth so far is how to actually ingest Debezium change events from a Kafka topic into Flink, in particular via Flink SQL. Several connectors and data formats exist for this, which can make things somewhat confusing at first. So let's dive into the different options and the considerations around them! ## Building a Native Binary for Apache Kafka on macOS DevFeed: [Building a Native Binary for Apache Kafka on macOS]() Original publisher: [Read original article]() Published: 2025-04-07T10:25:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [Kafka](), [macOS](), [Development](), [Testing]() Tags: [apache-kafka](), [building](), [development](), [kafka](), [macos](), [native](), [testing]() ### AI overview This tutorial explains how to build a self-contained native Apache Kafka binary for macOS using GraalVM and the configuration and tooling from Kafka's native container-image work. It covers obtaining the Kafka distribution, using the provided native-image configuration, and producing a macOS/AArch64 executable for development and testing. ### Source excerpt Table of Contents KIP-974: Docker Image for GraalVM based Native Kafka Broker With help of the GraalVM configuration developed for KIP-974 (Docker Image for GraalVM based Native Kafka Broker), you can easily build a self-contained native binary for Apache Kafka. Read on to learn how you can build a native Kafka executable yourself, starting in milli-seconds, making it a perfect fit for development and testing purposes. When I wrote about ahead-of-time class loading and linking in Java 24 recently, I also published the start-up time for Apache Kafka as a native binary for comparison. This was done via Docker, as there's no pre-built native binary of Kafka available for the operating system I'm running on, macOS. But there is a native Kafka container image, so this is what I chose for the sake of convenience. Now, running in a container adds a little bit of overhead of course, so it wasn't a surprise when Thomas Würthinger, lead of the GraalVM project at Oracle, brought up the question what the value would be when running Kafka natively on macOS. Needless to say I can't leave this kind of nice nerd snipe pass, so I set out to learn how to build a native Kafka binary on macOS, using GraalVM. ## JEP 483: Ahead-of-Time Class Loading and Linking in Java 24 DevFeed: [JEP 483: Ahead-of-Time Class Loading and Linking in Java 24]() Original publisher: [Read original article]() Published: 2025-03-27T13:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Java](), [openjdk](), [Kafka](), [apache-flink]() Tags: [building](), [java](), [jvm](), [openjdk](), [performance]() ### AI overview This article examines JEP 483, which loads and links Java classes ahead of time and caches the results to reduce application startup time. It explains the feature's role in Project Leyden, its relationship to AppCDS, the training-run and classpath requirements, and examples using Apache Kafka and Apache Flink. ### Source excerpt Table of Contents Building an AOT Cache for Apache Kafka AOT Caching With Apache Flink Summary In the "Let's Take a Look at...!" blog series I am exploring interesting projects, developments and technologies in the data and streaming space. This can be KIPs and FLIPs, open-source projects, services, relevant improvements to Java and the JVM, and more. The idea is to get some hands-on experience, learn about potential use cases and applications, and understand the trade-offs involved. If you think there's a specific subject I should take a look at, let me know in the comments below. Java 24 got released last week, and what a meaty release it is: more than twenty Java Enhancement Proposals (JEPs) have been shipped, including highlights such as compact object headers (JEP 450, I hope to spend some time diving into that one some time soon), a new class-file API (JEP 484), and more flexible constructor bodies (JEP 492, third preview). One other JEP which might fly a bit under the radar is JEP 483 ("Ahead-of-Time Class Loading & Linking"). It promises to reduce the start-up time of Java applications without requiring any modifications to the application itself, what's not to be liked about that? Let's take a closer look! ## The Synchrony Budget DevFeed: [The Synchrony Budget]() Original publisher: [Read original article]() Published: 2025-03-18T13:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Network](), [Availability](), [Kafka]() Tags: [asynchronous](), [availability](), [distributed](), [e-commerce](), [kafka](), [network](), [services]() ### AI overview The article introduces the "synchrony budget," a design principle for distributed services: minimize synchronous calls to reduce request latency and dependencies that can lower service availability. It uses an e-commerce order flow to argue that shipment notifications can be handled asynchronously, such as through a Kafka topic, when an immediate response is unnecessary. ### Source excerpt For building a system of distributed services, one concept I think is very valuable to keep in mind is what I call the synchrony budget: as much as possible, a service should minimize the number of synchronous requests which it makes to other services. ## KIP-932 explores queue semantics and Share Groups for Apache Kafka DevFeed: [KIP-932 explores queue semantics and Share Groups for Apache Kafka]() Original publisher: [Read original article]() Published: 2025-03-05T11:35:00Z Content type: opinion Language: en Sources: [Gunnar Morling]() Topics: [Kafka](), [Streaming](), [stream-processing](), [Architecture & Design]() Tags: [apache-kafka](), [fraud](), [fraud-detection](), [job](), [kafka](), [microservices](), [stream-processing](), [streaming]() ### AI overview The article examines KIP-932, which explores adding queue semantics to Apache Kafka through Share Groups. It explains how Kafka's partition-based consumer model limits parallelism and ordered processing, and why individual message acknowledgment and rejection are important for queueing workloads such as independent job processing. ### Source excerpt Table of Contents Towards Queue Support in Kafka--Introducing Share Groups Share Groups in Action Retry Behavior and State Management Share Group State Persistence Summary and Outlook In the "Let's Take a Look at...!" blog series I am going to explore interesting projects, developments and technologies in the data and streaming space. This can be KIPs and FLIPs, open-source projects, services, and more. The idea is to get some hands-on experience, learn about potential use cases and applications, and understand the trade-offs involved. If you think there's a specific subject I should take a look at, let me know in the comments below! That guy above? Yep, that's me, whenever someone says "Kafka queue". Because, that's not what Apache Kafka is. At its core, Kafka is a distributed durable event log. Producers write events to a topic, organized in partitions which are distributed amongst the brokers of a Kafka cluster. Consumers, organized in groups, divide the partitions they process amongst themselves, so that each partition of a topic is read by exactly one consumer in the group. [Next page]()