# Apache-Kafka Published articles for Apache-Kafka. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Netflix Reworks Conductor for 420 Million Monthly Workflow Executions and 10X Larger Workflows DevFeed: [Netflix Reworks Conductor for 420 Million Monthly Workflow Executions and 10X Larger Workflows]() Original publisher: [Read original article]() Author: Leela Kumili Published: 2026-09-11T14:17:00Z Content type: news Language: en Sources: [InfoQ]() Topics: [Orchestration](), [Amazon S3](), [Kafka](), [Apache Iceberg]() Tags: [amazon-s3](), [apache-iceberg](), [apache-kafka](), [architecture-design](), [asynchronous-architecture](), [cassandra](), [cloud-architecture](), [concurrency](), [development](), [devops](), [distributed-systems](), [elasticsearch](), [java-operator-sdk](), [kafka](), [latency](), [microservices](), [netflix](), [netflix-conductor-4-workflow](), [news](), [orchestration](), [s3](), [scalability](), [windows-workflow-foundation](), [workflow](), [workflow-bpm](), [workflow-foundation]() ### AI overview Netflix reworked Conductor 4.0 to scale workflow orchestration to roughly 200,000 definitions and 420 million monthly executions. The redesign raises supported workflow size to 30,000 tasks and reports a roughly 40% reduction in p99 evaluation latency by loading only task data needed for each decision. ### Source excerpt Netflix has reworked its Conductor workflow orchestration engine to handle larger workloads, increasing supported workflow size from about 2,500 to 30,000 tasks and reducing p99 workflow evaluation latency by about 40%. Conductor 4.0 separates workflow metadata from task data, moves evaluation to asynchronous processing, and introduces dynamic worker allocation and concurrency controls. By Leela Kumili ## Low-latency, high-throughput SQS event processing with AWS Lambda provisioned mode DevFeed: [Low-latency, high-throughput SQS event processing with AWS Lambda provisioned mode]() Original publisher: [Read original article]() Author: Ben Freiberg Published: 2026-08-10T13:42:03Z Content type: article Language: en Sources: [AWS Compute Blog]() Topics: [Amazon Simple Queue Service (SQS)](), [AWS Lambda](), [Latency](), [Low Latency](), [Amazon Web Services](), [Concurrency](), [Kafka](), [Internet of things](), [telemetry]() Tags: [amazon-simple-queue-service-sqs](), [amazon-sqs](), [apache-kafka](), [aws](), [aws-lambda](), [concurrency](), [event-driven](), [flash](), [fraud](), [kafka](), [latency](), [low-latency](), [payment-processing](), [real-time](), [telemetry]() ### AI overview AWS explains how provisioned mode for Amazon SQS event source mappings with AWS Lambda provides explicit control over event pollers to support low-latency, high-throughput event processing. The article describes default and provisioned scaling, concurrency limits, throughput, and use cases including real-time payments, fraud detection, IoT telemetry, and flash-sale order fulfillment. ### Source excerpt Customers building event-driven applications on AWS rely on Amazon Simple Queue Service (Amazon SQS) and AWS Lambda event source mappings (ESMs) to process millions of events every day. The fully managed polling infrastructure of ESMs eliminates the need to write and maintain custom code. You can focus on business logic while Lambda handles scaling, batching, [...] ## Announcing Confluent Platform 8.3: Powerful Apache Flink® SQL operations, Easier KRaft Migrations, Expanded Monitoring and more. DevFeed: [Announcing Confluent Platform 8.3: Powerful Apache Flink® SQL operations, Easier KRaft Migrations, Expanded Monitoring and more.]() Original publisher: [Read original article]() Author: Premika Srinivasan Published: 2026-07-29T15:00:10Z Content type: release Language: en Sources: [Confluent: Data in motion]() Topics: [apache-flink](), [Kafka](), [Monitoring](), [data-governance](), [Model Context Protocol](), [Claude Code](), [codex]() Tags: [apache-flink](), [apache-kafka](), [claude-code](), [codex](), [confluent](), [confluent-platform](), [data-governance](), [mcp](), [mcp-server](), [monitoring]() ### AI overview Confluent Platform 8.3.0, built on Apache Kafka 4.3.0, adds simplified Apache Flink SQL operations, expanded monitoring through Unified Stream Manager, easier KRaft migration, and structured governance for data in motion. The release also introduces an MCP server for Confluent Platform for Apache Flink, allowing AI agents such as Claude Code and Codex to inspect and manage Flink resources through existing security controls. ### Source excerpt Announcing Confluent Platform 8.3: Powerful Apache Flink® SQL operations, Easier KRaft Migrations, Expanded Monitoring and more ## Real-time Change Data Capture with Redpanda Connect and MySQL DevFeed: [Real-time Change Data Capture with Redpanda Connect and MySQL]() Original publisher: [Read original article]() Author: Chandler Mayo Published: 2026-07-02T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [MySQL](), [Kafka](), [Databases](), [Docker Compose](), [real-time](), [Docker](), [SQL]() Tags: [apache-kafka](), [connectors](), [container](), [docker](), [event](), [how-to](), [kafka](), [mysql](), [network](), [pipeline](), [real-time](), [sql](), [tutorial](), [volume]() ### AI overview A step-by-step tutorial for building a real-time change data capture pipeline with MySQL and Redpanda Connect. It explains how row-level inserts, updates, and deletes become downstream events without polling, using compatibility with the Kafka Connect ecosystem. ### Source excerpt A step-by-step tutorial on how to stream every insert, update, and delete from your database using MySQL and a faster, simpler alternative to Kafka Connect. ## How Redpanda Cloud Topics rethinks Kafka compaction DevFeed: [How Redpanda Cloud Topics rethinks Kafka compaction]() Original publisher: [Read original article]() Author: Willem Kaufmann Published: 2026-06-30T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Kafka](), [cloud-infrastructure](), [Cloud](), [Streaming](), [Algorithm]() Tags: [apache](), [apache-kafka](), [architecture](), [cloud-native](), [cloud-storage](), [clusters](), [cpu](), [cycles](), [engineering](), [kafka](), [object-storage](), [reduce](), [retention](), [scale](), [storage](), [streaming]() ### AI overview The article explains how Redpanda Cloud Topics redesigns Kafka log compaction for cloud-native streaming. It describes how the architecture reduces redundant processing and CPU use, lowers cloud storage costs, and preserves Kafka behavior while addressing scaling challenges such as limited memory, tombstone removal, and rewriting large volumes of object storage data. ### Source excerpt Compaction can overwhelm poorly sized Kafka clusters, leading to full disks and maxed-out CPUs. Learn how Redpanda's Cloud Topics architecture redesigns compaction to cut redundant work, reduce cloud storage costs, and preserve the Kafka semantics you rely on. ## Kafka's log compaction corrupts data. Here's how we fixed it DevFeed: [Kafka's log compaction corrupts data. Here's how we fixed it]() Original publisher: [Read original article]() Author: Alexey Bashtanov Published: 2026-06-25T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Kafka](), [Apache-Kafka](), [bug](), [Streaming](), [Testing]() Tags: [apache-kafka](), [bug](), [engineering](), [kafka](), [streaming](), [testing]() ### AI overview The article examines a bug in Apache Kafka log compaction that can cause replicas to permanently disagree about retained records. It describes how tombstones and transaction control batches may be removed independently on different brokers, allowing deleted data to reappear or aborted data to be served as committed, and presents coordinated compaction in Redpanda Streaming as the solution. ### Source excerpt There's a problem with Apache Kafka's log compaction. Here's what we found, how to reproduce it, and how we solved it in Redpanda. ## Event-Driven Architecture with Apache Kafka DevFeed: [Event-Driven Architecture with Apache Kafka]() Original publisher: [Read original article]() Author: hello@ankit-rana.com Published: 2026-03-16T00:00:00Z Content type: tutorial Language: en Sources: [Ankit Rana | Mechanical Sympathy]() Topics: [event driven](), [Kafka](), [Apache-Kafka](), [Scalability](), [Resilience](), [streaming-data-processing](), [Latency](), [systems]() Tags: [apache-kafka](), [architecture](), [asynchronous](), [consumer](), [distributed-systems](), [event-driven-architecture](), [event-sourcing](), [kafka](), [microservices](), [partitions](), [real-time](), [resilience](), [scalability](), [schemas](), [stream-processing]() ### AI overview This tutorial explains event-driven architecture and how Apache Kafka supports asynchronous, real-time data processing. It covers producers, consumers, immutable events, event sourcing, scalability, resilience, stream-processing pipelines, and challenges such as ordering, debugging, and eventual consistency. ### Source excerpt Event-driven architecture replaces synchronous point-to-point calls with immutable events on a durable log, so producers and consumers scale and fail independently. Kafka provides that log: topics sharded into ordered append-only partitions, replicated across brokers, with consumer groups sharing partitions to scale read throughput. ## System Migration: Minimize Downtime, Maximize Efficiency DevFeed: [System Migration: Minimize Downtime, Maximize Efficiency]() Original publisher: [Read original article]() Author: hello@ankit-rana.com Published: 2026-03-16T00:00:00Z Content type: tutorial Language: en Sources: [Ankit Rana | Mechanical Sympathy]() Topics: [migration](), [systems](), [async](), [client](), [event driven](), [Kafka](), [API]() Tags: [apache-kafka](), [api](), [architecture](), [async](), [bridge-layer](), [data-pipelines](), [downtime](), [event-driven-architecture](), [metrics](), [migration](), [observability](), [production](), [reliability](), [rollback](), [system-migration](), [traffic-leakage]() ### AI overview A practical guide to migrating an existing system with minimal disruption. It recommends isolated-environment testing, load testing, adapters for incompatible contracts, synchronized asynchronous pipelines, a Kafka-based shared stream, a bridge layer, staged traffic switching, monitoring, and rollback preparation. ### Source excerpt Migrate behind a bridge layer that routes all client traffic and supports three modes: old-only, dual, and new-only. Run dual mode to compare responses without user impact, keep a back-sync pipeline so the old system stays current for rollback, and shift traffic in stages while watching metrics at each step. ## FIPS-ing the Un-FIPS-able: Apache Kafka DevFeed: [FIPS-ing the Un-FIPS-able: Apache Kafka]() Original publisher: [Read original article]() Published: 2026-03-05T00:00:00Z Content type: release Language: en Sources: [Chainguard: Unchained]() Topics: [Apache-Kafka](), [chainguard](), [Cryptography](), [Security](), [Event-Streaming](), [TLS (Transport Layer Security)](), [Open Source]() Tags: [apache](), [apache-kafka](), [apache-kafka-fips](), [chainguard](), [chainguard-containers](), [chainguard-images](), [chainguard-kafka-container](), [chainguard-kafka-image](), [cryptography](), [event-streaming](), [fips](), [fips-140-3](), [fips-containers](), [kafka](), [open-source](), [tls]() ### AI overview Chainguard announces FIPS-validated images for Apache Kafka, describing targeted source changes, validated cryptographic providers, TLS and keystore requirements, testing, and image maintenance for regulated environments. ### Source excerpt Chainguard delivers the FIPS-validated Apache Kafka images, enabling secure event streaming for FedRAMP and regulated environments. ## Inside Laravel Nightwatch's Observability Pipeline: Real-Time Event Processing with Amazon MSK and ClickHouse Cloud DevFeed: [Inside Laravel Nightwatch's Observability Pipeline: Real-Time Event Processing with Amazon MSK and ClickHouse Cloud]() Original publisher: [Read original article]() Author: AWS; Laravel; ClickHouse Published: 2025-10-08T15:36:20Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [Laravel](), [observability](), [clickhouse](), [Kafka](), [Streaming](), [telemetry](), [AWS Lambda](), [Amazon RDS](), [PostgreSQL](), [Amazon API Gateway](), [Cloud](), [Grafana Cloud Metrics]() Tags: [amazon-api-gateway](), [amazon-rds](), [apache-kafka](), [api](), [aws-lambda](), [clickhouse](), [dashboards](), [json](), [laravel](), [low-latency](), [metrics](), [monitoring](), [observability](), [rds-for-postgresql](), [streaming](), [traces]() ### AI overview This article explains how Laravel Nightwatch processes more than one billion observability events per day using AWS managed services and ClickHouse Cloud. Its streaming-first architecture combines Amazon MSK, AWS Lambda, Amazon API Gateway, Amazon RDS for PostgreSQL, and ClickHouse Cloud to provide scalable ingestion, real-time dashboards, and sub-second analytical queries. ### Source excerpt Learn how Laravel Nightwatch platform process billions of observability events in real time while keeping query latency under a second. ## The Evolution of ClickPipes: Revamped UI/UX, ClickPipes API and Terraform Provider, Prometheus metrics, and more! DevFeed: [The Evolution of ClickPipes: Revamped UI/UX, ClickPipes API and Terraform Provider, Prometheus metrics, and more!]() Original publisher: [Read original article]() Author: The ClickPipes Team Published: 2025-05-22T00:00:00Z Content type: release Language: en Sources: [ClickHouse Blog]() Topics: [User interface design](), [telemetry](), [CI/CD](), [Kafka](), [Amazon S3]() Tags: [amazon-s3](), [apache-kafka](), [api](), [automation](), [aws](), [ci-cd](), [clickhouse](), [infrastructure-as-code](), [integration](), [metrics](), [observability](), [prometheus](), [terraform](), [ui](), [ux]() ### AI overview ClickPipes adds a redesigned UI, editing for running pipelines, API and Terraform-based configuration management, AWS PrivateLink, and observability features including Prometheus metrics. ### Source excerpt ClickPipes evolves with a revamped UI/UX, new API and Terraform Provider, AWS PrivateLink integration, notification system, Prometheus metrics for observability, and centralized system tables for enhanced data pipeline management. ## Building a Native Binary for Apache Kafka on macOS DevFeed: [Building a Native Binary for Apache Kafka on macOS]() Original publisher: [Read original article]() Published: 2025-04-07T10:25:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [Kafka](), [macOS](), [Development](), [Testing]() Tags: [apache-kafka](), [building](), [development](), [kafka](), [macos](), [native](), [testing]() ### AI overview This tutorial explains how to build a self-contained native Apache Kafka binary for macOS using GraalVM and the configuration and tooling from Kafka's native container-image work. It covers obtaining the Kafka distribution, using the provided native-image configuration, and producing a macOS/AArch64 executable for development and testing. ### Source excerpt Table of Contents KIP-974: Docker Image for GraalVM based Native Kafka Broker With help of the GraalVM configuration developed for KIP-974 (Docker Image for GraalVM based Native Kafka Broker), you can easily build a self-contained native binary for Apache Kafka. Read on to learn how you can build a native Kafka executable yourself, starting in milli-seconds, making it a perfect fit for development and testing purposes. When I wrote about ahead-of-time class loading and linking in Java 24 recently, I also published the start-up time for Apache Kafka as a native binary for comparison. This was done via Docker, as there's no pre-built native binary of Kafka available for the operating system I'm running on, macOS. But there is a native Kafka container image, so this is what I chose for the sake of convenience. Now, running in a container adds a little bit of overhead of course, so it wasn't a surprise when Thomas Würthinger, lead of the GraalVM project at Oracle, brought up the question what the value would be when running Kafka natively on macOS. Needless to say I can't leave this kind of nice nerd snipe pass, so I set out to learn how to build a native Kafka binary on macOS, using GraalVM. ## KIP-932 explores queue semantics and Share Groups for Apache Kafka DevFeed: [KIP-932 explores queue semantics and Share Groups for Apache Kafka]() Original publisher: [Read original article]() Published: 2025-03-05T11:35:00Z Content type: opinion Language: en Sources: [Gunnar Morling]() Topics: [Kafka](), [Streaming](), [stream-processing](), [Architecture & Design]() Tags: [apache-kafka](), [fraud](), [fraud-detection](), [job](), [kafka](), [microservices](), [stream-processing](), [streaming]() ### AI overview The article examines KIP-932, which explores adding queue semantics to Apache Kafka through Share Groups. It explains how Kafka's partition-based consumer model limits parallelism and ordered processing, and why individual message acknowledgment and rejection are important for queueing workloads such as independent job processing. ### Source excerpt Table of Contents Towards Queue Support in Kafka--Introducing Share Groups Share Groups in Action Retry Behavior and State Management Share Group State Persistence Summary and Outlook In the "Let's Take a Look at...!" blog series I am going to explore interesting projects, developments and technologies in the data and streaming space. This can be KIPs and FLIPs, open-source projects, services, and more. The idea is to get some hands-on experience, learn about potential use cases and applications, and understand the trade-offs involved. If you think there's a specific subject I should take a look at, let me know in the comments below! That guy above? Yep, that's me, whenever someone says "Kafka queue". Because, that's not what Apache Kafka is. At its core, Kafka is a distributed durable event log. Producers write events to a topic, organized in partitions which are distributed amongst the brokers of a Kafka cluster. Consumers, organized in groups, divide the partitions they process amongst themselves, so that each partition of a topic is read by exactly one consumer in the group. ## Building an Event-Driven Architecture with Managed Data Services DevFeed: [Building an Event-Driven Architecture with Managed Data Services]() Original publisher: [Read original article]() Author: Jonathan Brown Published: 2024-08-15T01:52:00Z Content type: tutorial Language: en Sources: [Heroku]() Topics: [event driven](), [Kafka](), [data-processing](), [Microservice](), [Heroku](), [Scalability](), [Architecture & Design]() Tags: [apache-kafka](), [architecture](), [cloud-infrastructure](), [data](), [data-processing](), [engineering](), [event-driven](), [heroku](), [kafka](), [microservices](), [product-features](), [scalability](), [services]() ### AI overview This article explains event-driven architecture for applications that process real-time data through decoupled microservices and components. It introduces Apache Kafka and discusses how Apache Kafka on Heroku can support reliable, scalable event-driven applications, along with core concepts such as events, topics, and producers. ### Source excerpt Modern applications have an unceasing buzz of user activity and data flows. Users send a flurry of one-click reactions to social media posts. Wearable tech and other IoT sensors work nonstop to transmit event data from their environments. Meanwhile, customers on e-commerce sites perform shopping cart actions or product searches which can bring immediate impact [...] The post Building an Event-Driven Architecture with Managed Data Services appeared first on Heroku. ## Kafka for System Design Interviews DevFeed: [Kafka for System Design Interviews]() Original publisher: [Read original article]() Author: Evan King Published: 2024-07-01T03:18:09Z Content type: tutorial Language: en Sources: [Evan King]() Topics: [Kafka](), [Apache-Kafka](), [Event-Streaming](), [stream-processing](), [Scalability](), [data]() Tags: [apache-kafka](), [distributed](), [durability](), [google](), [high-performance](), [kafka](), [meta](), [open-source](), [real-time](), [scalability](), [stream-processing](), [system-design-interview]() ### AI overview A tutorial on Apache Kafka for system design interviews. It explains Kafka as an open-source distributed event streaming platform that can function as a message queue or stream processing system, then uses a real-time sports statistics example to discuss producers, consumers, scaling, and event ordering. ### Source excerpt By the Co-founder of www.hellointerview.comIntro There is a good chance you've heard of Kafka. It's popular. In fact, according to their website, it's used by 80% of the Fortune 100. If it's good enough to help scale the largest companies in the world, it's probably good enough for your next system design interview. In fact, it's one of the top 5 technologies we see used in design interviews. Apache Kafka is an open-source distributed event streaming platform that can be used either as a message queue or as a stream processing system. Kafka excels in delivering high performance, scalability, and durability. It's engineered to handle vast volumes of data in real-time, ensuring that no message is ever lost and that each piece of data is processed as swiftly as possible. In this deep dive, we're going to take a top down approach. Starting with a zoomed out view of Kafka and progressing into more and more detail. If you know the basics, feel free to skip ahead to the more advanced sections. A Motivating Example It's the World Cup (my personal favorite competition). And we run a website that provides real-time statistics on the matches. Each time a goal is scored, a player is booked, or a substitution is made, we want to update our website with the latest information. Events are placed on a queue when they occur. We call the server or process responsible for putting these events on the queue the producer. Downstream, we have a server that reads events off the queue and updates the website. We call this the consumer. Now, imagine the World Cup expanded from just the top 48 teams to a hypothetical 1,000-team tournament, and all the games are now played at the same time. The number of events has increased significantly, and our single server hosting the queue is struggling to keep up. Similarly, our consumer feels like it has its mouth under a firehose and is crashing under the load. We need to scale the system by adding more servers to distribute our queue. But how do we e ## Evolving the Backend Storage for Platform Metrics DevFeed: [Evolving the Backend Storage for Platform Metrics]() Original publisher: [Read original article]() Author: David Murray Published: 2024-05-09T22:00:00Z Content type: article Language: en Sources: [Heroku]() Topics: [Heroku](), [Back end](), [Time Series](), [Apache Cassandra](), [Kafka](), [stream-processing](), [autoscaling]() Tags: [apache-kafka](), [app-architecture](), [autoscaling](), [aws](), [backend](), [cassandra](), [cloud-infrastructure](), [data-analytics](), [database](), [developer-tools](), [engineering](), [heroku](), [kafka](), [metrics](), [performance-optimization](), [platform](), [platform-updates](), [stream-processing](), [time-series]() ### AI overview Heroku describes an infrastructure upgrade that migrated the storage backend for platform metrics. The article explains how MetaaS collects application observations, aggregates them into time-series statistics, and uses Kafka, stream-processing jobs, and Cassandra to support dashboard metrics, alerting, autoscaling, retention, and querying. ### Source excerpt One of our most important goals at Heroku is to be boring. Don't get us wrong, we certainly hope that you're excited about the Heroku developer experience -- as heavy users of Heroku ourselves, we certainly are! But, even more so, we hope that you don't have to spend all that much time thinking about [...] The post Evolving the Backend Storage for Platform Metrics appeared first on Heroku. ## Kafka 101 DevFeed: [Kafka 101]() Original publisher: [Read original article]() Author: ByteByteGo Published: 2024-05-09T18:55:21Z Content type: tutorial Language: en Sources: [High Scalability]() Topics: [Kafka](), [distributed-systems](), [Streaming](), [Architecture & Design](), [Data structures](), [Microservice](), [Persistence]() Tags: [apache-kafka](), [architecture](), [data-structure](), [distributed-systems](), [kafka](), [microservices](), [open-source](), [persistence](), [real-time-data-streaming]() ### AI overview This introductory article explains Apache Kafka's origins, distributed streaming architecture, and use as a central platform for coordinating data among services and downstream systems. It also describes Kafka's log-based storage model, including immutability, ordered records, concurrent reads, and optimization for high-throughput, cost-efficient HDD storage. ### Source excerpt This is a guest article by Stanislav Kozlovski, an Apache Kafka Committer. If you would like to connect with Stanislav, you can do so on Twitter and LinkedIn. Originally developed in LinkedIn during 2011, Apache Kafka is one of the most popular open-source Apache projects out ## Build a Real-time Materialized View from Postgres Changes using Confluent's ksqlDB DevFeed: [Build a Real-time Materialized View from Postgres Changes using Confluent's ksqlDB]() Original publisher: [Read original article]() Author: Evan Shortiss Published: 2024-02-28T17:33:42Z Content type: tutorial Language: en Sources: [Blog -- Neon Docs]() Topics: [Kafka](), [Replication](), [Streaming](), [real-time](), [Messaging](), [Microservice](), [data](), [Databases](), [Cloud](), [Architecture & Design](), [Publish-subscribe pattern](), [ide]() Tags: [apache-kafka](), [architectures](), [cloud](), [community](), [components](), [confluent](), [confluent-cloud](), [data](), [events](), [messaging](), [microservices](), [performance](), [postgres](), [real-time](), [replication](), [sql](), [streaming]() ### AI overview This tutorial explains how to stream changes from a Neon Postgres database to Apache Kafka and use ksqlDB on Confluent Cloud to build a real-time materialized view. It contrasts this approach with refreshing a Postgres materialized view after each write, highlighting performance, history retention, and reliable downstream notifications. ### Source excerpt Neon's support for Postgres' logical replication features opens up a variety of interesting use cases, including for real-time streaming architectures based on change data capture. We previously demonstrated how to use Debezium to fan-out changes from Postgres by using Redis as a... ## Logical Replication From Postgres 16 Stand-By Servers--Debezium and Failover Slots DevFeed: [Logical Replication From Postgres 16 Stand-By Servers--Debezium and Failover Slots]() Original publisher: [Read original article]() Published: 2023-12-19T00:00:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [Replication](), [Databases](), [Kafka](), [Streaming](), [Open Source](), [Amazon DocumentDB]() Tags: [amazon-rds](), [apache-kafka](), [debezium](), [failover](), [open-source](), [postgres](), [replication]() ### AI overview A tutorial on using Debezium to stream logical replication changes from PostgreSQL 16 standby servers, including failover scenarios and replication slots. It demonstrates a Docker Compose environment with Kafka and Kafka Connect, and discusses PostgreSQL deployments on Amazon RDS. ### Source excerpt Table of Contents Stand-By Logical Replication With Debezium Towards Fail-Over Slots Wrap-Up This post originally appeared on the Decodable blog. All rights reserved. Welcome back to this series about logical replication from Postgres 16 stand-by servers, in which we'll discuss how to use this feature with Debezium--a popular open-source platform for Change Data Capture (CDC) for a wide range of databases--as well as how to manage logical replication in case of failover scenarios, i.e. a situation where your primary Postgres server becomes unavailable and a stand-by server needs to take over. ## Getting Started With PyFlink on Kubernetes DevFeed: [Getting Started With PyFlink on Kubernetes]() Original publisher: [Read original article]() Published: 2023-12-07T00:00:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [apache-flink](), [Kubernetes](), [stream-processing](), [Kafka](), [Python](), [data-engineering]() Tags: [apache-kafka](), [flink](), [kafka](), [kubernetes](), [operator](), [programming](), [python](), [stream-processing]() ### AI overview A practical guide to running PyFlink on Kubernetes with Apache Flink 1.18. It covers installing the Flink Kubernetes Operator and Kafka with Strimzi, creating a PyFlink job that generates data and writes it to Kafka using Flink SQL, building a container image, and deploying the job to Kubernetes. ### Source excerpt Table of Contents What Is PyFlink and Why Should You Care? Prerequisites Installing the Flink Kubernetes Operator Installing Strimzi and Apache Kafka A Simple PyFlink Job Building a Container Image With Your PyFlink Job Deploying a PyFlink Job On Kubernetes This post originally appeared on the Decodable blog. All rights reserved. The other day, I wanted to get my feet wet with PyFlink. While there is a fair amount of related information out there, I couldn't find really up-to-date documentation on using current versions of PyFlink with Flink on Kubernetes. ## Heroku is Now Available to Purchase in AWS Marketplace DevFeed: [Heroku is Now Available to Purchase in AWS Marketplace]() Original publisher: [Read original article]() Author: Michael Friis Published: 2023-11-28T22:00:00Z Content type: release Language: en Sources: [Heroku]() Topics: [Heroku](), [aws-marketplace](), [Amazon Web Services]() Tags: [announce](), [apache-kafka](), [aws](), [aws-marketplace](), [dynos](), [heroku](), [heroku-connect](), [heroku-postgres](), [news](), [private-spaces](), [redis](), [update]() ### AI overview This Heroku release announcement describes its availability for purchase through AWS Marketplace Private Offers, including consolidated billing and provisioning for eligible U.S. AWS Enterprise Discount Program customers. An update states that Heroku is no longer available through that purchasing option. ### Source excerpt Update: We've updated our purchasing options, and Heroku is no longer available via AWS Marketplace Private Offers. You can continue to provision and manage your services directly through your Heroku account or via Heroku.com. Today at AWS re:Invent, we're excited to announce that Heroku is available for purchase in AWS Marketplace through the Private Offers [...] The post Heroku is Now Available to Purchase in AWS Marketplace appeared first on Heroku. ## When Change Data Capture Can Break Application Encapsulation DevFeed: [When Change Data Capture Can Break Application Encapsulation]() Original publisher: [Read original article]() Published: 2023-11-21T00:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Architecture & Design](), [Software Engineering](), [streaming-data-processing](), [Kafka](), [Database]() Tags: [apache-kafka](), [data](), [debezium](), [schema](), [stream-processing](), [streaming]() ### AI overview This article examines whether exposing database change event feeds through Change Data Capture (CDC) breaks application encapsulation. It explains that CDC can expose internal table models as event APIs and create schema and downstream-consumer risks, then discusses ways to address those risks. ### Source excerpt Table of Contents CDC--A Quick Primer Does CDC Break Encapsulation? Entering Data Contracts Implementation Approaches For Data Contracts The Outbox Pattern Stream Processing Streaming Data Contracts--Beyond the Basics Handling Schema Changes Summary This post originally appeared on the Decodable blog. All rights reserved. Having worked on Debezium--an open-source platform for Change Data Capture (CDC)--for several years, one concern I've heard repeatedly is this: aren't you breaking the encapsulation of your application when you expose change event feeds directly from your database? After all, CDC exposes your internal persistent data model to the outside world, which may have unintended consequences, e.g. in terms of data exposure but also when it comes to changes to the schema of your data, which may break downstream consumers. ## Heroku Private Spaces Global Expansion: Canada, India, Singapore, and the UK DevFeed: [Heroku Private Spaces Global Expansion: Canada, India, Singapore, and the UK]() Original publisher: [Read original article]() Author: Ethan Limchayseng Published: 2023-08-31T21:00:00Z Content type: release Language: en Sources: [Heroku]() Topics: [Heroku](), [spaces](), [Availability](), [data-governance](), [Network]() Tags: [apac](), [apache-kafka](), [aws](), [canada](), [cloud-infrastructure](), [data](), [dynos](), [heroku](), [heroku-shield](), [india](), [kafka](), [london](), [news](), [platform-updates](), [private-spaces](), [uk]() ### AI overview Heroku announces general availability for Private Spaces in India and Canada, plus new regions in the United Kingdom and Singapore. The expansion supports data-location control and includes services such as Heroku Postgres, Apache Kafka on Heroku, Heroku Data for Redis, Heroku Connect, and most Heroku Add-ons. ### Source excerpt In May 2023, we announced our limited release of two new Heroku Private Spaces regions: India (Mumbai) and Canada (Montreal). This month, we're announcing the full general availability of those two regions, along with new Heroku Private Spaces regions for the United Kingdom (London) and Singapore. This expansion enables customers to maintain greater control over [...] The post Heroku Private Spaces Global Expansion: Canada, India, Singapore, and the UK appeared first on Heroku. ## Heroku Private Spaces Expand to Mumbai and Montreal DevFeed: [Heroku Private Spaces Expand to Mumbai and Montreal]() Original publisher: [Read original article]() Author: Ethan Limchayseng Published: 2023-05-03T20:06:00Z Content type: release Language: en Sources: [Heroku]() Topics: [Heroku](), [spaces](), [Heroku Postgres](), [Kafka](), [Redis]() Tags: [apache-kafka](), [canada](), [customers](), [deploy](), [heroku](), [heroku-enterprise](), [heroku-postgres](), [india](), [news](), [private-spaces](), [product-features](), [redis](), [release](), [security](), [support]() ### AI overview Heroku announces a limited release of Private Spaces in Mumbai and Montreal. The new regions support Private Spaces and several related Heroku products, with access initially limited to existing Private Spaces customers accepted into the Limited Release program. ### Source excerpt This month, we're expanding the Heroku platform with a limited release of our Private Spaces product in two new regions, India (Mumbai) and Canada (Montreal), enabling customers to maintain even greater control over where data is stored and processed. These two new regions will fully support Heroku Private Spaces, Heroku Shield Private Spaces, Heroku Postgres, [...] The post Heroku Private Spaces Expand to Mumbai and Montreal appeared first on Heroku. [Next page]()