# apache-flink Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Lyft Moves Streaming Fleet to Apache Flink Kubernetes Operator DevFeed: [Lyft Moves Streaming Fleet to Apache Flink Kubernetes Operator]() Original publisher: [Read original article]() Author: Mark Silvester Published: 2026-09-16T11:00:00Z Content type: news Language: en Sources: [InfoQ]() Topics: [flink](), [apache-flink](), [Kubernetes](), [autoscaling](), [Deployment](), [legacy](), [Testing]() Tags: [apache-flink](), [architecture-design](), [autoscaling](), [deployment](), [development](), [devops](), [flink](), [kubernetes](), [kubernetes-operator](), [legacy](), [lyft-flink-k8s-operator](), [news](), [testing](), [upgrade]() ### AI overview Lyft moved hundreds of production Apache Flink jobs from its in-house Kubernetes operator to the Apache Flink Kubernetes Operator. The change enabled last-state upgrades, in-place autoscaling, and resource autotuning, while Lyft adapted legacy deployment specifications through its deploy API and contributed a fix for a configuration-renaming bug. ### Source excerpt Lyft has moved hundreds of production Flink jobs from a 2020 in-house Kubernetes operator to the Apache Flink Kubernetes Operator, unlocking last-state upgrades, in-place autoscaling and resource autotuning across the fleet. By Mark Silvester ## A Tale of Two Flink Autoscalers DevFeed: [A Tale of Two Flink Autoscalers]() Original publisher: [Read original article]() Author: Netflix Technology Blog Published: 2026-08-21T16:01:01Z Content type: article Language: en Sources: [Netflix](), [Netflix TechBlog - Medium]() Topics: [apache-flink](), [autoscaling](), [Netflix](), [stream-processing](), [Kafka](), [Open Source](), [telemetry](), [Network]() Tags: [apache-flink](), [autoscaling](), [aws](), [infrastructure](), [kafka](), [metrics](), [netflix](), [open-source](), [operational-excellence](), [scale](), [stream-processing](), [telemetry]() ### AI overview Netflix describes running two Apache Flink autoscalers: an older in-house system and a newer one from the Apache Flink community. The article explains why autoscaling is necessary for more than 30,000 Flink jobs across AWS regions, discusses workload variability and the cost of resizing stateful jobs, and introduces the metrics-driven design of the first autoscaler. ### Source excerpt Samuel Yeboah, Francesco Di Chiara and Mingliang Liu Today, Netflix runs two Flink autoscalers. That is exactly one more than we want. We built the first one in-house years ago, when there was no mature option suited to our platform. The second came from the Apache Flink community, and it can scale workloads our homegrown system was never designed for. We now run both in production and are steadily converging on the open-source one. Along the way we learned some hard lessons about metrics, cost, and the real price of maintaining infrastructure you could instead adopt, and we hope they are useful whether you run a handful of Flink jobs or tens of thousands. Why autoscaling is not optional at our scale Netflix has run stream processing on Apache Flink since 2017. As of 2026 we operate more than 30,000 Flink jobs across multiple AWS regions. Most are not deployed by hand; they are generated by our managed platform Data Mesh, so the majority of users never touch a Flink job directly. A smaller but growing set are custom jobs, built and operated by teams across the company for use cases like personalization, Ads, and Live events. They range from single-operator jobs that shuttle records between Kafka topics to stateful pipelines with branches, joins, and terabytes of state, and their load swings with daily cycles, launches, and regional failovers. Provisioning every one of those jobs for its peak is wasteful; provisioning for the average causes lag during surges. And in our platform a scaling action is not free: by default it means taking a savepoint, stopping the job gracefully, and restarting it at the new size, which for a large stateful job can take minutes. That leaves a genuinely hard question: how do you give each job the resources it needs, when it needs them, without a human in the loop and without breaking anything? The first autoscaler: watching from outside Our first answer, built around 2019, was an autoscaler shaped like a stream-processing job. It ran on M ## New in Confluent Intelligence and AI Tools: Making Agents Native to the Stream, Expanded Model Support, New Agent Skills, and Copilot DevFeed: [New in Confluent Intelligence and AI Tools: Making Agents Native to the Stream, Expanded Model Support, New Agent Skills, and Copilot]() Original publisher: [Read original article]() Author: Confluent Staff Published: 2026-08-18T14:00:10Z Content type: article Language: en Sources: [Confluent: Data in motion]() Topics: [Artificial Intelligence](), [apache-flink](), [Kafka](), [streaming-data-processing](), [real-time](), [MCP Server](), [Agent Skills](), [MCP](), [Inference](), [Google](), [ibm]() Tags: [ai](), [ai-ready-data](), [ai-tools](), [apache-flink](), [confluent](), [confluent-cloud](), [google](), [ibm](), [kafka](), [mcp](), [mcp-server](), [real-time](), [skills](), [streaming-data-processing](), [support](), [tools]() ### AI overview Confluent announces updates to Confluent Intelligence and related AI tools for building production AI systems on Apache Kafka and Apache Flink. The release adds expanded time-series model support, a generally available Real-Time Context Engine, updates to the fully managed MCP Server, Agent Skills for AI coding assistants, and Confluent Copilot. The features are intended to provide agents and applications with fresh business context and governed access to live data and infrastructure. ### Source excerpt Explore new AI features and AI tools: support for IBM Granite Time Series models and TimesFM models (EA), enhanced Real-Time Context Engine experience, new Agent Skills, Confluent Copilot ## Confluent Cloud for Apache Flink: Engine for Mission-Critical, Real-Time Operational Systems and dbt/SQL-Native Home for Data Science and AI DevFeed: [Confluent Cloud for Apache Flink: Engine for Mission-Critical, Real-Time Operational Systems and dbt/SQL-Native Home for Data Science and AI]() Original publisher: [Read original article]() Author: Yashwanth Dasari Published: 2026-08-18T02:20:00Z Content type: release Language: en Sources: [Confluent: Data in motion]() Topics: [Confluent Cloud](), [apache-flink](), [stream-processing](), [Kafka](), [AI, ML & Data Engineering](), [Cloud](), [SDKs]() Tags: [ai](), [announcement](), [apache-flink](), [api](), [batch](), [confluent-cloud](), [data-science](), [developers](), [kafka](), [real-time]() ### AI overview Confluent announces an evolution of Confluent Cloud for Apache Flink that unifies mission-critical real-time operations with analytics and AI workflows. The release adds a serverless, co-designed Kafka and Flink platform, including the generally available Flink Table API in Java for code-first development. ### Source excerpt Flink now acts as a robust engine for developers through Table API, UDFs, and PTFs while offering a SQL-native, dbt-integrated platform for data science and AI teams. ## Announcing Confluent Platform 8.3: Powerful Apache Flink® SQL operations, Easier KRaft Migrations, Expanded Monitoring and more. DevFeed: [Announcing Confluent Platform 8.3: Powerful Apache Flink® SQL operations, Easier KRaft Migrations, Expanded Monitoring and more.]() Original publisher: [Read original article]() Author: Premika Srinivasan Published: 2026-07-29T15:00:10Z Content type: release Language: en Sources: [Confluent: Data in motion]() Topics: [apache-flink](), [Kafka](), [Monitoring](), [data-governance](), [Model Context Protocol](), [Claude Code](), [codex]() Tags: [apache-flink](), [apache-kafka](), [claude-code](), [codex](), [confluent](), [confluent-platform](), [data-governance](), [mcp](), [mcp-server](), [monitoring]() ### AI overview Confluent Platform 8.3.0, built on Apache Kafka 4.3.0, adds simplified Apache Flink SQL operations, expanded monitoring through Unified Stream Manager, easier KRaft migration, and structured governance for data in motion. The release also introduces an MCP server for Confluent Platform for Apache Flink, allowing AI agents such as Claude Code and Codex to inspect and manage Flink resources through existing security controls. ### Source excerpt Announcing Confluent Platform 8.3: Powerful Apache Flink® SQL operations, Easier KRaft Migrations, Expanded Monitoring and more ## Announcing Apache Iceberg 1.11.0 DevFeed: [Announcing Apache Iceberg 1.11.0]() Original publisher: [Read original article]() Author: Google Open Source (noreply@blogger.com) Published: 2026-05-27T18:30:00Z Content type: release Language: en Sources: [Google Open Source Blog]() Topics: [Apache Iceberg](), [Apache Spark](), [apache-flink](), [schema-evolution](), [Streaming](), [Encryption](), [Cryptography]() Tags: [apache-flink](), [apache-iceberg](), [apache-spark](), [encryption](), [kms](), [lakehouse](), [releases](), [schema-evolution](), [streaming]() ### AI overview Apache Iceberg 1.11.0 adds support for Apache Spark 4.1 and Apache Flink 2.1, makes both default build targets, and introduces changes including server-side REST catalog scan planning, partition statistics APIs, built-in envelope encryption with Google KMS support, and Google Storage Analytics integration. ### Source excerpt by Alex Stephen & Talat Uyarer, Lakehouse Apache Iceberg project has just launched version 1.11.0! A lot has happened since the last version. Iceberg 1.11.0 adds support for Apache Spark 4.1 and Apache Flink 2.1, the latest releases of the two engines and makes both the default build targets The rest are more structural. The REST catalog learns to plan scans server-side, shifting metadata work off the query engine. A new partition statistics scan API gives optimizers a clean, supported way to read a table's shape. Built-in table encryption arrives with envelope encryption and Google KMS support. And Google Storage Analytics library integration makes your Iceberg workloads faster than before. Let's take a look at some of the biggest changes. Spark & Flink Updates As Spark and Flink are moving forward, the 1.11.0 release is pushing forward for new version support in both. Spark 4.1 & DSv2 Migration: Spark 4.1 unlocks is MERGE INTO with automatic schema evolution: Spark's newer MERGE syntax accepts a WITH SCHEMA EVOLUTION clause, so a MERGE whose source carries columns the target table lacks can add those columns to the table within the same statement, with no separate ALTER TABLE round trip. Beyond the version bump, the 1.11 Spark connector also modernizes against Spark's newer DataSource V2 APIs and adds an asynchronous micro-batch planner that speeds up Structured Streaming. Flink Ecosystem Updates: Initial work for Flink 2.1 support has landed in the core repository, continuing Iceberg's promise of providing first-class, low-latency streaming sink capabilities. The centerpiece of the Flink work is the DynamicIcebergSink, an experimental sink that breaks the old one-sink-per-table model: a single sink routes each record to a table chosen at runtime, creating tables on demand and evolving their schemas and partition specs on the fly as the input changes including dropping columns once you opt in with dropUnusedColumns. In addition to DynamicIcebergSInk work Flink sta ## Building AI agents as streaming SQL queries with Apache Flink DevFeed: [Building AI agents as streaming SQL queries with Apache Flink]() Original publisher: [Read original article]() Published: 2025-06-18T13:30:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [AI Agent](), [SQL](), [Streaming](), [apache-flink](), [event driven](), [Scalability](), [Large Language Model](), [Developer experience]() Tags: [ai](), [ai-agent](), [apache-flink](), [developer-experience](), [event-driven](), [llms](), [scalability](), [sql](), [streaming]() ### AI overview This article explores whether AI agents can be built as streaming SQL queries. It presents Apache Flink and stream-processing concepts as potential foundations for event-driven, scalable, consistent, low-latency agentic systems, while noting that many deployed agents are defined AI-assisted workflows. ### Source excerpt Table of Contents Agents Need to Interact With LLMs Agents Should Be Event-Driven Agents Need Context Agents Require Memory When SQL Is Not Enough Parting Thoughts AI Agents have improved in leaps and bounds in recent times, moving beyond simple chatbots to sophisticated, autonomous systems. This post explores a novel approach to building agentic systems: using the power of streaming SQL queries. Discover how platforms like Apache Flink can transform the development of AI Agents, offering benefits in consistency, scalability, and developer experience. ## Backfilling Postgres TOAST Columns in Debezium Data Change Events DevFeed: [Backfilling Postgres TOAST Columns in Debezium Data Change Events]() Original publisher: [Read original article]() Published: 2025-05-26T14:40:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Databases](), [Replication](), [apache-flink](), [stream-processing](), [SQL](), [API]() Tags: [apache-flink](), [data](), [debezium](), [debezium-connector](), [flink](), [postgres](), [replication](), [sql](), [stateful](), [stream-processing]() ### AI overview This article explains how Postgres logical replication handles TOAST columns in Debezium change events. It discusses Debezium's reselect postprocessor and explores Apache Flink solutions using stateful stream processing, Flink SQL, and process table functions. ### Source excerpt Table of Contents Debezium Reselect Postprocessor Flink DataStream API Flink SQL With OVER Aggregation Flink Process Table Functions Summary and Discussion Postgres logical replication, while powerful for capturing real-time data changes, presents challenges with TOAST columns, whose values can be absent from data change events in specific situations. This post discusses how Debezium addresses this through its built-in reselect post processor, then explores more robust solutions leveraging Apache Flink's capabilities for stateful stream processing, including Flink SQL and the brand-new process table functions (PTFs) in Flink 2.1. ## A Deep Dive Into Ingesting Debezium Events From Kafka With Flink SQL DevFeed: [A Deep Dive Into Ingesting Debezium Events From Kafka With Flink SQL]() Original publisher: [Read original article]() Published: 2025-04-16T09:25:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [apache-flink](), [Kafka](), [Apache-Kafka]() Tags: [changelog](), [connectors](), [data](), [debezium](), [flink](), [how-to](), [kafka](), [schema](), [streams]() ### AI overview This tutorial explains how to ingest Debezium change events from Kafka into Apache Flink using Flink SQL. It compares the Apache Kafka SQL Connector and Upsert Kafka SQL Connector, discusses JSON and Avro formats, and explains how connector and format choices determine append-only or changelog semantics. ### Source excerpt Table of Contents Flink SQL Connectors for Apache Kafka The Apache Kafka SQL Connector in Append-Only Mode The Apache Kafka SQL Connector As a Changelog Source The Upsert Kafka SQL Connector Summary Over the years, I've spoken quite a bit about the use cases for processing Debezium data change events with Apache Flink, such as metadata enrichment, building denormalized data views, and creating data contracts for your CDC streams. One detail I haven't covered in depth so far is how to actually ingest Debezium change events from a Kafka topic into Flink, in particular via Flink SQL. Several connectors and data formats exist for this, which can make things somewhat confusing at first. So let's dive into the different options and the considerations around them! ## JEP 483: Ahead-of-Time Class Loading and Linking in Java 24 DevFeed: [JEP 483: Ahead-of-Time Class Loading and Linking in Java 24]() Original publisher: [Read original article]() Published: 2025-03-27T13:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Java](), [openjdk](), [Kafka](), [apache-flink]() Tags: [building](), [java](), [jvm](), [openjdk](), [performance]() ### AI overview This article examines JEP 483, which loads and links Java classes ahead of time and caches the results to reduce application startup time. It explains the feature's role in Project Leyden, its relationship to AppCDS, the training-run and classpath requirements, and examples using Apache Kafka and Apache Flink. ### Source excerpt Table of Contents Building an AOT Cache for Apache Kafka AOT Caching With Apache Flink Summary In the "Let's Take a Look at...!" blog series I am exploring interesting projects, developments and technologies in the data and streaming space. This can be KIPs and FLIPs, open-source projects, services, relevant improvements to Java and the JVM, and more. The idea is to get some hands-on experience, learn about potential use cases and applications, and understand the trade-offs involved. If you think there's a specific subject I should take a look at, let me know in the comments below. Java 24 got released last week, and what a meaty release it is: more than twenty Java Enhancement Proposals (JEPs) have been shipped, including highlights such as compact object headers (JEP 450, I hope to spend some time diving into that one some time soon), a new class-file API (JEP 484), and more flexible constructor bodies (JEP 492, third preview). One other JEP which might fly a bit under the radar is JEP 483 ("Ahead-of-Time Class Loading & Linking"). It promises to reduce the start-up time of Java applications without requiring any modifications to the application itself, what's not to be liked about that? Let's take a closer look! ## Get Running with Apache Flink on Kubernetes, part 2 of 2 DevFeed: [Get Running with Apache Flink on Kubernetes, part 2 of 2]() Original publisher: [Read original article]() Published: 2025-01-28T00:00:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [apache-flink](), [Kubernetes](), [Availability](), [observability](), [Deployment](), [container](), [MinIO](), [Amazon S3](), [rocksdb](), [GitHub]() Tags: [apache-flink](), [availability](), [container-images](), [deployment](), [flink](), [github](), [kubernetes](), [minio](), [observability](), [rocksdb](), [s3](), [ui]() ### AI overview Part two of a tutorial series on running Apache Flink jobs on Kubernetes. It explains checkpointing, job manager high availability, savepoints, external object storage, incremental RocksDB checkpoints, observability, and related job-management practices. ### Source excerpt Table of Contents Fault Tolerance and High Availability Manually Triggering Savepoints Observability Bonus: Managing Flink Jobs With the Heimdall UI Summary and Discussion This post originally appeared on the Decodable blog. All rights reserved. Welcome back to this two-part blog post series about running Apache Flink on Kubernetes, using the Flink Kubernetes operator. In part one, we discussed installation and setup of the operator, different deployment types, how to deploy Flink jobs using custom Kubernetes resources, and how to create container images for your own Flink jobs. In this part, we'll focus on aspects such as fault tolerance and high availability of your Flink jobs running on Kubernetes, savepoint management, observability, and more. You can find the complete source code for all the examples shown in this series in the Decodable examples repository on GitHub: on GitHub. ## Get Running with Apache Flink on Kubernetes, part 1 of 2 DevFeed: [Get Running with Apache Flink on Kubernetes, part 1 of 2]() Original publisher: [Read original article]() Published: 2025-01-21T00:00:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [apache-flink](), [Kubernetes](), [Deployment](), [Provisioning](), [container images]() Tags: [apache-flink](), [container-images](), [deployment](), [installation](), [kubernetes](), [operator](), [provisioning]() ### AI overview A two-part tutorial on deploying Apache Flink jobs on Kubernetes using the upstream Flink Kubernetes Operator. Part one covers installation, setup, custom-resource deployment, and building container images; part two addresses fault tolerance, high availability, savepoints, observability, and UI access. ### Source excerpt Table of Contents Installation and Setup Deployment Types Deploying Your First Flink Job on Kubernetes Building Custom Job Images This post originally appeared on the Decodable blog. All rights reserved. Kubernetes is a widely used deployment platform for Apache Flink. While Flink has had native support for Kubernetes for quite a while, it is in particular the operator pattern which makes deploying Flink jobs onto Kubernetes clusters a compelling option: you define jobs in a declarative resource, and a control loop running in a component called a Kubernetes operator takes care of provisioning and maintaining (e.g. scaling, updating) all the required resources. Automation is the keyword here, significantly reducing the manual effort required for running Flink jobs in production. ## Getting Started With PyFlink on Kubernetes DevFeed: [Getting Started With PyFlink on Kubernetes]() Original publisher: [Read original article]() Published: 2023-12-07T00:00:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [apache-flink](), [Kubernetes](), [stream-processing](), [Kafka](), [Python](), [data-engineering]() Tags: [apache-kafka](), [flink](), [kafka](), [kubernetes](), [operator](), [programming](), [python](), [stream-processing]() ### AI overview A practical guide to running PyFlink on Kubernetes with Apache Flink 1.18. It covers installing the Flink Kubernetes Operator and Kafka with Strimzi, creating a PyFlink job that generates data and writes it to Kafka using Flink SQL, building a container image, and deploying the job to Kubernetes. ### Source excerpt Table of Contents What Is PyFlink and Why Should You Care? Prerequisites Installing the Flink Kubernetes Operator Installing Strimzi and Apache Kafka A Simple PyFlink Job Building a Container Image With Your PyFlink Job Deploying a PyFlink Job On Kubernetes This post originally appeared on the Decodable blog. All rights reserved. The other day, I wanted to get my feet wet with PyFlink. While there is a fair amount of related information out there, I couldn't find really up-to-date documentation on using current versions of PyFlink with Flink on Kubernetes. ## CDC Use Cases: 7 Ways to Put CDC to Work DevFeed: [CDC Use Cases: 7 Ways to Put CDC to Work]() Original publisher: [Read original article]() Published: 2023-11-02T00:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [data-engineering](), [streaming-data-processing](), [Kafka](), [apache-flink](), [MySQL](), [Microservice](), [migration]() Tags: [apache-flink](), [data](), [data-engineering](), [debezium](), [engineering](), [kafka](), [migration](), [mysql](), [real-time]() ### AI overview This article explains change data capture (CDC), focusing on log-based CDC and its low-latency, resource-efficient handling of database changes. It introduces Debezium and describes how CDC works with data streaming and stream-processing tools such as Apache Kafka and Apache Flink, before outlining seven common use cases. ### Source excerpt Table of Contents What is CDC? CDC Tools Analytics Data Platforms Application Caches Full-Text Search Audit Logs Continuous Queries Microservices Data Exchange Monolith-to-Microservices Migration Summary This post originally appeared on the Decodable blog. All rights reserved. Change Data Capture (CDC) is a powerful tool in data engineering and has seen a tremendous uptake in organizations of all kinds over the last few years. This is because it enables the tight integration of transactional databases into many other systems in your business at a very low latency. ## Building Apache Flink with Maven: Resolving HTTP Repository Dependency Issues DevFeed: [Building Apache Flink with Maven: Resolving HTTP Repository Dependency Issues]() Original publisher: [Read original article]() Published: 2022-12-18T12:45:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [Maven](), [apache-flink](), [Security]() Tags: [apache](), [apache-flink](), [build-tool](), [dependencies](), [security](), [stream-processing]() ### AI overview This article describes issues encountered while building Apache Flink from source with Maven 3.8.6. It explains how dependency-defined HTTP repositories can be blocked by Maven 3.8.1 and later, and discusses tracing the problematic dependency and overriding the repository URL in settings.xml. ### Source excerpt As part of my new job at Decodable, I am also planning to contribute to the Apache Flink project (as Decodable's fully-managed stream processing platform is based on Flink). Right now, I am in the process of familiarizing myself with the Flink code base, and as such I am of course building the project from source, too. ## 3 (More) Tips for Optimizing Apache Flink Applications DevFeed: [3 (More) Tips for Optimizing Apache Flink Applications]() Original publisher: [Read original article]() Author: Kevin Lam Published: 2022-12-13T14:00:00Z Content type: tutorial Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [apache-flink](), [data-processing](), [Streaming](), [data]() Tags: [apache-flink](), [backpressure](), [best-practices](), [parallelism](), [performance](), [shopify](), [stateful]() ### AI overview Shopify presents additional best practices for optimizing large, stateful Apache Flink applications. The supplied text covers selecting appropriate parallelism to scale execution and improve resource utilization, and avoiding bottlenecks in data sinks that can reduce throughput and cause backpressure. ### Source excerpt Earlier this year, we shared our tips for optimizing large stateful Apache Flink applications. Below we'll walk you through 3 more best practices. ## Investigating WAL Growth from an Inactive Postgres Replication Slot on Amazon RDS DevFeed: [Investigating WAL Growth from an Inactive Postgres Replication Slot on Amazon RDS]() Original publisher: [Read original article]() Published: 2022-11-30T13:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Replication](), [Amazon RDS](), [apache-flink](), [Database]() Tags: [amazon](), [amazon-rds](), [apache-flink](), [database](), [debezium](), [disk-space](), [postgres](), [replication](), [storage]() ### AI overview This article investigates how an inactive Postgres replication slot on Amazon RDS can retain WAL segments and consume disk space even when the database appears idle. It reproduces the behavior and compares replication-slot LSN differences with RDS storage and transaction-log metrics. ### Source excerpt Table of Contents The Observation The Solution Take Away While working on a demo for processing change events from Postgres with Apache Flink, I noticed an interesting phenomenon: A Postgres database which I had set up for that demo on Amazon RDS, ran out of disk space. The machine had a disk size of 200 GiB which was fully used up in the course of less than two weeks. Now a common cause for this kind of issue are replication slots which are not advanced: in that case, Postgres will hold on to all WAL segments after the latest log sequence number (LSN) which was confirmed for that slot. Indeed I had set up a replication slot (via the Decodable CDC source connector for Postgres, which is based on Debezium). I then had stopped that connector, causing the slot to become inactive. The problem was though that I was really sure that there was no traffic in that database whatsoever! What could cause a WAL growth of ~18 GB/day then? ## Reducing BigQuery Costs: How We Fixed A $1 Million Query DevFeed: [Reducing BigQuery Costs: How We Fixed A $1 Million Query]() Original publisher: [Read original article]() Author: Calvin Zhou Published: 2022-11-03T18:14:51Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [BigQuery](), [Streaming](), [apache-flink](), [Kafka](), [parquet](), [Google Cloud Platform (GCP)](), [Shopify](), [rocksdb]() Tags: [apache-flink](), [bigquery](), [cloud-storage](), [cost](), [kafka](), [parquet](), [pipelines](), [rocksdb](), [shopify](), [sql]() ### AI overview The article describes how a Shopify team discovered that a BigQuery query could cost nearly $1 million USD per month while building a marketing data pipeline. The team used Apache Flink with RocksDB and Apache Kafka, then evaluated BigQuery as an external data warehouse capable of querying large datasets and exporting results to Google Cloud Storage. The supplied text ends as the team encounters the cost problem, before detailing the eventual fix. ### Source excerpt We discovered a query that nearly cost us $1 million USD a month in BigQuery. Below, we'll share our tips for lowering costs in BigQuery. ## Tiny Telematics: Tracking my truck's location offline with a Raspberry Pi, redis, Kafka, and Flink (Part 2) DevFeed: [Tiny Telematics: Tracking my truck's location offline with a Raspberry Pi, redis, Kafka, and Flink (Part 2)]() Original publisher: [Read original article]() Author: Christian Hollinger Published: 2022-09-08T00:00:00Z Content type: tutorial Language: en Sources: [Christian Hollinger]() Topics: [Raspberry Pi](), [Kafka](), [apache-flink](), [distributed-systems](), [Back end](), [Linux](), [USB]() Tags: [apache-flink](), [backend](), [distributed-systems](), [dongle](), [flink](), [functional-programming](), [gps](), [kafka](), [linux](), [raspberry-pi](), [redis](), [scala](), [streaming](), [telematics](), [types]() ### AI overview Part 2 of a project that tracks a truck's location offline with a Raspberry Pi and GPS dongle. It sets up backend infrastructure, builds a Kafka and Apache Flink streaming pipeline, persists location data, and tests the system on the road. ### Source excerpt Tracking vehicle location offline with a Raspberry Pi, Part 2: Apache Flink, scala, Kafka, and road-testing. ## 7 Tips For Optimizing Apache Flink Applications DevFeed: [7 Tips For Optimizing Apache Flink Applications]() Original publisher: [Read original article]() Author: Yaroslav Tkachenko Published: 2022-03-08T21:20:39Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [apache-flink](), [Shopify](), [Kubernetes](), [rocksdb](), [Streaming](), [Tooling](), [cpu]() Tags: [apache-flink](), [cpu](), [data](), [debugging](), [jvm](), [kubernetes](), [performance](), [profiling](), [rocksdb](), [shopify](), [streaming]() ### AI overview This Shopify Engineering article presents lessons for optimizing large, stateful Apache Flink applications. It discusses profiling and debugging tools, performance tuning, resiliency, high availability, Kubernetes deployment, RocksDB state management, and checkpoint and savepoint storage in Google Cloud Storage. ### Source excerpt We've spent a lot of time on performance tuning our Apache Flink application. We'll walk you through key lessons for optimizing large stateful Apache Flink applications. ## Scaling Shopify's BFCM Live Map: An Apache Flink Redesign DevFeed: [Scaling Shopify's BFCM Live Map: An Apache Flink Redesign]() Original publisher: [Read original article]() Author: Berkay Antmen Published: 2021-12-10T19:00:00Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [Shopify](), [apache-flink](), [Data Infrastructure](), [Scalability](), [streaming-data-processing]() Tags: [apache-flink](), [data-infrastructure](), [scalability](), [shopify](), [stateful](), [stream-processing](), [streaming]() ### AI overview Shopify Data Platform Engineering redesigned the infrastructure behind the BFCM live map with Apache Flink to support more than 1.7 million merchants, richer insights, higher data volume, and higher uptime without manual intervention. ### Source excerpt A deep dive into how Shopify Data revamped the data infrastructure powering our BFCM live map using Apache Flink. ## A Data Engineering Perspective on Go vs. Python (Part 2 - Dataflow) DevFeed: [A Data Engineering Perspective on Go vs. Python (Part 2 - Dataflow)]() Original publisher: [Read original article]() Author: Christian Hollinger Published: 2020-07-06T00:00:00Z Content type: article Language: en Sources: [Christian Hollinger]() Topics: [data-engineering](), [Go Language](), [Python](), [SDKs](), [AI, ML & Data Engineering](), [datasets](), [Google Cloud Platform (GCP)](), [etl](), [apache-flink](), [Apache Spark]() Tags: [apache-flink](), [apache-spark](), [beam](), [benchmarking](), [big-data](), [cloud-dataflow](), [data-engineering](), [dataflow](), [distributed](), [etl](), [gcp](), [go](), [golang](), [google-cloud](), [open-source](), [performance](), [programming](), [python](), [spark]() ### AI overview A comparison of Go and Python for data engineering using Apache Beam and Google Dataflow. It examines their SDK differences, drawbacks, benchmark performance, and the feasibility of switching between them. ### Source excerpt In Part 2 of our comparison of Python and go from a Data Engineering perspective, we'll finally take a look at Apache Beam and Google Dataflow and how the go SDK and the Python SDK differ, what drawbacks we're dealing with, how fast it is by running extensive benchmarks, and how feasible it is to make the switch ## Running Apache Flink on Amazon Elastic Mapreduce DevFeed: [Running Apache Flink on Amazon Elastic Mapreduce]() Original publisher: [Read original article]() Published: 2016-01-06T08:15:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [apache-flink](), [Amazon Web Services](), [Hadoop](), [Scalding](), [data-processing](), [Amazon S3](), [Apache Spark]() Tags: [apache-flink](), [aws](), [data-processing](), [ec2](), [emr](), [flink](), [hadoop](), [hdfs](), [linux](), [realtime](), [s3](), [scala](), [scalding](), [sclading](), [spark](), [streaming]() ### AI overview A tutorial on running Apache Flink jobs on Amazon EMR, including adapting Scalding jobs and launching and accessing an EMR cluster. It also discusses EMRFS support for reading data from Amazon S3. ### Source excerpt I love really Amazon EMR. Over the years it's grown from being "Hadoop on-demand" to a full-fledged cluster management system for running OSS big-data apps (Hadoop MR of course, but also Spark, Hue, Hive, Pig, Oozie and more). While Hadoop out of the box supports reading from S3, EMR has a proprietary implementation called EMRFS that has some nice features. For those reasons, it's really the best Hadoop cluster to use if you're storing your data in S3. Lately I've been experimenting a lot with Apache Flink to replace MR as the excution fabric. At work, we have many, many jobs written in Scalding. Flink can execute Scalding jobs with some very simple modifications which was a great way to move our jobs from MR to a more memory-centric data processing model. However, we really wanted to run our jobs on EMR using Flink. Flink is not an option for EMR (yet) but can we still get our jobs to run? Let's see! Start an EMR cluster The first thing we need is an EMR cluster. You can launch a small test cluster for very cheap. Once it's running, let's ssh onto it and see what's going on. ihummel at mm-mac-3270 in ~ $ ssh hadoop@ec2-54-226-25-85.compute-1.amazonaws.com Last login: Wed Jan 6 16:33:01 2016 from 172.85.47.138 __| __|_ ) _| ( / Amazon Linux AMI ___|\___|___| https://aws.amazon.com/amazon-linux-ami/2015.09-release-notes/ 23 package(s) needed for security, out of 49 available Run "sudo yum update" to apply all updates. EEEEEEEEEEEEEEEEEEEE MMMMMMMM MMMMMMMM RRRRRRRRRRRRRRR E::::::::::::::::::E M:::::::M M:::::::M R::::::::::::::R EE:::::EEEEEEEEE:::E M::::::::M M::::::::M R:::::RRRRRR:::::R E::::E EEEEE M:::::::::M M:::::::::M RR::::R R::::R E::::E M::::::M:::M M:::M::::::M R:::R R::::R E:::::EEEEEEEEEE M:::::M M:::M M:::M M:::::M R:::RRRRRR:::::R E::::::::::::::E M:::::M M:::M:::M M:::::M R:::::::::::RR E:::::EEEEEEEEEE M:::::M M:::::M M:::::M R:::RRRRRR::::R E::::E M:::::M M:::M M:::::M R:::R R::::R E::::E EEEEE M:::::M MMM M:::::M R:::R R::::R EE:::::EEEEEEEE::::E ## Running Scalding jobs on Apache Flink DevFeed: [Running Scalding jobs on Apache Flink]() Original publisher: [Read original article]() Published: 2015-12-20T08:15:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [Scalding](), [apache-flink](), [Scala](), [Streaming](), [Hadoop]() Tags: [apache-flink](), [flink](), [hadoop](), [hdfs](), [realtime](), [scala](), [scalding](), [sclading](), [streaming]() ### AI overview This tutorial explains how to run Scalding jobs on Apache Flink. It outlines the required compatibility work, including Cascading 3 support, a Scalding version compiled against Cascading 3, selection of hadoop2-flink as the Cascading fabric, and a fix for a Twitter Chill bug. It also provides a sample Scala word-count job and build configuration changes. ### Source excerpt My previous post showed a very simple Scalding workflow. Apache Flink is a real time streaming framework that's very promising. It also supports running Cascading workflows with very little modification. Surely there must be some way to run a Scalding job on top of Flink? Turns out... YES! In a nutshell Here are the high-level things we need to solve for We need a Scalding job to test this out with cascading-flink requires Cascading 3 We need a new version of Scalding - Compiled against Cascading 3 - Allows hadoop2-flink to be selected as the Cascading "fabric" There is a bug in Twitter Chill We need Flink packaged with the Chill fix Scalding job Let's start with a very simple Scalding job. You can download https://github.com/themodernlife/simple-scalding-example for some inspiration. package net.themodernlife import com.twitter.scalding._ class WordCount(args: Args) extends Job(args) { def tokenize(text: String): Array[String] = { text.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]", "").split("\\s+") } val input = args("input") val output = args("output") TextLine(args("input")) .flatMap[String, String]('line -> 'word)(tokenize) .groupBy('word)(_.size) .write(Tsv(output)) } We'll be making some updates to our build.sbt. Here's what we're starting with organization := "net.themodernlife" name := "simple-scalding-example" scalaVersion := "2.11.7" scalacOptions ++= Seq("-encoding", "utf-8", "-deprecation", "-unchecked", "-feature") resolvers ++= Seq( "Concurrent Maven Repo" at "http://conjars.org/repo", "Twitter Maven Repo" at "http://maven.twttr.com" ) libraryDependencies ++= Seq( "com.twitter" %% "scalding-core" % "0.15.0", "org.apache.hadoop" % "hadoop-client" % "2.2.0" % "provided", "org.slf4j" % "slf4j-log4j12" % "1.7.13" % "provided" ) A new Scalding build We need to update Scalding Apply https://github.com/twitter/scalding/pull/1446 Apply/hack https://github.com/twitter/scalding/pull/1220 Hack Build.scala to ignore some submodules According to http://www.cascading.org/2