# apache-flink Published articles for apache-flink. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Lyft Moves Streaming Fleet to Apache Flink Kubernetes Operator DevFeed: [Lyft Moves Streaming Fleet to Apache Flink Kubernetes Operator]() Original publisher: [Read original article]() Author: Mark Silvester Published: 2026-09-16T11:00:00Z Content type: news Language: en Sources: [InfoQ]() Topics: [flink](), [apache-flink](), [Kubernetes](), [autoscaling](), [Deployment](), [legacy](), [Testing]() Tags: [apache-flink](), [architecture-design](), [autoscaling](), [deployment](), [development](), [devops](), [flink](), [kubernetes](), [kubernetes-operator](), [legacy](), [lyft-flink-k8s-operator](), [news](), [testing](), [upgrade]() ### AI overview Lyft moved hundreds of production Apache Flink jobs from its in-house Kubernetes operator to the Apache Flink Kubernetes Operator. The change enabled last-state upgrades, in-place autoscaling, and resource autotuning, while Lyft adapted legacy deployment specifications through its deploy API and contributed a fix for a configuration-renaming bug. ### Source excerpt Lyft has moved hundreds of production Flink jobs from a 2020 in-house Kubernetes operator to the Apache Flink Kubernetes Operator, unlocking last-state upgrades, in-place autoscaling and resource autotuning across the fleet. By Mark Silvester ## Delivering Real-Time Personalization with Databricks and Redis DevFeed: [Delivering Real-Time Personalization with Databricks and Redis]() Original publisher: [Read original article]() Author: Philip Laussermair, Anant Pingle Published: 2026-09-08T00:00:00Z Content type: article Language: en Sources: [Redis Blog]() Topics: [data]() Tags: [analytics](), [apache-flink](), [batch](), [data](), [databricks](), [e-commerce](), [performance](), [personalization](), [real-time](), [redis](), [spark](), [streaming](), [tech]() ### AI overview The article explains how Databricks Real-Time Mode and Redis support low-latency personalization by continuously processing event streams and serving fresh results quickly. ### Source excerpt Why real-time matters A customer is browsing an e-commerce site. They search for running shoes, open a product, read reviews, and add an item to the cart. Every one of those actions is a signal about what they want right now. If the homepage they lan... ## Rerouting the Stream: How Lyft Moved to the Apache Flink Operator DevFeed: [Rerouting the Stream: How Lyft Moved to the Apache Flink Operator]() Original publisher: [Read original article]() Author: Maheep Myneni Published: 2026-08-31T19:08:16Z Content type: article Language: en Sources: [Lyft Engineering - Medium]() Topics: [Apache Iceberg](), [Streaming](), [Kubernetes](), [Open Source](), [autoscaling](), [Tooling](), [Architecture & Design]() Tags: [apache](), [apache-flink](), [autoscaling](), [data-platforms](), [infrastructure](), [kubernetes](), [lyft](), [open-source](), [production](), [streaming](), [streaming-data-processing](), [tooling]() ### AI overview Lyft describes its migration from an internally developed Flink Kubernetes operator to the open-source Apache Flink Kubernetes Operator. The move addressed maintenance burden, technical debt, feature gaps, outdated dependencies, and growing real-time streaming demands, while providing capabilities such as autoscaling, memory tuning, safer upgrades, and automatic rollbacks. ### Source excerpt Written by Maheep Myneni, Arda Kuyumcu, and Prem Santosh Udaya Shankar at Lyft. Why We Migrated: Technical Debt Meets Modern Streaming Demands Over the past several quarters, Lyft's Streaming Compute team retired our internally developed Flink Kubernetes operator and moved our entire streaming fleet onto the open-source Apache Flink Kubernetes operator. This post is about why we made the switch, how we pulled it off incrementally without disrupting users, and the follow-on work it took to actually get the benefits we were after. Back in 2020, when we first architected the Lyft Flink Kubernetes Operator, it was exactly what we needed. At that time, the open-source community hadn't yet built a dedicated control plane, so we built our own to manage all streaming applications on Kubernetes. It worked well for our initial workloads, but as our real-time data needs increased and our engineers' scope of ownership grew in both breadth and complexity, the cracks started to show. First came the maintenance burden. Our operator had become a relic of Lyft's early Kubernetes days, kept alive by a growing pile of custom code. Every Flink version upgrade meant carefully picking through layers of accumulated technical debt and hoping nothing broke on the way through. Second came the feature gap. Streaming tooling kept moving, and our engineers kept asking for capabilities that had become table stakes elsewhere, such as autoscaling to right-size jobs, automatic rollbacks on failed deploys, and an end to hand-tuning CPU and memory. Each request left us with two options, neither of which was ideal. We could explain why we couldn't support it yet, or spend weeks rebuilding something the open-source community had already shipped. Third was the dependency problem. We were pinned to outdated libraries. That doesn't break anything today, but it almost always creates new issues down the line. Security patches lagged, modern Kubernetes features stayed out of reach, and every quarter we waite ## A Tale of Two Flink Autoscalers DevFeed: [A Tale of Two Flink Autoscalers]() Original publisher: [Read original article]() Author: Netflix Technology Blog Published: 2026-08-21T16:01:01Z Content type: article Language: en Sources: [Netflix](), [Netflix TechBlog - Medium]() Topics: [apache-flink](), [autoscaling](), [Netflix](), [stream-processing](), [Kafka](), [Open Source](), [telemetry](), [Network]() Tags: [apache-flink](), [autoscaling](), [aws](), [infrastructure](), [kafka](), [metrics](), [netflix](), [open-source](), [operational-excellence](), [scale](), [stream-processing](), [telemetry]() ### AI overview Netflix describes running two Apache Flink autoscalers: an older in-house system and a newer one from the Apache Flink community. The article explains why autoscaling is necessary for more than 30,000 Flink jobs across AWS regions, discusses workload variability and the cost of resizing stateful jobs, and introduces the metrics-driven design of the first autoscaler. ### Source excerpt Samuel Yeboah, Francesco Di Chiara and Mingliang Liu Today, Netflix runs two Flink autoscalers. That is exactly one more than we want. We built the first one in-house years ago, when there was no mature option suited to our platform. The second came from the Apache Flink community, and it can scale workloads our homegrown system was never designed for. We now run both in production and are steadily converging on the open-source one. Along the way we learned some hard lessons about metrics, cost, and the real price of maintaining infrastructure you could instead adopt, and we hope they are useful whether you run a handful of Flink jobs or tens of thousands. Why autoscaling is not optional at our scale Netflix has run stream processing on Apache Flink since 2017. As of 2026 we operate more than 30,000 Flink jobs across multiple AWS regions. Most are not deployed by hand; they are generated by our managed platform Data Mesh, so the majority of users never touch a Flink job directly. A smaller but growing set are custom jobs, built and operated by teams across the company for use cases like personalization, Ads, and Live events. They range from single-operator jobs that shuttle records between Kafka topics to stateful pipelines with branches, joins, and terabytes of state, and their load swings with daily cycles, launches, and regional failovers. Provisioning every one of those jobs for its peak is wasteful; provisioning for the average causes lag during surges. And in our platform a scaling action is not free: by default it means taking a savepoint, stopping the job gracefully, and restarting it at the new size, which for a large stateful job can take minutes. That leaves a genuinely hard question: how do you give each job the resources it needs, when it needs them, without a human in the loop and without breaking anything? The first autoscaler: watching from outside Our first answer, built around 2019, was an autoscaler shaped like a stream-processing job. It ran on M ## New in Confluent Intelligence and AI Tools: Making Agents Native to the Stream, Expanded Model Support, New Agent Skills, and Copilot DevFeed: [New in Confluent Intelligence and AI Tools: Making Agents Native to the Stream, Expanded Model Support, New Agent Skills, and Copilot]() Original publisher: [Read original article]() Author: Confluent Staff Published: 2026-08-18T14:00:10Z Content type: article Language: en Sources: [Confluent: Data in motion]() Topics: [Artificial Intelligence](), [apache-flink](), [Kafka](), [streaming-data-processing](), [real-time](), [MCP Server](), [Agent Skills](), [MCP](), [Inference](), [Google](), [ibm]() Tags: [ai](), [ai-ready-data](), [ai-tools](), [apache-flink](), [confluent](), [confluent-cloud](), [google](), [ibm](), [kafka](), [mcp](), [mcp-server](), [real-time](), [skills](), [streaming-data-processing](), [support](), [tools]() ### AI overview Confluent announces updates to Confluent Intelligence and related AI tools for building production AI systems on Apache Kafka and Apache Flink. The release adds expanded time-series model support, a generally available Real-Time Context Engine, updates to the fully managed MCP Server, Agent Skills for AI coding assistants, and Confluent Copilot. The features are intended to provide agents and applications with fresh business context and governed access to live data and infrastructure. ### Source excerpt Explore new AI features and AI tools: support for IBM Granite Time Series models and TimesFM models (EA), enhanced Real-Time Context Engine experience, new Agent Skills, Confluent Copilot ## Confluent Cloud for Apache Flink: Engine for Mission-Critical, Real-Time Operational Systems and dbt/SQL-Native Home for Data Science and AI DevFeed: [Confluent Cloud for Apache Flink: Engine for Mission-Critical, Real-Time Operational Systems and dbt/SQL-Native Home for Data Science and AI]() Original publisher: [Read original article]() Author: Yashwanth Dasari Published: 2026-08-18T02:20:00Z Content type: release Language: en Sources: [Confluent: Data in motion]() Topics: [Confluent Cloud](), [apache-flink](), [stream-processing](), [Kafka](), [AI, ML & Data Engineering](), [Cloud](), [SDKs]() Tags: [ai](), [announcement](), [apache-flink](), [api](), [batch](), [confluent-cloud](), [data-science](), [developers](), [kafka](), [real-time]() ### AI overview Confluent announces an evolution of Confluent Cloud for Apache Flink that unifies mission-critical real-time operations with analytics and AI workflows. The release adds a serverless, co-designed Kafka and Flink platform, including the generally available Flink Table API in Java for code-first development. ### Source excerpt Flink now acts as a robust engine for developers through Table API, UDFs, and PTFs while offering a SQL-native, dbt-integrated platform for data science and AI teams. ## How and Why Netflix Built a Real-Time Distributed Graph: Part 3 -- Querying the graph with gRPC... DevFeed: [How and Why Netflix Built a Real-Time Distributed Graph: Part 3 -- Querying the graph with gRPC...]() Original publisher: [Read original article]() Author: Netflix Technology Blog Published: 2026-08-07T16:01:02Z Content type: article Language: en Sources: [Netflix](), [Netflix TechBlog - Medium]() Topics: [Graphs](), [gRPC](), [Netflix](), [real-time](), [Latency](), [API](), [Streaming](), [personalization](), [Security](), [apache-flink]() Tags: [apache](), [apache-flink](), [api](), [data](), [data-processing](), [events](), [latency](), [netflix](), [personalization](), [real-time](), [security](), [streaming]() ### AI overview Netflix's third-part blog post describes the serving layer for its Real-Time Distributed Graph and explains how gRPC enables efficient graph querying. It focuses on serving diverse workloads, including security lookups and personalization traces, with sub-100ms responses across a billion-edge graph. ### Source excerpt How and Why Netflix Built a Real-Time Distributed Graph: Part 3 -- Querying the graph with gRPC execution API Authors: Nilesh Mishra and Ajit Koti This is the third entry of a multi-part blog series describing how we built a Real-Time Distributed Graph (RDG). In Part 1, we discussed the motivation for creating the RDG and the architecture of the data processing pipeline that populates it. In Part 2, we discussed how we designed the storage layer to handle billions of nodes and edges while maintaining single-digit-millisecond latency. In Part 3, we will explore how we designed a fast, flexible serving layer to efficiently query the graph. Introduction In Part 1 of this series, we described why Netflix needed a Real-Time Distributed Graph (RDG) and how we used Apache Flink to build an ingestion and processing pipeline that turns streaming events into graph primitives. In Part 2, we explored how we designed a storage layer capable of handling billions of nodes and edges while still delivering single-digit-millisecond latency. In this post, we focus on the next challenge: querying the graph efficiently to power real-time insights for our internal partners. All of the work on ingestion and storage only matters if we can actually ask complex questions and get answers back quickly. As we optimized for lower latency, we found that the serving layer posed its own set of challenges, distinct from those of ingestion and storage. How do we turn a constantly evolving, billion-edge graph into sub-100ms responses across a wide variety of workloads? This is the problem we tackle in this post. The Real World Needs As we integrated the RDG into Netflix's ecosystem, we realized that "querying the graph" is not a one-size-fits-all operation. We needed to handle a wide range of access patterns: from high-volume security lookups to deep, exploratory personalization traces. Let's revisit our example from Part 1 and expand on it slightly. In the earlier posts, we focused on accounts, device ## Announcing Confluent Platform 8.3: Powerful Apache Flink® SQL operations, Easier KRaft Migrations, Expanded Monitoring and more. DevFeed: [Announcing Confluent Platform 8.3: Powerful Apache Flink® SQL operations, Easier KRaft Migrations, Expanded Monitoring and more.]() Original publisher: [Read original article]() Author: Premika Srinivasan Published: 2026-07-29T15:00:10Z Content type: release Language: en Sources: [Confluent: Data in motion]() Topics: [apache-flink](), [Kafka](), [Monitoring](), [data-governance](), [Model Context Protocol](), [Claude Code](), [codex]() Tags: [apache-flink](), [apache-kafka](), [claude-code](), [codex](), [confluent](), [confluent-platform](), [data-governance](), [mcp](), [mcp-server](), [monitoring]() ### AI overview Confluent Platform 8.3.0, built on Apache Kafka 4.3.0, adds simplified Apache Flink SQL operations, expanded monitoring through Unified Stream Manager, easier KRaft migration, and structured governance for data in motion. The release also introduces an MCP server for Confluent Platform for Apache Flink, allowing AI agents such as Claude Code and Codex to inspect and manage Flink resources through existing security controls. ### Source excerpt Announcing Confluent Platform 8.3: Powerful Apache Flink® SQL operations, Easier KRaft Migrations, Expanded Monitoring and more ## The CI/CD moment for data analytics DevFeed: [The CI/CD moment for data analytics]() Original publisher: [Read original article]() Author: Gaurav Nanda Published: 2026-07-23T05:40:01Z Content type: article Language: en Sources: [Platform Engineering Blog]() Topics: [data analytics](), [DataOps](), [real-time](), [data-engineering](), [CI/CD](), [Kafka](), [apache-flink]() Tags: [ai](), [analytics](), [apache-flink](), [architectures](), [bridging](), [cost](), [data-analytics](), [data-engineering](), [engineering](), [event](), [fraud](), [google](), [insights](), [kafka](), [latency](), [real-time](), [scale](), [streaming](), [systems](), [tools]() ### AI overview The article argues that data analytics is approaching a CI/CD-like transition toward continuous analytics. It describes how multi-step pipelines introduce latency, operational risk, and maintenance burden, and explains why real-time insight is becoming a baseline platform capability for applications such as personalization, fraud detection, reliability, and AI-driven features. ### Source excerpt How converging OLTP and OLAP architectures are driving 'Continuous Analytics', the CI/CD moment for data to deliver real-time, unified insights and operational simplicity. ## Announcing Apache Iceberg 1.11.0 DevFeed: [Announcing Apache Iceberg 1.11.0]() Original publisher: [Read original article]() Author: Google Open Source (noreply@blogger.com) Published: 2026-05-27T18:30:00Z Content type: release Language: en Sources: [Google Open Source Blog]() Topics: [Apache Iceberg](), [Apache Spark](), [apache-flink](), [schema-evolution](), [Streaming](), [Encryption](), [Cryptography]() Tags: [apache-flink](), [apache-iceberg](), [apache-spark](), [encryption](), [kms](), [lakehouse](), [releases](), [schema-evolution](), [streaming]() ### AI overview Apache Iceberg 1.11.0 adds support for Apache Spark 4.1 and Apache Flink 2.1, makes both default build targets, and introduces changes including server-side REST catalog scan planning, partition statistics APIs, built-in envelope encryption with Google KMS support, and Google Storage Analytics integration. ### Source excerpt by Alex Stephen & Talat Uyarer, Lakehouse Apache Iceberg project has just launched version 1.11.0! A lot has happened since the last version. Iceberg 1.11.0 adds support for Apache Spark 4.1 and Apache Flink 2.1, the latest releases of the two engines and makes both the default build targets The rest are more structural. The REST catalog learns to plan scans server-side, shifting metadata work off the query engine. A new partition statistics scan API gives optimizers a clean, supported way to read a table's shape. Built-in table encryption arrives with envelope encryption and Google KMS support. And Google Storage Analytics library integration makes your Iceberg workloads faster than before. Let's take a look at some of the biggest changes. Spark & Flink Updates As Spark and Flink are moving forward, the 1.11.0 release is pushing forward for new version support in both. Spark 4.1 & DSv2 Migration: Spark 4.1 unlocks is MERGE INTO with automatic schema evolution: Spark's newer MERGE syntax accepts a WITH SCHEMA EVOLUTION clause, so a MERGE whose source carries columns the target table lacks can add those columns to the table within the same statement, with no separate ALTER TABLE round trip. Beyond the version bump, the 1.11 Spark connector also modernizes against Spark's newer DataSource V2 APIs and adds an asynchronous micro-batch planner that speeds up Structured Streaming. Flink Ecosystem Updates: Initial work for Flink 2.1 support has landed in the core repository, continuing Iceberg's promise of providing first-class, low-latency streaming sink capabilities. The centerpiece of the Flink work is the DynamicIcebergSink, an experimental sink that breaks the old one-sink-per-table model: a single sink routes each record to a table chosen at runtime, creating tables on demand and evolving their schemas and partition specs on the fly as the input changes including dropping columns once you opt in with dropUnusedColumns. In addition to DynamicIcebergSInk work Flink sta ## January 2026 newsletter DevFeed: [January 2026 newsletter]() Original publisher: [Read original article]() Author: Mark Needham Published: 2026-01-15T09:50:24Z Content type: news Language: en Sources: [ClickHouse Blog]() Topics: [data](), [migration](), [Kafka](), [Advent of Code]() Tags: [apache-flink](), [clickhouse](), [community](), [data](), [integration](), [kafka](), [migration](), [performance](), [real-time](), [release](), [sql](), [storage]() ### AI overview January 2026 ClickHouse newsletter covering query-performance improvements, zero-copy chDB integration with Pandas DataFrames, a migration from TimescaleDB, and a ClickHouse SQL Advent of Code deep dive. ### Source excerpt Welcome to the January 2026 ClickHouse newsletter, which will round up what's happened in real-time data warehouses over the last month. ## How Changan Ford cut costs by 40% and powered precision marketing with ClickHouse Enterprise Edition DevFeed: [How Changan Ford cut costs by 40% and powered precision marketing with ClickHouse Enterprise Edition]() Original publisher: [Read original article]() Author: ClickHouse Published: 2025-09-22T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [data](), [dashboards]() Tags: [ads](), [analytics](), [apache-flink](), [architecture](), [clickhouse](), [cloud](), [customer](), [dashboards](), [data](), [enterprise](), [errors](), [open-source](), [performance](), [platform](), [real-time](), [serverless](), [spark](), [storage]() ### AI overview Changan Ford migrated its customer data platform from ClickHouse Community Edition to ClickHouse Enterprise Edition on Alibaba Cloud. The company reports a 40% cost reduction, lower operational workload, and improved stability for high-demand marketing campaigns. ### Source excerpt Changan Ford migrated their Customer Data Platform to ClickHouse Enterprise Edition, cut costs by 40%, eased the operational workload, and gave the team stability and confidence even during their most demanding campaigns. ## From siloed DataOps, MLOps, and LLMOps to a unified data-intelligence platform DevFeed: [From siloed DataOps, MLOps, and LLMOps to a unified data-intelligence platform]() Original publisher: [Read original article]() Author: Rajit Saha Published: 2025-08-04T18:03:19Z Content type: opinion Language: en Sources: [Udemy Engineering]() Topics: [DataOps](), [MLOps](), [Generative AI](), [Apache Spark](), [Amazon Bedrock](), [Amazon S3](), [Amazon Redshift](), [Amazon SageMaker](), [apache-flink]() Tags: [ai](), [aiops](), [amazon-s3](), [analytics](), [apache-flink](), [apache-spark](), [bedrock](), [dataops](), [large-language-models](), [llmops](), [mlops]() ### AI overview The article describes how DataOps, MLOps, and AI/LLM Ops commonly rely on separate systems and teams for data processing, model deployment, inference, evaluation, orchestration, governance, and monitoring. It then introduces Databricks' Data Intelligence Platform as a unified environment intended to bring these domains together. ### Source excerpt Introduction In modern data-driven businesses, the pace of innovation in analytics and artificial intelligence has outstripped the capacity of many teams. Three distinct disciplines emerged to handle this expansion: Data platform (DataOps) teams built data lakes on cloud storage such as Amazon S3, processed them with Apache Spark and Hive on EMR, ingested streaming data with Spark Structured Streaming or Apache Flink, and loaded tabular copies into MPP warehouses like Redshift for interactive SQL and BI. Cataloguing and governance were offloaded to external tools such as DataHub, and fine-grained access controls required third-party services like Privacera. This architecture worked, but it required separate workflows for batch and streaming, extra systems for lineage and governance, and a mosaic of operational teams. MLOps teams provided an additional layer. Data scientists used notebook environments (for example, Amazon SageMaker) to preprocess data, train, and evaluate models. Deploying models meant writing integration code to move features into a serving layer, to register models in disparate registries and to build custom APIs for inference. Feature stores and model registries were bought from additional vendors. Updates and monitoring were often manual processes. AI/LLM Ops teams are a new addition because generative AI requires specialized components: LLM gateways (e.g., Amazon Bedrock) to proxy access to foundation models; evaluation tooling to compare large language models; orchestration frameworks for agents; vector databases for retrieval augmented generation; and of course another layer of security, access management and cost control. These tools seldom integrate seamlessly with existing data and ML pipelines. This fragmented state makes it difficult to react quickly when product requirements change. Each new capability requires another system, another integration, and another team. Meanwhile, budgets tighten and go-to-market timelines shrink. The questio ## Building AI agents as streaming SQL queries with Apache Flink DevFeed: [Building AI agents as streaming SQL queries with Apache Flink]() Original publisher: [Read original article]() Published: 2025-06-18T13:30:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [AI Agent](), [SQL](), [Streaming](), [apache-flink](), [event driven](), [Scalability](), [Large Language Model](), [Developer experience]() Tags: [ai](), [ai-agent](), [apache-flink](), [developer-experience](), [event-driven](), [llms](), [scalability](), [sql](), [streaming]() ### AI overview This article explores whether AI agents can be built as streaming SQL queries. It presents Apache Flink and stream-processing concepts as potential foundations for event-driven, scalable, consistent, low-latency agentic systems, while noting that many deployed agents are defined AI-assisted workflows. ### Source excerpt Table of Contents Agents Need to Interact With LLMs Agents Should Be Event-Driven Agents Need Context Agents Require Memory When SQL Is Not Enough Parting Thoughts AI Agents have improved in leaps and bounds in recent times, moving beyond simple chatbots to sophisticated, autonomous systems. This post explores a novel approach to building agentic systems: using the power of streaming SQL queries. Discover how platforms like Apache Flink can transform the development of AI Agents, offering benefits in consistency, scalability, and developer experience. ## Backfilling Postgres TOAST Columns in Debezium Data Change Events DevFeed: [Backfilling Postgres TOAST Columns in Debezium Data Change Events]() Original publisher: [Read original article]() Published: 2025-05-26T14:40:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Databases](), [Replication](), [apache-flink](), [stream-processing](), [SQL](), [API]() Tags: [apache-flink](), [data](), [debezium](), [debezium-connector](), [flink](), [postgres](), [replication](), [sql](), [stateful](), [stream-processing]() ### AI overview This article explains how Postgres logical replication handles TOAST columns in Debezium change events. It discusses Debezium's reselect postprocessor and explores Apache Flink solutions using stateful stream processing, Flink SQL, and process table functions. ### Source excerpt Table of Contents Debezium Reselect Postprocessor Flink DataStream API Flink SQL With OVER Aggregation Flink Process Table Functions Summary and Discussion Postgres logical replication, while powerful for capturing real-time data changes, presents challenges with TOAST columns, whose values can be absent from data change events in specific situations. This post discusses how Debezium addresses this through its built-in reselect post processor, then explores more robust solutions leveraging Apache Flink's capabilities for stateful stream processing, including Flink SQL and the brand-new process table functions (PTFs) in Flink 2.1. ## What are Apache Iceberg tables? Benefits and challenges | Redpanda DevFeed: [What are Apache Iceberg tables? Benefits and challenges | Redpanda]() Original publisher: [Read original article]() Author: Redpanda Published: 2025-05-21T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Apache Iceberg](), [datasets](), [data-processing](), [Scalability](), [Architecture & Design](), [Streaming](), [Netflix]() Tags: [amazon-redshift](), [analytics](), [apache-flink](), [apache-iceberg](), [apache-iceberg-acid-compliance](), [apache-iceberg-architecture](), [apache-iceberg-challenges](), [apache-iceberg-metadata-management](), [apache-iceberg-table-format](), [apache-iceberg-tables](), [apache-iceberg-vs-data-lakes](), [architecture](), [benefits-of-apache-iceberg](), [data](), [data-lakes-and-apache-iceberg](), [datasets](), [fundamentals](), [managing-large-datasets-with-apache-iceberg](), [netflix](), [object-storage](), [open-source](), [pipelines](), [real-time-analytics-with-apache-iceberg](), [scalability](), [scalability-of-apache-iceberg-tables](), [streaming]() ### AI overview This article explains how Apache Iceberg tables add database-like structure to data lakes for large analytic datasets. It covers schemas, partitioning, metadata catalogs, version history, schema changes, data rewrites, queryability, consistency, scalability, and interoperability across batch and streaming pipelines and analytics engines. ### Source excerpt Apache Iceberg tables introduce a reliable framework for querying large datasets in data lakes. Explore their use cases, benefits, and more. ## Get Running with Apache Flink on Kubernetes, part 2 of 2 DevFeed: [Get Running with Apache Flink on Kubernetes, part 2 of 2]() Original publisher: [Read original article]() Published: 2025-01-28T00:00:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [apache-flink](), [Kubernetes](), [Availability](), [observability](), [Deployment](), [container](), [MinIO](), [Amazon S3](), [rocksdb](), [GitHub]() Tags: [apache-flink](), [availability](), [container-images](), [deployment](), [flink](), [github](), [kubernetes](), [minio](), [observability](), [rocksdb](), [s3](), [ui]() ### AI overview Part two of a tutorial series on running Apache Flink jobs on Kubernetes. It explains checkpointing, job manager high availability, savepoints, external object storage, incremental RocksDB checkpoints, observability, and related job-management practices. ### Source excerpt Table of Contents Fault Tolerance and High Availability Manually Triggering Savepoints Observability Bonus: Managing Flink Jobs With the Heimdall UI Summary and Discussion This post originally appeared on the Decodable blog. All rights reserved. Welcome back to this two-part blog post series about running Apache Flink on Kubernetes, using the Flink Kubernetes operator. In part one, we discussed installation and setup of the operator, different deployment types, how to deploy Flink jobs using custom Kubernetes resources, and how to create container images for your own Flink jobs. In this part, we'll focus on aspects such as fault tolerance and high availability of your Flink jobs running on Kubernetes, savepoint management, observability, and more. You can find the complete source code for all the examples shown in this series in the Decodable examples repository on GitHub: on GitHub. ## Get Running with Apache Flink on Kubernetes, part 1 of 2 DevFeed: [Get Running with Apache Flink on Kubernetes, part 1 of 2]() Original publisher: [Read original article]() Published: 2025-01-21T00:00:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [apache-flink](), [Kubernetes](), [Deployment](), [Provisioning](), [container images]() Tags: [apache-flink](), [container-images](), [deployment](), [installation](), [kubernetes](), [operator](), [provisioning]() ### AI overview A two-part tutorial on deploying Apache Flink jobs on Kubernetes using the upstream Flink Kubernetes Operator. Part one covers installation, setup, custom-resource deployment, and building container images; part two addresses fault tolerance, high availability, savepoints, observability, and UI access. ### Source excerpt Table of Contents Installation and Setup Deployment Types Deploying Your First Flink Job on Kubernetes Building Custom Job Images This post originally appeared on the Decodable blog. All rights reserved. Kubernetes is a widely used deployment platform for Apache Flink. While Flink has had native support for Kubernetes for quite a while, it is in particular the operator pattern which makes deploying Flink jobs onto Kubernetes clusters a compelling option: you define jobs in a declarative resource, and a control loop running in a component called a Kubernetes operator takes care of provisioning and maintaining (e.g. scaling, updating) all the required resources. Automation is the keyword here, significantly reducing the manual effort required for running Flink jobs in production. ## CDC Use Cases: 7 Ways to Put CDC to Work DevFeed: [CDC Use Cases: 7 Ways to Put CDC to Work]() Original publisher: [Read original article]() Published: 2023-11-02T00:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [data-engineering](), [streaming-data-processing](), [Kafka](), [apache-flink](), [MySQL](), [Microservice](), [migration]() Tags: [apache-flink](), [data](), [data-engineering](), [debezium](), [engineering](), [kafka](), [migration](), [mysql](), [real-time]() ### AI overview This article explains change data capture (CDC), focusing on log-based CDC and its low-latency, resource-efficient handling of database changes. It introduces Debezium and describes how CDC works with data streaming and stream-processing tools such as Apache Kafka and Apache Flink, before outlining seven common use cases. ### Source excerpt Table of Contents What is CDC? CDC Tools Analytics Data Platforms Application Caches Full-Text Search Audit Logs Continuous Queries Microservices Data Exchange Monolith-to-Microservices Migration Summary This post originally appeared on the Decodable blog. All rights reserved. Change Data Capture (CDC) is a powerful tool in data engineering and has seen a tremendous uptake in organizations of all kinds over the last few years. This is because it enables the tight integration of transactional databases into many other systems in your business at a very low latency. ## Building Apache Flink with Maven: Resolving HTTP Repository Dependency Issues DevFeed: [Building Apache Flink with Maven: Resolving HTTP Repository Dependency Issues]() Original publisher: [Read original article]() Published: 2022-12-18T12:45:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [Maven](), [apache-flink](), [Security]() Tags: [apache](), [apache-flink](), [build-tool](), [dependencies](), [security](), [stream-processing]() ### AI overview This article describes issues encountered while building Apache Flink from source with Maven 3.8.6. It explains how dependency-defined HTTP repositories can be blocked by Maven 3.8.1 and later, and discusses tracing the problematic dependency and overriding the repository URL in settings.xml. ### Source excerpt As part of my new job at Decodable, I am also planning to contribute to the Apache Flink project (as Decodable's fully-managed stream processing platform is based on Flink). Right now, I am in the process of familiarizing myself with the Flink code base, and as such I am of course building the project from source, too. ## 3 (More) Tips for Optimizing Apache Flink Applications DevFeed: [3 (More) Tips for Optimizing Apache Flink Applications]() Original publisher: [Read original article]() Author: Kevin Lam Published: 2022-12-13T14:00:00Z Content type: tutorial Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [apache-flink](), [data-processing](), [Streaming](), [data]() Tags: [apache-flink](), [backpressure](), [best-practices](), [parallelism](), [performance](), [shopify](), [stateful]() ### AI overview Shopify presents additional best practices for optimizing large, stateful Apache Flink applications. The supplied text covers selecting appropriate parallelism to scale execution and improve resource utilization, and avoiding bottlenecks in data sinks that can reduce throughput and cause backpressure. ### Source excerpt Earlier this year, we shared our tips for optimizing large stateful Apache Flink applications. Below we'll walk you through 3 more best practices. ## Investigating WAL Growth from an Inactive Postgres Replication Slot on Amazon RDS DevFeed: [Investigating WAL Growth from an Inactive Postgres Replication Slot on Amazon RDS]() Original publisher: [Read original article]() Published: 2022-11-30T13:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Replication](), [Amazon RDS](), [apache-flink](), [Database]() Tags: [amazon](), [amazon-rds](), [apache-flink](), [database](), [debezium](), [disk-space](), [postgres](), [replication](), [storage]() ### AI overview This article investigates how an inactive Postgres replication slot on Amazon RDS can retain WAL segments and consume disk space even when the database appears idle. It reproduces the behavior and compares replication-slot LSN differences with RDS storage and transaction-log metrics. ### Source excerpt Table of Contents The Observation The Solution Take Away While working on a demo for processing change events from Postgres with Apache Flink, I noticed an interesting phenomenon: A Postgres database which I had set up for that demo on Amazon RDS, ran out of disk space. The machine had a disk size of 200 GiB which was fully used up in the course of less than two weeks. Now a common cause for this kind of issue are replication slots which are not advanced: in that case, Postgres will hold on to all WAL segments after the latest log sequence number (LSN) which was confirmed for that slot. Indeed I had set up a replication slot (via the Decodable CDC source connector for Postgres, which is based on Debezium). I then had stopped that connector, causing the slot to become inactive. The problem was though that I was really sure that there was no traffic in that database whatsoever! What could cause a WAL growth of ~18 GB/day then? ## Reducing BigQuery Costs: How We Fixed A $1 Million Query DevFeed: [Reducing BigQuery Costs: How We Fixed A $1 Million Query]() Original publisher: [Read original article]() Author: Calvin Zhou Published: 2022-11-03T18:14:51Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [BigQuery](), [Streaming](), [apache-flink](), [Kafka](), [parquet](), [Google Cloud Platform (GCP)](), [Shopify](), [rocksdb]() Tags: [apache-flink](), [bigquery](), [cloud-storage](), [cost](), [kafka](), [parquet](), [pipelines](), [rocksdb](), [shopify](), [sql]() ### AI overview The article describes how a Shopify team discovered that a BigQuery query could cost nearly $1 million USD per month while building a marketing data pipeline. The team used Apache Flink with RocksDB and Apache Kafka, then evaluated BigQuery as an external data warehouse capable of querying large datasets and exporting results to Google Cloud Storage. The supplied text ends as the team encounters the cost problem, before detailing the eventual fix. ### Source excerpt We discovered a query that nearly cost us $1 million USD a month in BigQuery. Below, we'll share our tips for lowering costs in BigQuery. ## 7 Tips For Optimizing Apache Flink Applications DevFeed: [7 Tips For Optimizing Apache Flink Applications]() Original publisher: [Read original article]() Author: Yaroslav Tkachenko Published: 2022-03-08T21:20:39Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [apache-flink](), [Shopify](), [Kubernetes](), [rocksdb](), [Streaming](), [Tooling](), [cpu]() Tags: [apache-flink](), [cpu](), [data](), [debugging](), [jvm](), [kubernetes](), [performance](), [profiling](), [rocksdb](), [shopify](), [streaming]() ### AI overview This Shopify Engineering article presents lessons for optimizing large, stateful Apache Flink applications. It discusses profiling and debugging tools, performance tuning, resiliency, high availability, Kubernetes deployment, RocksDB state management, and checkpoint and savepoint storage in Google Cloud Storage. ### Source excerpt We've spent a lot of time on performance tuning our Apache Flink application. We'll walk you through key lessons for optimizing large stateful Apache Flink applications. [Next page]()