# flink Published articles for flink. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Lyft Moves Streaming Fleet to Apache Flink Kubernetes Operator DevFeed: [Lyft Moves Streaming Fleet to Apache Flink Kubernetes Operator]() Original publisher: [Read original article]() Author: Mark Silvester Published: 2026-09-16T11:00:00Z Content type: news Language: en Sources: [InfoQ]() Topics: [flink](), [apache-flink](), [Kubernetes](), [autoscaling](), [Deployment](), [legacy](), [Testing]() Tags: [apache-flink](), [architecture-design](), [autoscaling](), [deployment](), [development](), [devops](), [flink](), [kubernetes](), [kubernetes-operator](), [legacy](), [lyft-flink-k8s-operator](), [news](), [testing](), [upgrade]() ### AI overview Lyft moved hundreds of production Apache Flink jobs from its in-house Kubernetes operator to the Apache Flink Kubernetes Operator. The change enabled last-state upgrades, in-place autoscaling, and resource autotuning, while Lyft adapted legacy deployment specifications through its deploy API and contributed a fix for a configuration-renaming bug. ### Source excerpt Lyft has moved hundreds of production Flink jobs from a 2020 in-house Kubernetes operator to the Apache Flink Kubernetes Operator, unlocking last-state upgrades, in-place autoscaling and resource autotuning across the fleet. By Mark Silvester ## Why Flink may be unnecessarily complex for most streaming data processing users DevFeed: [Why Flink may be unnecessarily complex for most streaming data processing users]() Original publisher: [Read original article]() Author: Javi Santana Published: 2025-10-21T00:00:00Z Content type: opinion Language: en Sources: [Tinybird]() Topics: [streaming-data-processing](), [Streaming]() Tags: [flink](), [scalable-analytics-architecture](), [streaming](), [streaming-data-processing]() ### AI overview The article argues that Flink's complexity may make it unnecessary for most people who need streaming data processing. ### Source excerpt Flink might sound like the holy grail of streaming data processing, but for 95% of us, it's just a complex headache we don't need. ## Backfilling Postgres TOAST Columns in Debezium Data Change Events DevFeed: [Backfilling Postgres TOAST Columns in Debezium Data Change Events]() Original publisher: [Read original article]() Published: 2025-05-26T14:40:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [Databases](), [Replication](), [apache-flink](), [stream-processing](), [SQL](), [API]() Tags: [apache-flink](), [data](), [debezium](), [debezium-connector](), [flink](), [postgres](), [replication](), [sql](), [stateful](), [stream-processing]() ### AI overview This article explains how Postgres logical replication handles TOAST columns in Debezium change events. It discusses Debezium's reselect postprocessor and explores Apache Flink solutions using stateful stream processing, Flink SQL, and process table functions. ### Source excerpt Table of Contents Debezium Reselect Postprocessor Flink DataStream API Flink SQL With OVER Aggregation Flink Process Table Functions Summary and Discussion Postgres logical replication, while powerful for capturing real-time data changes, presents challenges with TOAST columns, whose values can be absent from data change events in specific situations. This post discusses how Debezium addresses this through its built-in reselect post processor, then explores more robust solutions leveraging Apache Flink's capabilities for stateful stream processing, including Flink SQL and the brand-new process table functions (PTFs) in Flink 2.1. ## A Deep Dive Into Ingesting Debezium Events From Kafka With Flink SQL DevFeed: [A Deep Dive Into Ingesting Debezium Events From Kafka With Flink SQL]() Original publisher: [Read original article]() Published: 2025-04-16T09:25:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [apache-flink](), [Kafka](), [Apache-Kafka]() Tags: [changelog](), [connectors](), [data](), [debezium](), [flink](), [how-to](), [kafka](), [schema](), [streams]() ### AI overview This tutorial explains how to ingest Debezium change events from Kafka into Apache Flink using Flink SQL. It compares the Apache Kafka SQL Connector and Upsert Kafka SQL Connector, discusses JSON and Avro formats, and explains how connector and format choices determine append-only or changelog semantics. ### Source excerpt Table of Contents Flink SQL Connectors for Apache Kafka The Apache Kafka SQL Connector in Append-Only Mode The Apache Kafka SQL Connector As a Changelog Source The Upsert Kafka SQL Connector Summary Over the years, I've spoken quite a bit about the use cases for processing Debezium data change events with Apache Flink, such as metadata enrichment, building denormalized data views, and creating data contracts for your CDC streams. One detail I haven't covered in depth so far is how to actually ingest Debezium change events from a Kafka topic into Flink, in particular via Flink SQL. Several connectors and data formats exist for this, which can make things somewhat confusing at first. So let's dive into the different options and the considerations around them! ## Get Running with Apache Flink on Kubernetes, part 2 of 2 DevFeed: [Get Running with Apache Flink on Kubernetes, part 2 of 2]() Original publisher: [Read original article]() Published: 2025-01-28T00:00:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [apache-flink](), [Kubernetes](), [Availability](), [observability](), [Deployment](), [container](), [MinIO](), [Amazon S3](), [rocksdb](), [GitHub]() Tags: [apache-flink](), [availability](), [container-images](), [deployment](), [flink](), [github](), [kubernetes](), [minio](), [observability](), [rocksdb](), [s3](), [ui]() ### AI overview Part two of a tutorial series on running Apache Flink jobs on Kubernetes. It explains checkpointing, job manager high availability, savepoints, external object storage, incremental RocksDB checkpoints, observability, and related job-management practices. ### Source excerpt Table of Contents Fault Tolerance and High Availability Manually Triggering Savepoints Observability Bonus: Managing Flink Jobs With the Heimdall UI Summary and Discussion This post originally appeared on the Decodable blog. All rights reserved. Welcome back to this two-part blog post series about running Apache Flink on Kubernetes, using the Flink Kubernetes operator. In part one, we discussed installation and setup of the operator, different deployment types, how to deploy Flink jobs using custom Kubernetes resources, and how to create container images for your own Flink jobs. In this part, we'll focus on aspects such as fault tolerance and high availability of your Flink jobs running on Kubernetes, savepoint management, observability, and more. You can find the complete source code for all the examples shown in this series in the Decodable examples repository on GitHub: on GitHub. ## Getting Started With PyFlink on Kubernetes DevFeed: [Getting Started With PyFlink on Kubernetes]() Original publisher: [Read original article]() Published: 2023-12-07T00:00:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [apache-flink](), [Kubernetes](), [stream-processing](), [Kafka](), [Python](), [data-engineering]() Tags: [apache-kafka](), [flink](), [kafka](), [kubernetes](), [operator](), [programming](), [python](), [stream-processing]() ### AI overview A practical guide to running PyFlink on Kubernetes with Apache Flink 1.18. It covers installing the Flink Kubernetes Operator and Kafka with Strimzi, creating a PyFlink job that generates data and writes it to Kafka using Flink SQL, building a container image, and deploying the job to Kubernetes. ### Source excerpt Table of Contents What Is PyFlink and Why Should You Care? Prerequisites Installing the Flink Kubernetes Operator Installing Strimzi and Apache Kafka A Simple PyFlink Job Building a Container Image With Your PyFlink Job Deploying a PyFlink Job On Kubernetes This post originally appeared on the Decodable blog. All rights reserved. The other day, I wanted to get my feet wet with PyFlink. While there is a fair amount of related information out there, I couldn't find really up-to-date documentation on using current versions of PyFlink with Flink on Kubernetes. ## ApacheCon Europe 2019 -- Thoughts and Insights by Airflow Committers DevFeed: [ApacheCon Europe 2019 -- Thoughts and Insights by Airflow Committers]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2019-11-22T00:00:00Z Content type: opinion Language: en Sources: [Apache Airflow Blog]() Topics: [Open Source](), [airflow](), [Apache Spark](), [flink]() Tags: [airflow](), [apache](), [apache-spark](), [community](), [flink](), [foundation](), [open-source](), [organization](), [projects]() ### AI overview The article shares insights from Apache project committers after ApacheCon Europe 2019, focusing on the Apache Software Foundation's community-centered model for organizing and sustaining open-source projects such as Airflow, Spark, Flink, and Beam. ### Source excerpt Is it possible to create an organization that delivers tens of projects used by millions, nearly no one is paid for doing their job, and still, it has been fruitfully carrying on for more than 20 years? Apache Software Foundation proves it is possible. For the last two decades, ASF has been crafting a model called the Apache Way--a way of organizing and leading tech open source projects. Due to this approach, which is strongly based on the "community over code" motto, we can enjoy such awesome projects like Apache Spark, Flink, Beam, or Airflow (and many more). After this year's ApacheCon, Polidea's engineers talked with Committers of Apache projects, such as--Aizhamal Nurmamat kyzy, Felix Uellendall, and Fokko Driesprong--about insights to what makes the ASF such an amazing organization. You can read the insights after the ApacheCon 2019. ## One-box stream processing with CSP DevFeed: [One-box stream processing with CSP]() Original publisher: [Read original article]() Published: 2018-02-18T00:00:00Z Content type: tutorial Language: en Sources: [Adam Bard]() Topics: [streaming-data-processing](), [Clojure](), [stream-processing](), [Concurrent Programming](), [distributed-systems]() Tags: [clojure](), [concurrent](), [core-async](), [coroutine](), [distributed-computing](), [distributed-system](), [flink](), [spark](), [stream](), [stream-processing]() ### AI overview This article presents a small-scale stream-processing architecture in Clojure using core.async and component libraries. It describes modular components connected by asynchronous queues and explains how stream-processing design principles can support reusable, loosely coupled systems without requiring a distributed stream processor. ### Source excerpt If you're like me (that is, employed by an ad tech company), stream processing is usually associated with frameworks like Storm, Flink, Spark Streaming, and other such solutions. However, a lot of real-life software can be described as stream processing - data comes in one end, is transformed or aggregated, and goes somewhere else. Many of these workloads don't justify the overhead of a stream processor, but that doesn't mean they can't benefit from some of the lessons of stream processing systems. ## Running Apache Flink on Amazon Elastic Mapreduce DevFeed: [Running Apache Flink on Amazon Elastic Mapreduce]() Original publisher: [Read original article]() Published: 2016-01-06T08:15:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [apache-flink](), [Amazon Web Services](), [Hadoop](), [Scalding](), [data-processing](), [Amazon S3](), [Apache Spark]() Tags: [apache-flink](), [aws](), [data-processing](), [ec2](), [emr](), [flink](), [hadoop](), [hdfs](), [linux](), [realtime](), [s3](), [scala](), [scalding](), [sclading](), [spark](), [streaming]() ### AI overview A tutorial on running Apache Flink jobs on Amazon EMR, including adapting Scalding jobs and launching and accessing an EMR cluster. It also discusses EMRFS support for reading data from Amazon S3. ### Source excerpt I love really Amazon EMR. Over the years it's grown from being "Hadoop on-demand" to a full-fledged cluster management system for running OSS big-data apps (Hadoop MR of course, but also Spark, Hue, Hive, Pig, Oozie and more). While Hadoop out of the box supports reading from S3, EMR has a proprietary implementation called EMRFS that has some nice features. For those reasons, it's really the best Hadoop cluster to use if you're storing your data in S3. Lately I've been experimenting a lot with Apache Flink to replace MR as the excution fabric. At work, we have many, many jobs written in Scalding. Flink can execute Scalding jobs with some very simple modifications which was a great way to move our jobs from MR to a more memory-centric data processing model. However, we really wanted to run our jobs on EMR using Flink. Flink is not an option for EMR (yet) but can we still get our jobs to run? Let's see! Start an EMR cluster The first thing we need is an EMR cluster. You can launch a small test cluster for very cheap. Once it's running, let's ssh onto it and see what's going on. ihummel at mm-mac-3270 in ~ $ ssh hadoop@ec2-54-226-25-85.compute-1.amazonaws.com Last login: Wed Jan 6 16:33:01 2016 from 172.85.47.138 __| __|_ ) _| ( / Amazon Linux AMI ___|\___|___| https://aws.amazon.com/amazon-linux-ami/2015.09-release-notes/ 23 package(s) needed for security, out of 49 available Run "sudo yum update" to apply all updates. EEEEEEEEEEEEEEEEEEEE MMMMMMMM MMMMMMMM RRRRRRRRRRRRRRR E::::::::::::::::::E M:::::::M M:::::::M R::::::::::::::R EE:::::EEEEEEEEE:::E M::::::::M M::::::::M R:::::RRRRRR:::::R E::::E EEEEE M:::::::::M M:::::::::M RR::::R R::::R E::::E M::::::M:::M M:::M::::::M R:::R R::::R E:::::EEEEEEEEEE M:::::M M:::M M:::M M:::::M R:::RRRRRR:::::R E::::::::::::::E M:::::M M:::M:::M M:::::M R:::::::::::RR E:::::EEEEEEEEEE M:::::M M:::::M M:::::M R:::RRRRRR::::R E::::E M:::::M M:::M M:::::M R:::R R::::R E::::E EEEEE M:::::M MMM M:::::M R:::R R::::R EE:::::EEEEEEEE::::E ## Running Scalding jobs on Apache Flink DevFeed: [Running Scalding jobs on Apache Flink]() Original publisher: [Read original article]() Published: 2015-12-20T08:15:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [Scalding](), [apache-flink](), [Scala](), [Streaming](), [Hadoop]() Tags: [apache-flink](), [flink](), [hadoop](), [hdfs](), [realtime](), [scala](), [scalding](), [sclading](), [streaming]() ### AI overview This tutorial explains how to run Scalding jobs on Apache Flink. It outlines the required compatibility work, including Cascading 3 support, a Scalding version compiled against Cascading 3, selection of hadoop2-flink as the Cascading fabric, and a fix for a Twitter Chill bug. It also provides a sample Scala word-count job and build configuration changes. ### Source excerpt My previous post showed a very simple Scalding workflow. Apache Flink is a real time streaming framework that's very promising. It also supports running Cascading workflows with very little modification. Surely there must be some way to run a Scalding job on top of Flink? Turns out... YES! In a nutshell Here are the high-level things we need to solve for We need a Scalding job to test this out with cascading-flink requires Cascading 3 We need a new version of Scalding - Compiled against Cascading 3 - Allows hadoop2-flink to be selected as the Cascading "fabric" There is a bug in Twitter Chill We need Flink packaged with the Chill fix Scalding job Let's start with a very simple Scalding job. You can download https://github.com/themodernlife/simple-scalding-example for some inspiration. package net.themodernlife import com.twitter.scalding._ class WordCount(args: Args) extends Job(args) { def tokenize(text: String): Array[String] = { text.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]", "").split("\\s+") } val input = args("input") val output = args("output") TextLine(args("input")) .flatMap[String, String]('line -> 'word)(tokenize) .groupBy('word)(_.size) .write(Tsv(output)) } We'll be making some updates to our build.sbt. Here's what we're starting with organization := "net.themodernlife" name := "simple-scalding-example" scalaVersion := "2.11.7" scalacOptions ++= Seq("-encoding", "utf-8", "-deprecation", "-unchecked", "-feature") resolvers ++= Seq( "Concurrent Maven Repo" at "http://conjars.org/repo", "Twitter Maven Repo" at "http://maven.twttr.com" ) libraryDependencies ++= Seq( "com.twitter" %% "scalding-core" % "0.15.0", "org.apache.hadoop" % "hadoop-client" % "2.2.0" % "provided", "org.slf4j" % "slf4j-log4j12" % "1.7.13" % "provided" ) A new Scalding build We need to update Scalding Apply https://github.com/twitter/scalding/pull/1446 Apply/hack https://github.com/twitter/scalding/pull/1220 Hack Build.scala to ignore some submodules According to http://www.cascading.org/2