# A Tale of Two Flink Autoscalers

DevFeed: [A Tale of Two Flink Autoscalers](<https://devfeed.tech/articles/a-tale-of-two-flink-autoscalers-134.md>)

Original publisher: [Read original article](<https://netflixtechblog.com/a-tale-of-two-flink-autoscalers-e9f6a1b1492b?source=rss----2615bd06b42e---4>)

Author: Netflix Technology Blog

Published: 2026-08-21T16:01:01Z

Content type: article

Language: en

Sources: [Netflix](<https://devfeed.tech/sources/netflix.md>), [Netflix TechBlog - Medium](<https://devfeed.tech/sources/netflix-techblog-medium.md>)

Topics: [apache-flink](<https://devfeed.tech/topics/apache-flink.md>), [autoscaling](<https://devfeed.tech/topics/autoscaling.md>), [Netflix](<https://devfeed.tech/topics/netflix.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [Network](<https://devfeed.tech/topics/network.md>)

Tags: [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [autoscaling](<https://devfeed.tech/tags/autoscaling.md>), [aws](<https://devfeed.tech/tags/aws.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [netflix](<https://devfeed.tech/tags/netflix.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [operational-excellence](<https://devfeed.tech/tags/operational-excellence.md>), [scale](<https://devfeed.tech/tags/scale.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>)

## AI overview

Netflix describes running two Apache Flink autoscalers: an older in-house system and a newer one from the Apache Flink community. The article explains why autoscaling is necessary for more than 30,000 Flink jobs across AWS regions, discusses workload variability and the cost of resizing stateful jobs, and introduces the metrics-driven design of the first autoscaler.

## Source excerpt

Samuel Yeboah, Francesco Di Chiara and Mingliang Liu Today, Netflix runs two Flink autoscalers. That is exactly one more than we want. We built the first one in-house years ago, when there was no mature option suited to our platform. The second came from the Apache Flink community, and it can scale workloads our homegrown system was never designed for. We now run both in production and are steadily converging on the open-source one. Along the way we learned some hard lessons about metrics, cost, and the real price of maintaining infrastructure you could instead adopt, and we hope they are useful whether you run a handful of Flink jobs or tens of thousands. Why autoscaling is not optional at our scale Netflix has run stream processing on Apache Flink since 2017. As of 2026 we operate more than 30,000 Flink jobs across multiple AWS regions. Most are not deployed by hand; they are generated by our managed platform Data Mesh, so the majority of users never touch a Flink job directly. A smaller but growing set are custom jobs, built and operated by teams across the company for use cases like personalization, Ads, and Live events. They range from single-operator jobs that shuttle records between Kafka topics to stateful pipelines with branches, joins, and terabytes of state, and their load swings with daily cycles, launches, and regional failovers. Provisioning every one of those jobs for its peak is wasteful; provisioning for the average causes lag during surges. And in our platform a scaling action is not free: by default it means taking a savepoint, stopping the job gracefully, and restarting it at the new size, which for a large stateful job can take minutes. That leaves a genuinely hard question: how do you give each job the resources it needs, when it needs them, without a human in the loop and without breaking anything? The first autoscaler: watching from outside Our first answer, built around 2019, was an autoscaler shaped like a stream-processing job. It ran on M