# Corrosion

DevFeed: [Corrosion](<https://devfeed.tech/articles/corrosion-1692.md>)

Original publisher: [Read original article](<https://fly.io/blog/corrosion/>)

Published: 2025-10-22T00:00:00Z

Content type: article

Language: en

Sources: [The Fly Blog](<https://devfeed.tech/sources/the-fly-blog.md>)

Topics: [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [Concurrent Programming](<https://devfeed.tech/topics/concurrent-programming.md>), [Deadlock](<https://devfeed.tech/topics/deadlock.md>), [Rust](<https://devfeed.tech/topics/rust.md>), [Network](<https://devfeed.tech/topics/network.md>), [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [fly](<https://devfeed.tech/topics/fly.md>), [fly.io](<https://devfeed.tech/topics/fly-io.md>)

Tags: [cdn](<https://devfeed.tech/tags/cdn.md>), [close-to-users](<https://devfeed.tech/tags/close-to-users.md>), [deadlock](<https://devfeed.tech/tags/deadlock.md>), [deploy-app-servers](<https://devfeed.tech/tags/deploy-app-servers.md>), [distributed-system](<https://devfeed.tech/tags/distributed-system.md>), [docker](<https://devfeed.tech/tags/docker.md>), [docker-containers](<https://devfeed.tech/tags/docker-containers.md>), [elixir](<https://devfeed.tech/tags/elixir.md>), [fly](<https://devfeed.tech/tags/fly.md>), [fly-io](<https://devfeed.tech/tags/fly-io.md>), [heroku-alternative](<https://devfeed.tech/tags/heroku-alternative.md>), [heroku-competitor](<https://devfeed.tech/tags/heroku-competitor.md>), [hosting](<https://devfeed.tech/tags/hosting.md>), [i](<https://devfeed.tech/tags/i.md>), [networking](<https://devfeed.tech/tags/networking.md>), [outage](<https://devfeed.tech/tags/outage.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [postgresql-clusters](<https://devfeed.tech/tags/postgresql-clusters.md>), [routing](<https://devfeed.tech/tags/routing.md>), [rust](<https://devfeed.tech/tags/rust.md>), [servers](<https://devfeed.tech/tags/servers.md>), [synchronization](<https://devfeed.tech/tags/synchronization.md>)

## AI overview

This article introduces Corrosion, Fly.io's open-source distributed state synchronization and service discovery system. It explains how Fly.io propagates workload and routing state across globally distributed servers and edge proxies, and recounts a severe outage caused by a Rust concurrency bug that triggered a contagious deadlock. The article also describes Fly.io's decentralized orchestration model, in which individual servers are authoritative for their workloads instead of relying on a centralized database.

## Source excerpt

Fly.io transmogrifies Docker containers into Fly Machines: micro-VMs running on our own hardware all over the world. The hardest part of running this platform isn't managing the servers, and it isn't operating the network; it's gluing those two things together. Several times a second, as customer CI/CD pipelines tear up or bring down Fly Machines, our state synchronization system blasts updates across our internal mesh, so that edge proxies from Tokyo to Amsterdam can keep the accurate routing table that allows them to route requests for applications to the nearest customer instances. On September 1, 2024, at 3:30PM EST, a new Fly Machine came up with a new "virtual service" configuration option a developer had just shipped. Within a few seconds every proxy in our fleet had locked up hard. It was the worst outage we've experienced: a period during which no end-user requests could reach our customer apps at all. Distributed systems are blast amplifiers. By propagating data across a network, they also propagate bugs in the systems that depend on that data. In the case of Corrosion, our state distribution system, those bugs propagate quickly. The proxy code that handled that Corrosion update had succumbed to a notorious Rust concurrency footgun: an if let expression over an RWLock assumed (reasonably, but incorrectly) in its else branch that the lock had been released. Instant and virulently contagious deadlock. A lesson we've learned the hard way: never trust a distributed system without an interesting failure story. If a distributed system hasn't ruined a weekend or kept you up overnight, you don't understand it yet. Which is why that's how we're introducing Corrosion, an unconventional service discovery system we built for our platform and open sourced. Our Face-Seeking Rake State synchronization is the hardest problem in running a platform like ours. So why build a risky new distributed system for it? Because no matter what we try, that rake is waiting for our foot