# parking\_lot: ffffffffffffffff...

DevFeed: [parking\_lot: ffffffffffffffff...](<https://devfeed.tech/articles/parking-lot-ffffffffffffffff-1714.md>)

Original publisher: [Read original article](<https://fly.io/blog/parking-lot-ffffffffffffffff/>)

Published: 2025-05-28T00:00:00Z

Content type: article

Language: en

Sources: [Fly](<https://devfeed.tech/sources/the-fly-blog.md>)

Topics: [fly.io](<https://devfeed.tech/topics/fly-io.md>), [virtual machines](<https://devfeed.tech/topics/virtual-machines.md>), [Programming language](<https://devfeed.tech/topics/programming-language.md>), [async](<https://devfeed.tech/topics/async.md>), [Cloud Networking](<https://devfeed.tech/topics/cloud-networking.md>), [Network](<https://devfeed.tech/topics/network.md>)

Tags: [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [bug](<https://devfeed.tech/tags/bug.md>), [cdn](<https://devfeed.tech/tags/cdn.md>), [certificate](<https://devfeed.tech/tags/certificate.md>), [close-to-users](<https://devfeed.tech/tags/close-to-users.md>), [containers](<https://devfeed.tech/tags/containers.md>), [deploy-app-servers](<https://devfeed.tech/tags/deploy-app-servers.md>), [docker](<https://devfeed.tech/tags/docker.md>), [elixir](<https://devfeed.tech/tags/elixir.md>), [fly](<https://devfeed.tech/tags/fly.md>), [fly-io](<https://devfeed.tech/tags/fly-io.md>), [heroku-alternative](<https://devfeed.tech/tags/heroku-alternative.md>), [heroku-competitor](<https://devfeed.tech/tags/heroku-competitor.md>), [hosting](<https://devfeed.tech/tags/hosting.md>), [i](<https://devfeed.tech/tags/i.md>), [machine](<https://devfeed.tech/tags/machine.md>), [network](<https://devfeed.tech/tags/network.md>), [networking](<https://devfeed.tech/tags/networking.md>), [postgresql-clusters](<https://devfeed.tech/tags/postgresql-clusters.md>), [project](<https://devfeed.tech/tags/project.md>), [protocols](<https://devfeed.tech/tags/protocols.md>), [proxy](<https://devfeed.tech/tags/proxy.md>), [public-cloud](<https://devfeed.tech/tags/public-cloud.md>), [regions](<https://devfeed.tech/tags/regions.md>), [request](<https://devfeed.tech/tags/request.md>), [router](<https://devfeed.tech/tags/router.md>), [routing](<https://devfeed.tech/tags/routing.md>), [rust](<https://devfeed.tech/tags/rust.md>), [servers](<https://devfeed.tech/tags/servers.md>)

## AI overview

Fly.io's Anycast router, fly-proxy, must track rapidly changing Machine availability across thousands of servers so it can route requests correctly. Machines may start in under a second or terminate instantly, making distribution of routing state across millions of connections and apps the central challenge. Fly.io describes this as a routing protocol problem and identifies Corrosion as its current protocol. The excerpt introduces the system and its operational constraints but does not include the bug's diagnosis or fix.

## Source excerpt

We're Fly.io, a public cloud that runs apps in a bunch of locations all over the world. This is a post about a gnarly bug in our Anycast router, the largest Rust project in our codebase. You won't need much of any Rust to follow it. The basic idea is simple: customers give us Docker containers, and tell us which one of 30+ regions around the world they want them to run in. We convert the containers into lightweight virtual machines, and then link them to an Anycast network. If you boot up an app here in Sydney and Frankfurt, and a request for it lands in Narita, it'll get routed to Sydney. The component doing that work is called fly-proxy. It's a Rust program, and it has been ill behaved of late. Dramatis Personae fly-proxy, our intrepid Anycast router. corrosion, our intrepid Anycast routing protocol. Rust, a programming language you probably don't use. read-write locks, a synchronization primitive that allows for many readers or one single writer. parking_lot, a well-regarded optimized implementation of locks in Rust. Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing Mathewson 6:31 Anycast Routing You already know how to build a proxy. Connection comes in, connection goes out, copy back and forth. So for the amount of time we spend writing about fly-proxy, you might wonder what the big deal is. To be fair, in the nuts and bolts of actually proxying requests, fly-proxy does some interesting stuff. For one thing, it's written in Rust, which is apparently a big deal all on its own. It's a large, asynchronous codebase that handles multiple protocols, TLS termination, and certificate issuance. It exercises a lot of Tokio features. But none of this is the hard part of fly-proxy. We operate thousands of servers around the world. A customer Fly Machine might get scheduled onto any of them; in some places, the expectation we set with customers is that their Machines will potentially start in l