# Taming A Voracious Rust Proxy

DevFeed: [Taming A Voracious Rust Proxy](<https://devfeed.tech/articles/taming-a-voracious-rust-proxy-1718.md>)

Original publisher: [Read original article](<https://fly.io/blog/taming-rust-proxy/>)

Published: 2025-02-26T00:00:00Z

Content type: article

Language: en

Sources: [The Fly Blog](<https://devfeed.tech/sources/the-fly-blog.md>)

Topics: [Rust](<https://devfeed.tech/topics/rust.md>), [fly](<https://devfeed.tech/topics/fly.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [incident](<https://devfeed.tech/topics/incident.md>), [Tokio](<https://devfeed.tech/topics/tokio.md>), [Containers](<https://devfeed.tech/topics/containers.md>), [Network](<https://devfeed.tech/topics/network.md>), [virtual machines](<https://devfeed.tech/topics/virtual-machines.md>), [HTTP](<https://devfeed.tech/topics/http.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [cpu](<https://devfeed.tech/topics/cpu.md>)

Tags: [async](<https://devfeed.tech/tags/async.md>), [cdn](<https://devfeed.tech/tags/cdn.md>), [close-to-users](<https://devfeed.tech/tags/close-to-users.md>), [containers](<https://devfeed.tech/tags/containers.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [deploy-app-servers](<https://devfeed.tech/tags/deploy-app-servers.md>), [docker](<https://devfeed.tech/tags/docker.md>), [elixir](<https://devfeed.tech/tags/elixir.md>), [fly](<https://devfeed.tech/tags/fly.md>), [fly-io](<https://devfeed.tech/tags/fly-io.md>), [heroku-alternative](<https://devfeed.tech/tags/heroku-alternative.md>), [heroku-competitor](<https://devfeed.tech/tags/heroku-competitor.md>), [hosting](<https://devfeed.tech/tags/hosting.md>), [http](<https://devfeed.tech/tags/http.md>), [i](<https://devfeed.tech/tags/i.md>), [incident](<https://devfeed.tech/tags/incident.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [network](<https://devfeed.tech/tags/network.md>), [networking](<https://devfeed.tech/tags/networking.md>), [postgresql-clusters](<https://devfeed.tech/tags/postgresql-clusters.md>), [rust](<https://devfeed.tech/tags/rust.md>), [servers](<https://devfeed.tech/tags/servers.md>), [tokio](<https://devfeed.tech/tags/tokio.md>), [tracing](<https://devfeed.tech/tags/tracing.md>), [virtual-machines](<https://devfeed.tech/tags/virtual-machines.md>)

## AI overview

The article investigates a recurring CPU and HTTP-error incident affecting Fly.io edge proxies. Profiling reveals unexpectedly high overhead in Rust tracing's Subscriber while the proxy handles largely idle asynchronous work, leading into a review of async Rust futures and Tokio execution.

## Source excerpt

Here's a fun bug. The basic idea of our service is that we run containers for our users, as hardware-isolated virtual machines (Fly Machines), on hardware we own around the world. What makes that interesting is that we also connect every Fly Machine to a global Anycast network. If your app is running in Hong Kong and Dallas, and a request for it arrives in Singapore, we'll route it to HKG. Our own hardware fleet is roughly divided into two kinds of servers: edges, which receive incoming requests from the Internet, and workers, which run Fly Machines. Edges exist almost solely to run a Rust program called fly-proxy, the router at the heart of our Anycast network. So: a week or so ago, we flag an incident. Lots of things generate incidents: synthetic monitoring failures, metric thresholds, health check failures. In this case two edge tripwires tripped: elevated fly-proxy HTTP errors, and skyrocketing CPU utilization, on a couple hosts in IAD. Our incident process is pretty ironed out at this point. We created an incident channel (we ❤ Rootly for this, seriously check out Rootly, an infra MVP here for years now), and incident responders quickly concluded that, while something hinky was definitely going on, the platform was fine. We have a lot of edges, and we've also recently converted many of our edge servers to significantly beefier hardware. Bouncing fly-proxy clears the problem up on an affected proxy. But this wouldn't be much of an interesting story if the problem didn't later come back. So, for some number of hours, we're in an annoying steady-state of getting paged and bouncing proxies. While this is happening, Pavel, on our proxy team, pulls a profile from an angry proxy. So, this is fuckin' weird: a huge chunk of the profile is dominated by Rust tracing's Subscriber. But that doesn't make sense. The entire point of Rust tracing, which generates fine-grained span records for program activity, is that entering and exiting a span is very, very fast. If the mere