# Don't let overload protection be the enemy of scaling

DevFeed: [Don't let overload protection be the enemy of scaling](<https://devfeed.tech/articles/don-t-let-overload-protection-be-the-enemy-of-scaling-131132.md>)

Original publisher: [Read original article](<https://medium.com/whatnot-engineering/dont-let-overload-protection-be-the-enemy-of-scaling-7f59abbcb070?source=rss----162aeca881b0---4>)

Author: Whatnot Engineering

Published: 2026-11-12T13:05:42Z

Content type: article

Language: en

Sources: [Whatnot Engineering](<https://devfeed.tech/sources/whatnot-engineering.md>)

Topics: [consistency](<https://devfeed.tech/topics/consistency.md>), [read-replica](<https://devfeed.tech/topics/read-replica.md>), [database-optimization](<https://devfeed.tech/topics/database-optimization.md>), [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [backend](<https://devfeed.tech/tags/backend.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [database](<https://devfeed.tech/tags/database.md>), [fleet](<https://devfeed.tech/tags/fleet.md>), [graphql](<https://devfeed.tech/tags/graphql.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [ip](<https://devfeed.tech/tags/ip.md>), [limit](<https://devfeed.tech/tags/limit.md>), [load](<https://devfeed.tech/tags/load.md>), [protection](<https://devfeed.tech/tags/protection.md>), [query](<https://devfeed.tech/tags/query.md>), [queue](<https://devfeed.tech/tags/queue.md>), [redis](<https://devfeed.tech/tags/redis.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [request](<https://devfeed.tech/tags/request.md>), [rest](<https://devfeed.tech/tags/rest.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [site-reliability-engineer](<https://devfeed.tech/tags/site-reliability-engineer.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [sre](<https://devfeed.tech/tags/sre.md>), [traffic](<https://devfeed.tech/tags/traffic.md>)

## AI overview

Whatnot replaced its concurrency limiter's precise, synchronous Redis counting after the old design remained a bottleneck even across nine primaries. The new distributed semaphore tolerates slightly stale counts: pods check current and previous window counters on replicas, accumulate request deltas locally, and batch INCRBY updates to primaries every 100 ms. Window keys can land on different shards, spreading hot traffic; Redis LRU eviction replaces per-request TTL updates. The limiter still throttles slow request types, returning 503 when a type exceeds its cap and 429 when an IP does. It can miss very short requests that start and finish between syncs, a tradeoff the team accepted because slow requests are the overload concern. In pre-Black-Friday 2025 load tests, the hottest primary across eight shards stayed below 50% CPU; ahead of Black Friday 2026, the semaphore handled four times the traffic that would overwhelm the old limiter, with headroom. Limits remain static; an adaptive controller is planned.

## Source excerpt

Marcin Mierzejewski | Reliability Platform (Poland) The concurrency limiter in front of our main API at Whatnot has one job: protect the backend from overload. As we scaled, though, it became the component most likely to fall over under a traffic surge. We had grown its Redis fleet from one instance to nine, and the hottest instance still peaked at 92% CPU in load tests. We had even configured that instance to send 60% of its traffic elsewhere, but it still stayed the hottest. Keeping track of every request was a losing effort. The way out was to stop counting so precisely. Concurrency limiting in a nutshell In short, we limit concurrency rather than rate. We count in-flight requests per request type (GraphQL operation names or REST endpoints), so a request type that slows down gets throttled harder. Why count requests in flight instead of requests per second? The number in flight is roughly the arrival rate times how long each request takes. If a database query slows from, say, 50 ms to 500 ms, the same traffic keeps ten times as many requests in flight. Each one holds a worker and a database connection while it waits, so the pile-up slows everything else down, which makes requests slower still. A rate limiter doesn't notice, because the request rate hasn't changed. A concurrency limiter does. The slow request type hits its cap on in-flight requests, and the extras get a fast 503 instead of joining the queue. That request type sheds load early, and the rest of the API keeps its workers and connections. A separate per-IP cap stops any one client from taking more than its share. For simplicity, we'll only talk about request-type concurrency limiting, unless stated otherwise. How the limiter became the bottleneck In 2024, my colleague Karol Gil wrote about how we initially built the limiter. That design worked, and a few weeks later it throttled a query storm that nearly tripled request volume. However, it paid for a precise, synchronous count on every request. Each A