# Surprising Economics of Load-Balanced Systems

DevFeed: [Surprising Economics of Load-Balanced Systems](<https://devfeed.tech/articles/surprising-economics-of-load-balanced-systems-12498.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2020/08/06/erlang.html>)

Author: Marc Brooker

Published: 2020-08-06T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog.md>), [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog-2.md>)

Topics: [Latency](<https://devfeed.tech/topics/latency.md>), [systems](<https://devfeed.tech/topics/systems.md>), [Server](<https://devfeed.tech/topics/server.md>), [Back end](<https://devfeed.tech/topics/backend.md>), [client](<https://devfeed.tech/topics/client.md>)

Tags: [backend](<https://devfeed.tech/tags/backend.md>), [economics](<https://devfeed.tech/tags/economics.md>), [latency](<https://devfeed.tech/tags/latency.md>), [load-balancer](<https://devfeed.tech/tags/load-balancer.md>), [queuing](<https://devfeed.tech/tags/queuing.md>), [server](<https://devfeed.tech/tags/server.md>), [servers](<https://devfeed.tech/tags/servers.md>), [systems](<https://devfeed.tech/tags/systems.md>)

## AI overview

The article examines an M/M/c queuing system in which single-request servers sit behind a load balancer with an infinite queue. Using Erlang's C formula, it explains that increasing the number of servers while proportionally increasing load reduces queuing and causes mean client-observed latency to asymptotically approach the one-second service time.

## Source excerpt

Surprising Economics of Load-Balanced Systems The M/M/c model may not behave like you expect. I have a system with c servers, each of which can only handle a single concurrent request, and has no internal queuing. The servers sit behind a load balancer, which contains an infinite queue. An unlimited number of clients offer c * 0.8 requests per second to the load balancer on average. In other words, we increase the offered load linearly with c to keep the per-server load constant. Once a request arrives at a server, it takes one second to process, on average. How does the client-observed mean request time vary with c? Option A is that the mean latency decreases quickly, asymptotically approaching one second as c increases (in other words, the time spent in queue approaches zero). Option B is constant. Option C is a linear improvement, and D is a linear degradation in latency. Which curve do you, intuitively, think that the latency will follow? I asked my Twitter followers the same question, and got an interestingly mixed result: Breaking down the problem a bit will help figure out which is the right answer. First, names. In the terminology of queue theory, this is an M/M/c queuing system: Poisson arrival process, exponentially distributed client service time, and c backend servers. In teletraffic engineering, it's Erlang's delay system (or, because terminology is fun, M/M/n). We can use a classic result of queuing theory to analyze this system: Erlang's C formula E2,n(A), which calculates the probability that an incoming customer request is enqueued (rather than handled immediately), based on the number of servers (n aka c), and the offered traffic A. For the details, see page 194 of the Teletraffic Engineering Handbook. Here's the basic shape of the curve (using our same parameters): Follow the blue line up to half the saturation point, at 2.5 rps offered load, and see how the probability is around 13%. Now look at the purple line at half its saturation point, at 5