# Latency Sneaks Up On You

DevFeed: [Latency Sneaks Up On You](<https://devfeed.tech/articles/latency-sneaks-up-on-you-12509.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2021/08/05/utilization.html>)

Author: Marc Brooker

Published: 2021-08-05T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog.md>), [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog-2.md>)

Topics: [Latency](<https://devfeed.tech/topics/latency.md>), [Server](<https://devfeed.tech/topics/server.md>), [client](<https://devfeed.tech/topics/client.md>), [systems](<https://devfeed.tech/topics/systems.md>), [Statistics](<https://devfeed.tech/topics/statistics.md>)

Tags: [latency](<https://devfeed.tech/tags/latency.md>), [server](<https://devfeed.tech/tags/server.md>), [statistics](<https://devfeed.tech/tags/statistics.md>), [systems](<https://devfeed.tech/tags/systems.md>)

## AI overview

The article explains how increasing system utilization can cause queue depth and latency to rise nonlinearly. It uses a simple client-queue-server model and defines utilization as the ratio of mean arrival rate to mean completion rate.

## Source excerpt

Latency Sneaks Up On You And is a bad way to measure efficiency. As systems get big, people very reasonably start investing more in increasing efficiency and decreasing costs. That's a good thing, for the business, for the environment, and often for the customer. Much of the time efficient systems have lower and more predictable latencies, and everybody enjoys lower and more predictable latencies. Most folks around me think about latency using percentiles or other order statistics. Common practice is to look at the median (50th percentile), and some high percentile like the 99.9th or 99.99th. As they do efficiency work, they often see that not only does their 50th percentile improve a lot, but so does the high percentiles. Then, just a short while later, the high percentiles have crept back up without the code getting slower again. What's going on? A lot of things, as usual, but one of them is the non-linear effect of utilization. To explain that, let's consider a simple system with one server that can serve one thing at a time, and a queue in front of it1. ┌────────┐ ┌─────┐ ┌──────┐ │ Client │────▶│Queue│───▶│Server│ └────────┘ └─────┘ └──────┘ Simple as it comes, really. We can then define the Utilization of the server (calling it ⍴ for traditional reasons) in terms of two other numbers: the mean arrival rate λ, and the mean completion rate μ, both in units of jobs/second. ⍴ = λ/μ Another way to think of ⍴ is that the server is idle (no work to do, empty queue) 1-⍴ of the time. So if ⍴=0.4, then the server is idle 60% of the time. Clearly, if ⍴>1 for a long time then the queue grows without bound, because more stuff is arriving than leaving. Let's ignore that for now, because infinite queues are silly things. To understand what happens here, we need to look at the diagram above, and notice that there's no feedback loop. The client sends work2 at random, on it's own schedule. Sometimes that's when the server is idle, and sometimes when it's busy. When the server i