# Good Performance for Bad Days

DevFeed: [Good Performance for Bad Days](<https://devfeed.tech/articles/good-performance-for-bad-days-12572.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2025/05/20/icpe.html>)

Author: Marc Brooker

Published: 2025-05-20T00:00:00Z

Content type: opinion

Language: en

Sources: [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog.md>), [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog-2.md>)

Topics: [systems](<https://devfeed.tech/topics/systems.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [Scalability](<https://devfeed.tech/topics/scalability.md>)

Tags: [latency](<https://devfeed.tech/tags/latency.md>), [performance](<https://devfeed.tech/tags/performance.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [systems](<https://devfeed.tech/tags/systems.md>)

## AI overview

The article argues that performance evaluation should examine systems under saturation and overload, not only happy-case throughput, latency, and scalability. Unpredictable behavior under overload can drive unavailability, persistent metastable failures, and downtime.

## Source excerpt

Good Performance for Bad Days Good things are good, one finds. Two weeks ago, I flew to Toronto to give one of the keynotes at the International Conference on Performance Evaluation. It was fun. Smart people. Cool dark squirrels. Interesting conversations. The core of what I tried to communicate is that, in my view, a lot of the performance evaluation community is overly focused on happy case performance (throughput, latency, scalability), and not focusing as much as we need to on performance under saturation and overload. In fact, the opposite is potentially more interesting. For builders and operators of large systems, a lack of performance predictability under overload is a big driver of unavailability. This is a common theme in postmortems and outage reports across the industry. Overload drives systems into regimes they aren't used to handling, which leads to downtime. Sometimes, in the case of metastable failures, this leads to downtime that persists even after the overload has passed. How did we get into this situation? Not Measuring the Hard Stuff At least one reason is immediately obvious if you pay attention to the performance evaluation in the majority of systems papers. Most of them show throughput, latency, or some other measure of goodness at a load far from the saturation point of the system. The first-order reason for this is unsurprising: folks want to show the system they built in a good light. But there are some second-order reasons too. One is that performance evaluation is easiest, and most repeatable, in this part of the performance curve, and it takes expertise that many don't have to push beyond it. Some bolder authors will compare saturation points, showing that their systems are able to do more good stuff even when the load is excessive. Only the boldest will go beyond this saturation point to show the performance of their system under truly excessive amounts of load, after the point where performance starts to drop. This regime is important