# Metastability and Distributed Systems

DevFeed: [Metastability and Distributed Systems](<https://devfeed.tech/articles/metastability-and-distributed-systems-12508.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2021/05/24/metastable.html>)

Author: Marc Brooker

Published: 2021-05-24T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog.md>), [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog-2.md>)

Topics: [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [systems](<https://devfeed.tech/topics/systems.md>), [Caching](<https://devfeed.tech/topics/caching.md>), [Deployment](<https://devfeed.tech/topics/deployment.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>)

Tags: [caching](<https://devfeed.tech/tags/caching.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [latency](<https://devfeed.tech/tags/latency.md>), [outages](<https://devfeed.tech/tags/outages.md>), [systems](<https://devfeed.tech/tags/systems.md>)

## AI overview

The article explains metastable failures in distributed systems: triggers such as brief failures, load spikes, deployments, retries, caching, slow error handling, and load-balancing behavior can push a system into a stable but ineffective state. Queues build up, latency increases, clients time out, and the system may continue processing work that produces no useful results, potentially causing prolonged outages.

## Source excerpt

Metastability and Distributed Systems What if computer science had different parents? There's no more time-honored way to get things working again, from toasters to global-scale distributed systems, than turning them off and on again. The reasons that works so well are varied, but one reason is especially important for the developers and operators of distributed systems: metastability. I'll let the authors of Metastable Failures in Distributed Systems define what that means: Metastable failures occur in open systems with an uncontrolled source of load where a trigger causes the system to enter a bad state that persists even when the trigger is removed. What they're identifying here is a kind of stable down state, where the system is stable but not doing useful work, even though it's only being offered a load that it successfully handled sometime in the past. One classic version of this problem involves queues. A system is ticking along nicely, and something happens. Could be a short failure, a spike of load, a deployment, or one of many other things. This causes queues to back up in the system, causing an increase in latency. That increased latency causes clients to time out before the system responds to them. Clients continue to send work, and the system continues to complete that work. Throughput is great. None of the work is useful, though, because clients aren't waiting for the results, so goodput is zero. The system is mostly stable in this state, and without an external kick, could continue going along that way indefinitely. Up, but down. Working, but broken. In Metastable Failures in Distributed Systems, Bronson et al correctly observe that these types of failure modes are well-known1 to the builders of large-scale systems: By reviewing experiences from a decade of operating hyperscale distributed systems, we identify a class of failures that can disrupt them, even when there are no hardware failures, configuration errors, or software bugs. These metastable f