# Why Must Systems Be Operated?

DevFeed: [Why Must Systems Be Operated?](<https://devfeed.tech/articles/why-must-systems-be-operated-12479.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2016/01/03/correlation.html>)

Author: Marc Brooker

Published: 2016-01-03T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog.md>), [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog-2.md>)

Topics: [systems](<https://devfeed.tech/topics/systems.md>), [data](<https://devfeed.tech/topics/data.md>), [Disaster Recovery](<https://devfeed.tech/topics/disaster-recovery.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [servers](<https://devfeed.tech/topics/servers.md>)

Tags: [backup](<https://devfeed.tech/tags/backup.md>), [data](<https://devfeed.tech/tags/data.md>), [disaster-recovery](<https://devfeed.tech/tags/disaster-recovery.md>), [load-balancer](<https://devfeed.tech/tags/load-balancer.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [outages](<https://devfeed.tech/tags/outages.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [safety](<https://devfeed.tech/tags/safety.md>), [server](<https://devfeed.tech/tags/server.md>), [servers](<https://devfeed.tech/tags/servers.md>), [storage](<https://devfeed.tech/tags/storage.md>), [systems](<https://devfeed.tech/tags/systems.md>)

## AI overview

The article explains why systems must be operated with awareness of dependent and latent failures. Using mirrored RAID as the main example, it shows how correlated failures, failures triggered by an initial failure, and hidden latent failures can defeat simple independent-failure models. It also applies these ideas to load balancers and web servers, arguing that external black-box monitoring alone is insufficient.

## Source excerpt

Why Must Systems Be Operated? Latent Failures and the Safety Margin of Systems Mirrored RAID1 is a classic way of increasing storage durability. It's also a classic example of a system that's robust against independent failures, but fragile against dependent failure. Patterson et al's 1988 paper, which popularized mirroring, even covered the problem: As mentioned above we make the same assumptions that disk manufacturers make - that the failures are exponential and independent. (An earthquake or power surge is a situation where an array of disks might not fail independently.) A 2-way striped RAID can be in three possible states: a state with no failures, a state with one failure, or a state with two failures. The system moves between the first and second states, and the second and third states, when a failure happens. It can return from the second state to the first by repair. In the third state, data is lost, and returning becomes an exercise in disaster recovery (like restoring a backup). The classic Markov model looks like this, with the failure rate λ and repair rate μ: This model clearly displays its naive thinking: it assumes that the failure rate of 2 disks is double the failure rate of a single disk2. All experienced system operators know that's not true in practice. A second disk failure seems more likely to happen soon after a first. This happens for three reasons. Failures with the same cause. These failures, like Patterson's earthquakes and power surges, affect both drives at the same time. A roof falling in on a server can move its RAID from state 1 to state 3 pretty quickly. Operator mistakes are also a common (and maybe dominant) source of these kinds of failures. Failures triggered by the first failure. When the first drive fails, it triggers a failure of the second drive. In a RAID, the second drive is going to be put under high load as the system attempts to get back to two good copies. This extra load increases the probability of the second drive