# Incident Response Isn't Enough

DevFeed: [Incident Response Isn't Enough](<https://devfeed.tech/articles/incident-response-isn-t-enough-12504.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2021/02/22/postmortem.html>)

Author: Marc Brooker

Published: 2021-02-22T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog.md>), [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog-2.md>)

Topics: [Availability](<https://devfeed.tech/topics/availability.md>), [Incident response](<https://devfeed.tech/topics/incident-response.md>), [incident](<https://devfeed.tech/topics/incident.md>), [systems](<https://devfeed.tech/topics/systems.md>)

Tags: [availability](<https://devfeed.tech/tags/availability.md>), [incident](<https://devfeed.tech/tags/incident.md>), [incident-response](<https://devfeed.tech/tags/incident-response.md>), [organizational](<https://devfeed.tech/tags/organizational.md>), [systems](<https://devfeed.tech/tags/systems.md>), [technical](<https://devfeed.tech/tags/technical.md>)

## AI overview

The article argues that postmortems, COEs, and incident reports are useful for investigating failures and driving change, but they are not sufficient for improving long-term system availability. Effective availability work also requires deep understanding of system design, especially shared components that can create a large blast radius. The house-and-community analogy illustrates the trade-off between the economic benefits of shared infrastructure and the correlated failures it can introduce, along with organizational and cultural effects on which issues receive attention.

## Source excerpt

Incident Response Isn't Enough Single points of failure become invisible. Postmortems, COEs, incident reports. Whatever your organization calls them, when done right they are a popular and effective way of formalizing the process of digging into system failures, and driving change. The success of this approach has lead some to believe that postmortems are the best, or even only, way to improve the long-term availability of systems. Unfortunately, that isn't true. A good availability program requires deep insight into the design of the system. To understand why, let's build a house, then a small community. Our house has four walls, a roof, and a few things it needs to be a habitable home. We've got a well for water, a field of corn for food, a wood pile for heat, and a septic tank. If any one of these things is not working, let's say that the house is unavailable. Our goal is to build many houses, and make sure they are unavailable for as little of the time as possible. When we want to build a second house, we're faced with a choice. The simple approach is just to stamp out a second copy of the entire house, with it's own field, wood, well, and tank. That approach is great: the failure of the two houses is completely independent, and availability is very easy to reason about. As we scale this approach up, however, we're met with the economic pressure to share components. This makes a lot of sense: wells are expensive to drill, and don't break down often, so sharing one between many houses could save the home owners a lot of money. Not only does sharing a well reduce construction costs, but thanks to the averaging effect of adding the demand of multiple houses together, reduces the peak-to-average ratio of water demand. That improves ongoing economics, too. In exchange for the improved economics, we've bought ourselves a potential problem. The failure of the well will cause all the houses in our community to become unavailable. The well has high blast radius. Mitigati