# How we page ourselves if incident.io goes down

DevFeed: [How we page ourselves if incident.io goes down](<https://devfeed.tech/articles/how-we-page-ourselves-if-incident-io-goes-down-12078.md>)

Original publisher: [Read original article](<https://incident.io/blog/who-watches-the-watchers>)

Author: Lawrence Jones

Published: 2024-11-27T08:37:35Z

Content type: article

Language: en

Sources: [The incident.io Blog](<https://devfeed.tech/sources/the-incident-io-blog.md>)

Topics: [incident](<https://devfeed.tech/topics/incident.md>), [Incident response](<https://devfeed.tech/topics/incident-response.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>), [cpu](<https://devfeed.tech/topics/cpu.md>), [Kubernetes](<https://devfeed.tech/topics/kubernetes.md>), [systems](<https://devfeed.tech/topics/systems.md>)

Tags: [cpu](<https://devfeed.tech/tags/cpu.md>), [grafana](<https://devfeed.tech/tags/grafana.md>), [incident](<https://devfeed.tech/tags/incident.md>), [incident-channel](<https://devfeed.tech/tags/incident-channel.md>), [incident-management](<https://devfeed.tech/tags/incident-management.md>), [incident-response](<https://devfeed.tech/tags/incident-response.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [observability](<https://devfeed.tech/tags/observability.md>), [on-call](<https://devfeed.tech/tags/on-call.md>), [operational](<https://devfeed.tech/tags/operational.md>), [outage](<https://devfeed.tech/tags/outage.md>), [post-mortem](<https://devfeed.tech/tags/post-mortem.md>), [process](<https://devfeed.tech/tags/process.md>), [production](<https://devfeed.tech/tags/production.md>), [slack-incident](<https://devfeed.tech/tags/slack-incident.md>), [software](<https://devfeed.tech/tags/software.md>), [systems](<https://devfeed.tech/tags/systems.md>)

## AI overview

incident.io explains how it ensures its alerting system can page the on-call team if incident.io itself fails. The article describes a dead man's switch, a normal incident involving high CPU usage detected by Grafana and resolved by replacing a Kubernetes pod, and game-day testing that stresses the alert ingestion pipeline.

## Source excerpt

Learn how we tackle the ultimate paradox: ensuring our alerting system pages us, even when it's the one failing. It's a common question - let's dive into detail on our "dead man's switch", how we stress-test our systems, and why we care so much about our setup allowing us to dogfood our own product.