# Using SLOs and OpenTelemetry to reduce alert fatigue

DevFeed: [Using SLOs and OpenTelemetry to reduce alert fatigue](<https://devfeed.tech/articles/how-to-make-your-customers-happy-and-your-engineers-even-happier-33272.md>)

Original publisher: [Read original article](<https://8thlight.com/insights/how-to-make-your-customers-happy-and-your-engineers-even-happier>)

Author: Andy Smith

Published: 2025-08-18T19:03:00Z

Content type: tutorial

Language: en

Sources: [8th Light Insights](<https://devfeed.tech/sources/8th-light-insights.md>)

Topics: [observability](<https://devfeed.tech/topics/observability.md>), [site-reliability-engineering](<https://devfeed.tech/topics/site-reliability-engineering.md>), [Prometheus](<https://devfeed.tech/topics/prometheus.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [incident](<https://devfeed.tech/topics/incident.md>)

Tags: [alert-fatigue](<https://devfeed.tech/tags/alert-fatigue.md>), [delivery-and-practice](<https://devfeed.tech/tags/delivery-and-practice.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [observability](<https://devfeed.tech/tags/observability.md>), [prometheus](<https://devfeed.tech/tags/prometheus.md>)

## AI overview

The article describes how frequent, non-customer-impacting alerts can cause alert fatigue and delay detection of production incidents. It presents Service Level Objectives and increased observability with OpenTelemetry as changes intended to focus engineers on user-impacting failures.

## Source excerpt

How to make your customers happy and your engineers even happier It's common for monitoring and alerting solutions on production services to be quite chatty, sometimes producing hundreds of notifications per day. There's a scene in the classic film The Matrix where one of the characters (Cypher) is looking at a screen with green unintelligible code. He's asked: "Do you always look at it encoded?" He responds by telling Neo he doesn't see code, just what it represents. After some time in The Matrix like Cypher, in the sea of notifications you understand what they mean and know what is important. Unfortunately, for your new team members they have to learn what they mean and may have no idea what customer impact these alerts have. If too many of these notifications are not customer impacting and just informational what follows is "alert fatigue", you begin to tune out from them because of the volume. This becomes "dangerous" when you miss something in the noise that IS important. You can bet that the notification you missed will lead to a 3AM support call from an irate customer which is SLO not fun! Scenario I joined an eight person team for a large enterprise client that had a few production incidents. There were two in July which were high stress problems, the team worked impeccably to resolve them. However, users of the website noticed them first, called Customer Support, then it was "all hands on deck" to try and resolve. This is known internally as a "Major Incident"*. The team fixed the issues and everyone was happy but of course the damage was already done, reputation was harmed. We were in reactive fire fighting mode. It doesn't have to be like this. It was not the case that there was no monitoring, there were SLO many Prometheus alerts firing, and that's the problem. Due to the frequency of these alerts, a lot of them were ignored. They didn't tell the engineers specifically that a key feature of the site was not working. Just because a pod has crashed with "o