# Outage Resolution Through Automation

DevFeed: [Outage Resolution Through Automation](<https://devfeed.tech/articles/outage-resolution-through-automation-20401.md>)

Original publisher: [Read original article](<https://target.github.io/devops/outage-resolution-through-automation>)

Author: Target Brands, Inc

Published: 2014-12-29T06:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [dashboards](<https://devfeed.tech/topics/dashboards.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [Automation](<https://devfeed.tech/topics/automation.md>), [GitHub](<https://devfeed.tech/topics/github.md>), [Jenkins](<https://devfeed.tech/topics/jenkins.md>), [Ruby](<https://devfeed.tech/topics/ruby.md>), [Agile](<https://devfeed.tech/topics/agile.md>), [data](<https://devfeed.tech/topics/data.md>), [Script](<https://devfeed.tech/topics/script.md>)

Tags: [agile](<https://devfeed.tech/tags/agile.md>), [automation](<https://devfeed.tech/tags/automation.md>), [code](<https://devfeed.tech/tags/code.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [devops](<https://devfeed.tech/tags/devops.md>), [flashbuilds](<https://devfeed.tech/tags/flashbuilds.md>), [github](<https://devfeed.tech/tags/github.md>), [jenkins](<https://devfeed.tech/tags/jenkins.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [outage](<https://devfeed.tech/tags/outage.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [server](<https://devfeed.tech/tags/server.md>), [support](<https://devfeed.tech/tags/support.md>)

## AI overview

The article describes Project Argus, a monitoring-as-code initiative that used GitHub, Chef, Kitchen, Jenkins, and Ruby to build dashboards, automate metric processing, and alert support teams about missing data. It also introduces an outage in the monitoring data-processing servers that caused the dashboards to go blank.

## Source excerpt

Recently we launched Project Argus, a 30-day "monitoring challenge" to improve visibility of key performance indicators (KPIs) across our technology stack prior to our peak retail season (in Greek Mythology, Argus is a 100-eyed giant). This effort was structured as a mix between a FlashBuild and agile. We used two day sprints, twice a day stand-ups, and feature tracking through Kanban boards. Quickly in this effort we decided to build our product as monitoring-as-code. Through the use of tools such as GitHub, Chef, Kitchen, Jenkins, and Ruby, we were able to quickly build several monitoring and dashboard solutions for use within our Technology Operations Center. These dashboards and the iterative process we use to continue delivering more content have been embraced by our support teams who now heavily rely on them to proactively detect and resolve issues. One deliverable from Argus includes a cookbook per core business function that represents the KPIs identified by each business product owner. When run, the cookbooks establish connections to our data sources, build the scripts that process our data, and create crontab entries to automatically run the scripts at our predefined intervals. All of these resources and actions are defined in code. We designed our solution so that the dashboards are decoupled from our centralized processing and metric creation. Additionally, our cookbooks are written to be dashboard agnostic and independent so that anyone can make changes to the appearance and we can switch dashboard solutions easily. The processing of our metrics is centralized and managed on-site with each cookbook receiving its own server. Finally, our dashboard solution detects lapses in data and generates alerts that post into our persistent chat client. This allows us to close the loop on the health of the monitoring system, which can be a painful endeavor! ##The Problem Unfortunately, on 2014-12-06 at around 1:30pm the servers we use to process the monitoring data