# Monitoring and alert system using Graphite and Cabot

DevFeed: [Monitoring and alert system using Graphite and Cabot](<https://devfeed.tech/articles/monitoring-and-alert-system-using-graphite-and-cabot-20008.md>)

Original publisher: [Read original article](<http://engineering.hackerearth.com/2017/03/21/monitoring-and-alert-system-using-graphite-and-cabot/>)

Published: 2017-03-21T00:00:00Z

Content type: article

Language: en

Sources: [HackerEarth](<https://devfeed.tech/sources/hackerearth.md>)

Topics: [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [site-reliability-engineering](<https://devfeed.tech/topics/site-reliability-engineering.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [dashboards](<https://devfeed.tech/topics/dashboards.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [backends](<https://devfeed.tech/topics/backends.md>)

Tags: [apis](<https://devfeed.tech/tags/apis.md>), [aws](<https://devfeed.tech/tags/aws.md>), [backend](<https://devfeed.tech/tags/backend.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [on-call](<https://devfeed.tech/tags/on-call.md>), [visualization](<https://devfeed.tech/tags/visualization.md>)

## AI overview

This article describes a monitoring and alerting system for complex product infrastructure. It covers collecting, forwarding, storing, visualizing, and querying time-series metrics, along with monitoring checks, dashboards, and alerts for service owners and on-call teams. It also explains the motivation for replacing ad hoc monitoring scripts and AWS CloudWatch email alarms.

## Source excerpt

Introduction The infrastructure that powers a product and all of the services that it provides can be huge and complex because the product is scaled to serve millions of users. In most cases, each service might depend on various components for seamless functioning. With a product that houses a variety of features with critical infrastructure components and services powering these features, it becomes vital to monitor these components and services and keep them running at any cost. This monitoring system has to handle the following: Gathering data from all the components and services Storing the data efficiently and in an easily accessible manner Visualizing the data for faster comprehension Making sense of this data and relaying alerts to the respective owners of the services and components Managing the on call team and alerting them immediately At HackerEarth... Initially when we began facing problems because of some of our machines and services going down, we wrote ad hoc monitoring scripts that ran as crons to send email alerts. We also set up AWS Cloudwatch alarms to send notifications via email. There came a time when we had a very high number of components to be monitored and we realised that we were not getting enough insight into the load and usage of our machines. This is when we decided to put a system in place to collect data from these monitors and services. We also added a monitoring component to send alerts in more reliable ways (through phone calls) to the product owners and our on-call team, in case of any downtime. Components of this system This system consists of the following components that work together, Collection: Tool for collecting metrics from all the infrastructure components Forwarding: Tool for aggregating the metrics that are recieved from various machines and services and routing it to different storage backends Visualization: Tool for generating graphs and visualizing metrics APIs: Backend that provides APIs to query the metrics data fo