# Hypothesis Testing in Production

DevFeed: [Hypothesis Testing in Production](<https://devfeed.tech/articles/hypothesis-testing-in-production-31928.md>)

Original publisher: [Read original article](<http://engineering.remind.com/hypothesis-testing-in-production/>)

Author: Remind

Published: 2019-12-18T00:00:00Z

Content type: tutorial

Language: en

Sources: [Remind](<https://devfeed.tech/sources/remind.md>)

Topics: [Testing](<https://devfeed.tech/topics/testing.md>), [debugging](<https://devfeed.tech/topics/debugging.md>), [incident](<https://devfeed.tech/topics/incident.md>), [Microservice](<https://devfeed.tech/topics/microservice.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Databases](<https://devfeed.tech/topics/databases.md>), [Cache](<https://devfeed.tech/topics/cache.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [availability](<https://devfeed.tech/tags/availability.md>), [aws](<https://devfeed.tech/tags/aws.md>), [cache](<https://devfeed.tech/tags/cache.md>), [databases](<https://devfeed.tech/tags/databases.md>), [debugging](<https://devfeed.tech/tags/debugging.md>), [incident](<https://devfeed.tech/tags/incident.md>), [load-balancer](<https://devfeed.tech/tags/load-balancer.md>), [testing](<https://devfeed.tech/tags/testing.md>)

## AI overview

This tutorial explains how to use hypothesis testing to investigate and resolve production incidents. It presents a debugging process based on monitoring, notifications, measurements, data, and verification, using an example involving a routing service in a microservice architecture with a legacy monolith.

## Source excerpt

At Remind we have built a robust communication platform that makes it easy for educators, students, and parents to work together. Our users use Remind to keep classes organized, to help each other with school assignments, and to send out announcements for school events. Given these important use cases, users depend on Remind to deliver out their messages on time and reliably. As we scale, grow and change our systems to support all our users and meet their expectations, issues are bound to happen. When issues do happen, it is critical to identify and resolve them quickly to reduce the impact on our users. Throughout the years, we have developed techniques and common patterns that are useful when fixing issues. In this post, we'll walk through a process of using hypothesis testing to guide us in fixing an issue. An overview of our architecture Our architecture is a pretty standard microservice architecture (mixed with a legacy monolith). We have a routing layer that takes each user request and routes it to the appropriate service. Each service consumes other systems like databases or cache and lives behind a load balancer so that we can scale independently to handle our load. All of this is running on AWS spread out across multiple availability zones for robustness. While this serves us well, it adds an additional layer of complexity when we are debugging issues and incidents. We'll be going through an example which touches multiple services as well as our legacy monolith, which is called r101-api. Debugging an incident The first step for debugging an incident is to have monitors in place. With a monitor in place, notifications can be put in place to alert the appropriate people when an issue arises. Creating a good monitor is a topic in and of itself, but a good place to start is to alert when a threshold of failing requests is reached. Once we have an initial measurement of a potential issue, we can start to diagnose it and look at the data to form a hypothesis of w