# product-experimentation

Published articles for product-experimentation.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Mindful Experimentation: Evaluate Recommendation System Performance using A/B Testing at Headspace

DevFeed: [Mindful Experimentation: Evaluate Recommendation System Performance using A/B Testing at Headspace](<https://devfeed.tech/articles/mindful-experimentation-evaluate-recommendation-system-performance-using-a-b-testing-at-headspace-24577.md>)

Original publisher: [Read original article](<https://medium.com/headspace-engineering/mindful-experimentation-evaluate-recommendation-system-performance-using-a-b-testing-at-headspace-3c8c05d0ae3b?source=rss-3da90e297190------2>)

Author: Headspace

Published: 2021-11-29T23:34:41Z

Content type: article

Language: en

Sources: [Stories by Headspace on Medium](<https://devfeed.tech/sources/stories-by-headspace-on-medium.md>)

Topics: [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [data](<https://devfeed.tech/topics/data.md>), [Machine Learning & Artificial Intelligence](<https://devfeed.tech/topics/machine-learning-artificial-intelligence.md>)

Tags: [a-b-testing](<https://devfeed.tech/tags/a-b-testing.md>), [causal-inference](<https://devfeed.tech/tags/causal-inference.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [experimentation](<https://devfeed.tech/tags/experimentation.md>), [headspace](<https://devfeed.tech/tags/headspace.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [product-experimentation](<https://devfeed.tech/tags/product-experimentation.md>), [recommendation-system](<https://devfeed.tech/tags/recommendation-system.md>)

### AI overview

This article describes how Headspace evaluates its personalized Content Customizer recommendation system with online controlled experiments, or A/B tests. It explains why the full recommendation system should be assessed before production and outlines experiment design, including recommendation surfaces, supported platforms, and audience selection.

### Source excerpt

Author: Rohan Singh Rajput. Rohan is a Senior Data Scientist at Headspace. He combines his passion for Machine Learning with Causal Inference to improve the mindfulness and meditation practices of Headspace users. "If you can't measure it, you can't improve it." -- Peter Drucker. Motivation Content Customizer is Headspace's personalized recommendation system. Content Customizer uses historical data to train its machine learning model and provide personalized recommendations to the users, helping them discover more relevant content. There are various components involved in building a recommendation system, and the ML model is only one of them. Therefore, it is essential to evaluate the effectiveness of the recommendation system as a whole before deploying it to production. Online Controlled Experiments help us to assess our system's impact with statistical evidence. Online Controlled Experiments, a.k.a A/B testing, are the gold standard for estimating causality with high probability. A data-driven decision-making culture helps estimate the measurement's uncertainty to refute the null hypothesis based on experimental data. Furthermore, a random assignment of the users into a control-treatment group allows us to safely ignore the unobserved factors and model the parameters as random variables1. Experiment Design The following components are required to design the experiment. Recommendation Surface Area: We have a total of three surface areas for this experiment: First, in the Today tab, we will use the last three slots to display ML recommendations. Figure 1: Dynamic Playlist on Today's Tab Second is the Hero module, which is the top banner area of the Meditate/Sleep/Focus/Move tabs. Lastly, we will be using the recommended sub-tabs that also have three slots each for ML-powered content. Figure 2: Hero and Recommended Module of other four tabs In total, we have 19 places available for the experiment. The platform for Recommendation: Headspace serves on multiple platform