# Outlier Handling at Scale in Experimentation

DevFeed: [Outlier Handling at Scale in Experimentation](<https://devfeed.tech/articles/outlier-handling-at-scale-in-experimentation-30453.md>)

Original publisher: [Read original article](<https://booking.ai/outlier-handling-at-scale-in-experimentation-a8bb140e1ab8?source=rss----4d265f07defc---4>)

Author: Margarida Moreira da Silva

Published: 2026-07-01T13:44:26Z

Content type: article

Language: en

Sources: [Booking.com Data Science](<https://devfeed.tech/sources/booking-com-data-science.md>)

Topics: [experiments](<https://devfeed.tech/topics/experiments.md>), [data](<https://devfeed.tech/topics/data.md>), [Simulation](<https://devfeed.tech/topics/simulation.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [plotting](<https://devfeed.tech/topics/plotting.md>)

Tags: [data](<https://devfeed.tech/tags/data.md>), [experimentation](<https://devfeed.tech/tags/experimentation.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [false-positive](<https://devfeed.tech/tags/false-positive.md>), [outlier-detection](<https://devfeed.tech/tags/outlier-detection.md>), [simulation](<https://devfeed.tech/tags/simulation.md>), [variance](<https://devfeed.tech/tags/variance.md>)

## AI overview

The article examines how extreme values affect experimentation at Booking.com. It describes permutation tests and simulated A/A experiments for diagnosing distorted p-value distributions, and reports that increasing outlier magnitude and frequency can cause test failures.

## Source excerpt

At Booking.com, thousands of experiments run simultaneously across highly heterogeneous users, from individual travellers to large travel agencies. This means our experiment data regularly contains legitimate but extreme values. When these go unhandled, they distort the statistical conclusions we draw, leading us to scale ideas that don't create value, or to discard ones that do. So, we need outlier handling methods that are reliable, automated, and applicable across diverse metrics without manual intervention. The Problem When extreme values are present in experiment data, they can compromise the estimation of average treatment effects (ATE), leading to unreliable test results and reduced statistical power. Even a single observation can inflate variance enough to mask a real effect or produce a spurious one. In practice, this means we risk shipping changes that appear positive but are not, or killing promising features because noise masked their real effect. At Booking.com's scale, this increase in false conclusions quickly compounds into a meaningful impact on customer experience and business outcomes. A Diagnostic Tool: the Permutation Test One way to assess whether extreme values are distorting results is the permutation test. By permuting over experiment data, we generate hundreds of simulated AA experiments where we know the ground truth: there is no real effect. Plotting the resulting p-values, we expect a uniform distribution. If it instead looks skewed, the underlying data distribution is compromising the validity of results. Plot 1: P-value distributions from simulated A/A tests. Clean normally-distributed estimated effects produce a uniform distribution (left), while the presence of extreme outliers results in skewed p-values (right), indicating a distorted false positive rate.Simulation Evidence: What Drives Failure? We ran AA permutation tests across a range of simulated data distributions to understand when they fail (i.e. not show a uniform p-value di