# Variance Reduction Below the Randomization Grain

DevFeed: [Variance Reduction Below the Randomization Grain](<https://devfeed.tech/articles/variance-reduction-below-the-randomization-grain-20111.md>)

Original publisher: [Read original article](<https://tech.instacart.com/variance-reduction-below-the-randomization-grain-31719f87a7d2?source=rss----587883b5d2ee---4>)

Author: Tilman Drerup

Published: 2026-07-01T16:28:36Z

Content type: article

Language: en

Sources: [Instacart](<https://devfeed.tech/sources/instacart.md>)

Topics: [experiments](<https://devfeed.tech/topics/experiments.md>)

Tags: [capacity](<https://devfeed.tech/tags/capacity.md>), [causal-inference](<https://devfeed.tech/tags/causal-inference.md>), [economics](<https://devfeed.tech/tags/economics.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [experimentation](<https://devfeed.tech/tags/experimentation.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [marketplaces](<https://devfeed.tech/tags/marketplaces.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [reduce](<https://devfeed.tech/tags/reduce.md>), [science](<https://devfeed.tech/tags/science.md>), [techniques](<https://devfeed.tech/tags/techniques.md>), [variance](<https://devfeed.tech/tags/variance.md>)

## AI overview

This article explains how marketplace experiments can reduce metric variance below the level at which treatment is randomized. It describes cluster-level randomization for containing interference and shows how fine-grained outcome predictability can improve statistical power and reduce experimentation time.

## Source excerpt

Sergio Camelo, Caitlin Kearns, Matias Cersosimo, and Tilman Drerup As artificial intelligence increases the velocity of engineering and science teams, experimental throughput is set to become a bottleneck for many product decisions. Many companies can now build faster than they can experiment, with queues of good ideas running the risk of not being tested because of lack of experimental capacity. This problem is particularly severe in marketplaces, where the presence of spillover and cannibalization effects between experimental units requires cluster-level randomization techniques. That randomization, in turn, has the unfortunate tendency to substantially reduce statistical power and slow down experimentation. In this post, we show that the predictability of outcomes at fine grains can be exploited to reduce the variance of aggregate metrics, even when experiments themselves are run at a coarse level. Since statistical power depends on metric variability, this yields considerable reductions in experimentation time. The Interference Problem In marketplace settings, behavior and outcomes for individual participants are inherently intertwined. In a delivery marketplace like Instacart, for example, the dispatch system solves a bipartite matching problem between shoppers and customer orders. Since assignments are global and interdependent, matching an order to one shopper means that the same order cannot be matched to another shopper. As a result, changing the handling for a single order creates ripples that affect the orders around it. If an experimenter were to assign a treatment intervention to one of these orders while leaving neighboring orders as controls, the latter would evidently be contaminated. A common response to this problem is to randomize treatments at the level of a cluster, chosen so that interference can stay within it. In food and grocery delivery, that cluster is typically a geographical region. Since every order within a region sees the same treatme