# Accelerating Online Experiments that Target Quantile Treatment Effects DevFeed: [Accelerating Online Experiments that Target Quantile Treatment Effects]() Original publisher: [Read original article]() Author: Zach Flynn Published: 2023-08-31T14:27:25Z Content type: tutorial Language: en Sources: [Udemy Engineering]() Topics: [experiments]() Tags: [analysis](), [automated](), [data-science](), [experiment](), [experimentation](), [experiments](), [metric](), [platform](), [precision](), [statistics](), [tests]() ## AI overview An Udemy engineering article explains how variance-reduction methods such as CUPED accelerate A/B experiments targeting average treatment effects, while introducing the challenge of applying variance reduction to quantile treatment effects such as the median or 90th percentile. It describes the motivation for keeping the analysis practical enough for automated experiment analysis, but the supplied text ends before the proposed quantile approach is presented. ## Source excerpt At Udemy, we run hundreds of A/B-style experiments per year on our experimentation platform. One of the biggest challenges we've faced is the need to run experiments faster to enable our product and engineering teams to rapidly ship more new product innovations to our customers. One way we've accelerated experimentation at Udemy is by building variance reduction methods -- such as CUPED -- into our experimentation platform. Doing so has had a powerful impact, helping us reduce the average time to run some product A/B tests by 30% or more. However, variance reduction methods like CUPED are designed to work for experiments that target mean or average treatment effects. They are not designed to work with quantile treatment effects, such as the median or 90th percentile impact of an experiment. In this short blog, we show how we've also reduced the variance of quantile treatment effects at Udemy, while still keeping the analysis practical and simple enough to build into our automated experiment analysis system. Variance Reduction for Average Treatment Effects Control function methods, like CUPED, are a type of variance reduction method used to increase the precision of average treatment effect estimates. They control for differences between the experimental units (e.g., site visitors, accounts) that could not have been the result of the change the experiment introduces. Removing this irrelevant variation in the outcome metric reduces noise in estimating the treatment effect. Usually, these methods control for features that existed prior to the experiment being launched, such as total minutes a user spent learning in the 30 days prior to entering the experiment. Clearly, this feature is unrelated to the experiment because it predates the unit's exposure to the experiment. But it likely explains a good portion of the variation in post-treatment minutes spent learning because, even in the absence of the experiment, people who learned more last month are likely to learn more