# experiments Published articles for experiments. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Agent and Model Evaluations in Gemini Enterprise Agent Platform are now GA DevFeed: [Agent and Model Evaluations in Gemini Enterprise Agent Platform are now GA]() Original publisher: [Read original article]() Author: Alex Martin; Dima Melnyk Published: 2026-09-12T11:04:33.891311Z Content type: release Language: en Sources: [Google Developers Blog]() Topics: [ai observability](), [SDKs](), [ci]() Tags: [agent](), [agents](), [ci](), [cli](), [development](), [enterprise](), [evaluation](), [experiments](), [gemini](), [llm](), [metrics](), [model](), [platform](), [production](), [telemetry](), [testing](), [traces]() ### AI overview Gemini Enterprise Agent Platform's evaluation service is generally available. It provides consistent evaluation of agents and models across local experiments and production traffic, with pre-built metrics, adaptive rubrics, custom metrics, simulators, and workflow integrations. ### Source excerpt Agent Platform's evaluation service is now generally available, providing developers with a unified engine to measure agent quality consistently across local development experiments and live production traffic. You can evaluate agents using over 20 pre-built metrics, DeepMind-backed adaptive rubrics, or custom code-based and LLM-as-a-judge metrics stored in a centralized, versioned registry. The service integrates directly into existing workflows via the Agent Platform SDK, agents-cli, and ADK, offering built-in user and environment simulators to automate complex multi-turn testing and streamline CI pipelines. ## Analyze your experiments in ChatGPT with the Datadog Experiments plugin DevFeed: [Analyze your experiments in ChatGPT with the Datadog Experiments plugin]() Original publisher: [Read original article]() Author: Jonathan Fulton; Amy Zhou; Uday Tennety Published: 2026-09-11T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [experiments](), [data](), [Latency]() Tags: [ai](), [analysis](), [chatgpt](), [experimentation](), [experiments](), [openai]() ### AI overview Datadog introduces a ChatGPT Work plugin that lets teams query current experiment results in plain language while retaining Datadog's statistical methods, guardrails, and connected context. ### Source excerpt Learn how the Datadog Experiments OpenAI Data plugin lets your team read, question, and act on experiment results directly in ChatGPT. ## DiscoSign: Discourse-Aware Text to Sign Language Gloss Translation DevFeed: [DiscoSign: Discourse-Aware Text to Sign Language Gloss Translation]() Original publisher: [Read original article]() Published: 2026-09-11T00:00:00Z Content type: article Language: en Sources: [Apple Machine Learning Research]() Topics: [AI, ML & Data Engineering](), [Language models]() Tags: [evaluation](), [experiments](), [framework](), [llm](), [metrics]() ### AI overview DiscoSign is an LLM-based framework for translating text to sign-language gloss while preserving discourse-level coherence. It targets spatial coreference, Question-Answer Clauses, and consistent English-concept-to-ASL-sign mappings, with evaluation metrics for these dimensions. ### Source excerpt Sign language processing systems have traditionally operated at the sentence level, ignoring critical discourse phenomena fundamental to sign language comprehension. We introduce DiscoSign, a computational approach for discourse-aware text to sign language gloss translation grounded in linguistic research. We address three key phenomena within our modular Large Language Model (LLM)-based translation framework: (i) spatial coreference resolution, where entities maintain consistent spatial locations throughout discourse; (ii) Question-Answer Clauses (QACs), pseudocleft structures serving... ## Inside the First Three.js Conference in Paris DevFeed: [Inside the First Three.js Conference in Paris]() Original publisher: [Read original article]() Author: Manoela Ilic Published: 2026-09-10T08:30:11Z Content type: news Language: en Sources: [Codrops]() Topics: [Three.js community](), [webgpu]() Tags: [3d-web-development](), [ai](), [articles](), [browser](), [coding](), [codrops](), [community](), [conference](), [creative-coding](), [creative-development](), [creative-technology](), [creativity](), [creators](), [developers](), [digital-experiences](), [event](), [experiments](), [frontend-development](), [generative-art](), [immersive-web](), [interactive-web-experiences](), [javascript-3d](), [shaders](), [three-js](), [three-js-community](), [three-js-conference](), [three-js-conference-paris](), [three-js-demos](), [three-js-developers](), [three-js-projects](), [three-js-talks](), [three-js-web-development](), [three-js-workshops](), [web-3d](), [web-design](), [webgl](), [webgpu]() ### AI overview A live look at the first Three.js Conference in Paris, covering community moments, opening talks, a Zelda rebuild moving from WebGL to WebGPU, and a discussion about AI and coding. ### Source excerpt A look inside the first-ever Three.js Conference, straight from Paris. ## Creating an AI Platform for classic ML online inference DevFeed: [Creating an AI Platform for classic ML online inference]() Original publisher: [Read original article]() Author: Rohith Leeladharan Published: 2026-09-10T07:26:46Z Content type: tutorial Language: en Sources: [Amex GBT Technology]() Topics: [AI Platform](), [Inference](), [MLOps](), [Machine learning](), [model-deployment]() Tags: [ai](), [ai-platform](), [ai-platform-engineering](), [deploy](), [experiments](), [feature-store](), [inference](), [machine-learning](), [ml](), [mlops](), [monitoring](), [predictions]() ### AI overview This article describes how American Express Global Business Travel built an AI platform for deploying classic machine-learning systems and supporting online inference. It explains the platform's requirements--simplicity, self-service, experimentation, and continuous improvement--and details the pre-process, predict, post-process pattern used by inference engines. ### Source excerpt Introduction In 2021, we were given the mission to have AI Systems running in production. The team, instead of just following a classical MLOps process, that involves transforming a Jupyter notebook into a product running in production, decided to go further by creating a platform to deploy AI systems in production. The team decided the platform should respect these requirements: Simplicity: The code powering AI systems should be simple, readable, and easy to maintain -- less intricacy means fewer bugs in production and greater reliability. Self-service: Anyone should be able to build and deploy AI systems autonomously, without depending on a central team. Experimentation: The platform should make it easy to run and iterate on experiments. Continuous improvement: Data related to events and interactions within AI systems must be captured, enabling monitoring and continuous improvement over time. In this article, we will walk through the work done to build a platform that fulfills these four requirements. Background At American Express Global Business Travel, we use machine learning (ML) models for a variety of user experiences like ranking hotel and flight search results. Our ML models are wrapped in inference engines that handle both pre-processing of input data before we run a prediction with the model, and post-processing of output data before returning the output to the caller. The overall flow looks something like this: Figure 1: Handling an inference request A client service that would like the ML model's predictions provides necessary context about the request like which user the request is for. Then, optionally, the inference engine fetches any necessary features for inference from our feature store [part 1][part 2]. Finally, it pre-processes the data, runs the predictions using the trained ML model, and does any necessary post-processing of the model output before returning the response to the caller. We call this the pre-process, predict, post-process patter ## How we built Datadog Experiments DevFeed: [How we built Datadog Experiments]() Original publisher: [Read original article]() Author: Chas DeVeas; Aaron Silverman; Tyler Buffington; Jonathan Fulton; Taylor Overturf Published: 2026-09-10T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [experiments](), [real user monitoring]() Tags: [a-b-testing](), [acquisition](), [data-analytics](), [data-engineering](), [experimentation](), [experiments](), [metrics](), [observability](), [product-analytics](), [real-user-monitoring]() ### AI overview Datadog describes rebuilding its experimentation platform to speed up confident A/B-test decisions. The article explains a flexible CUPED approach that reduces metric variance and can be applied to segments. ### Source excerpt Datadog Experiments shortens the time from result to decision with CUPED on percentiles, verifiable warehouse results, and near real-time RUM metrics. ## Testing application resilience with Amazon SQS and AWS Fault Injection Service DevFeed: [Testing application resilience with Amazon SQS and AWS Fault Injection Service]() Original publisher: [Read original article]() Author: Richard Whitworth Published: 2026-09-09T21:33:59Z Content type: tutorial Language: en Sources: [AWS Architecture Blog]() Topics: [Amazon Simple Queue Service (SQS)](), [AWS Fault Injection Service (FIS)](), [Chaos Engineering](), [AWS IAM](), [AWS Identity and Access Management (IAM)]() Tags: [advanced-300](), [amazon-cloudwatch](), [amazon-simple-queue-service-sqs](), [amazon-sqs](), [automation](), [aws](), [aws-fault-injection-service-fis](), [experiments](), [how-to](), [iam](), [observability](), [resilience](), [technical-how-to](), [testing]() ### AI overview A tutorial on testing application resilience when Amazon SQS data-plane operations fail. It uses AWS Fault Injection Service and Systems Manager Automation to progressively deny queue access, evaluate recovery and observability with CloudWatch metrics, and avoid IAM deny-policy lockouts. ### Source excerpt Learn how to use AWS Fault Injection Service and AWS Systems Manager Automation to run progressive chaos experiments against Amazon SQS queues. Validate that your retry logic, circuit breakers, and dead-letter queues actually work under failure before a real outage hits production. ## How we built data-driven AI Golden Paths at Datadog DevFeed: [How we built data-driven AI Golden Paths at Datadog]() Original publisher: [Read original article]() Author: Addie Beach; Rui Martins Lacerda Published: 2026-09-09T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [AI Bots]() Tags: [agent](), [agent-observability](), [ai](), [cli](), [cost](), [dashboards](), [development](), [documentation](), [experiments](), [performance](), [security](), [skills](), [workflows]() ### AI overview Datadog describes a data-driven process for establishing AI Golden Paths: standardized AI-agent workflows governed by controls such as skills, hooks, and tests. The approach evaluates controls for their effects on code quality, security, token usage, cost, and agent performance. ### Source excerpt See how a Datadog guild achieved 13% faster agent runs by building Golden Paths for AI-assisted development using controls, experiments, and dashboards. ## How GPT-5.6 Sol helps run quantum computing experiments DevFeed: [How GPT-5.6 Sol helps run quantum computing experiments]() Original publisher: [Read original article]() Published: 2026-09-08T17:00:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [AI research agents]() Tags: [agents](), [ai](), [ai-agents](), [analysis](), [applied-ai](), [codex](), [experiments](), [gpt](), [quantum](), [quantum-computing]() ### AI overview An MIT researcher connected GPT-5.6 Sol through Codex to laboratory software for superconducting-qubit experiments. The system ran routine measurements, analyzed results, and selected next steps, reducing the need for constant supervision during calibration workflows. ### Source excerpt See how an MIT researcher uses GPT-5.6 Sol with Codex to autonomously run quantum computing experiments, analyze results, and calibrate qubits. ## Safety for Whom? Refusing the Right Subset of a Topic, Not the Whole Topic DevFeed: [Safety for Whom? Refusing the Right Subset of a Topic, Not the Whole Topic]() Original publisher: [Read original article]() Author: Antonio Tiene; Alejo Lopez Avila; Iker García-Ferrero Published: 2026-09-08T14:23:07Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [AI Chat]() Tags: [blog](), [deployment](), [experiments](), [hugging-face](), [llm](), [model](), [policy](), [safety](), [training]() ### AI overview The article examines LLM safety policies that refuse harmful requests within a topic while continuing to answer benign requests in that same topic. ### Source excerpt A Blog post by Multiverse Computing on Hugging Face ## Coordinate product launches with Datadog DevFeed: [Coordinate product launches with Datadog]() Original publisher: [Read original article]() Author: Milene Darnis; Adam Virani Published: 2026-09-08T00:00:00Z Content type: tutorial Language: en Sources: [Datadog | The Monitor blog]() Topics: [Instrumentation](), [experiments]() Tags: [ai](), [bits-ai](), [digital-experience-monitoring](), [experimentation](), [experiments](), [feature-flags](), [instrumentation](), [launch](), [product-analytics](), [session-replay]() ### AI overview A tutorial on using Datadog Product Analytics Launches to plan product releases, define measurement questions, create tracking plans, and identify missing events and properties before rollout. ### Source excerpt Learn how to turn a product brief and feature flag into a connected launch workflow for instrumentation, experimentation, QA, and reporting. ## Research acceleration: The view inside OpenAI DevFeed: [Research acceleration: The view inside OpenAI]() Original publisher: [Read original article]() Published: 2026-09-06T08:00:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [AI Research](), [AI-assisted coding](), [coding]() Tags: [agents](), [ai-research](), [coding](), [deep-learning](), [experiments](), [frontier-ai](), [openai](), [research]() ### AI overview OpenAI describes how coding agents are being used throughout its AI research workflow, with reported increases in code contribution, experiment execution, task complexity, and success rates. It frames this progress as a step toward supervised automated AI research while emphasizing human control over research priorities and deployment decisions. ### Source excerpt Inside OpenAI, coding agents are reshaping AI research. Explore early data on agent usage, experiment velocity, task complexity, and research acceleration. ## How we make AI coding more cost efficient without sacrificing task quality DevFeed: [How we make AI coding more cost efficient without sacrificing task quality]() Original publisher: [Read original article]() Author: Erik Kristensen Published: 2026-09-02T18:00:00Z Content type: article Language: en Sources: [GitHub](), [GitHub Engineering]() Topics: [GitHub Copilot](), [AI-assisted coding](), [GitHub Copilot CLI](), [coding](), [GitHub Copilot app](), [GitHub Copilot code review]() Tags: [agentic-ai](), [agentic-coding](), [ai](), [ai-coding](), [ai-ml](), [architecture-optimization](), [benchmarks](), [compression](), [cost](), [engineering](), [experiments](), [github-copilot](), [github-copilot-cli](), [github-copilot-code-review](), [llms](), [prompt-engineering]() ### AI overview GitHub Copilot's efficiency work focuses on total task cost and duration rather than minimizing tokens in individual tool responses. The article describes evaluating changes with coding benchmarks and controlled experiments, and explains that overly compressed output can cause agents to repeat work. ### Source excerpt Why shorter outputs can cost more, and how GitHub Copilot reduces wasted work across the complete coding task. The post How we make AI coding more cost efficient without sacrificing task quality appeared first on The GitHub Blog. ## From traces to experiments: A loop for improving AI agents DevFeed: [From traces to experiments: A loop for improving AI agents]() Original publisher: [Read original article]() Author: Adam Virani; Lukas Goetz-Weiss; Natasha Silva Published: 2026-09-01T00:00:00Z Content type: tutorial Language: en Sources: [Datadog | The Monitor blog]() Topics: [ai observability](), [AI Bots]() Tags: [agent-observability](), [agents](), [ai](), [ai-agents](), [ai-observability](), [evaluation](), [experiments](), [latency](), [optimization](), [production](), [telemetry](), [traces]() ### AI overview The article explains how teams can use AI-agent trace data, evaluations, and production experiments to identify performance issues and test whether changes improve outcomes. ### Source excerpt Learn how to read AI agent traces as a roadmap and how to run production experiments that measure whether improvements hold in production. ## Visualize how CUPED adjusts experiment results with Datadog DevFeed: [Visualize how CUPED adjusts experiment results with Datadog]() Original publisher: [Read original article]() Author: Tyler Buffington; Lukas Goetz-Weiss; Ryan Lucht Published: 2026-09-01T00:00:00Z Content type: tutorial Language: en Sources: [Datadog | The Monitor blog]() Topics: [experiments]() Tags: [analysis](), [digital-experience-monitoring](), [experiment](), [experiments](), [how-to](), [metrics](), [product-analytics](), [visualization]() ### AI overview This tutorial explains Datadog Experiments' CUPED adjustments visualization, which breaks the difference between raw and CUPED-adjusted experiment lift into individual covariate contributions. ### Source excerpt Learn how Datadog visualizes CUPED adjustments so you can trace which covariates change experiment lift estimates and improve precision. ## Harness RT Agents Detect Resilience Risks and Generate Tests for CD Pipelines and Kubernetes Workloads DevFeed: [Harness RT Agents Detect Resilience Risks and Generate Tests for CD Pipelines and Kubernetes Workloads]() Original publisher: [Read original article]() Author: Uma Mukkara Published: 2026-08-24T00:00:00Z Content type: release Language: en Sources: [Harness Blog]() Topics: [Resilience](), [Testing](), [Continuous Delivery (CD)](), [Kubernetes](), [dashboards]() Tags: [agentic](), [agents](), [blog](), [chaos](), [configuration](), [continuous-delivery](), [dashboards](), [deployment](), [experiments](), [infrastructure](), [insights](), [kubernetes](), [load](), [pipelines](), [product](), [resilience](), [services](), [teams](), [testing](), [tests]() ### AI overview Harness announces an update to Resilience Testing called RT Agents. The agents analyze CD pipelines and Kubernetes workloads for resilience risks, recommend the testing needed to confirm those risks, and can generate and run chaos experiments or load tests and interpret the results. ### Source excerpt RT Agents detect resilience risk in your CD pipelines and Kubernetes workloads, then generate and run chaos experiments or load tests to confirm it. | Blog ## Two ways to measure the cumulative impact of experiments DevFeed: [Two ways to measure the cumulative impact of experiments]() Original publisher: [Read original article]() Author: Lukas Goetz-Weiss; Eddie Cai Published: 2026-08-18T00:00:00Z Content type: opinion Language: en Sources: [Datadog | The Monitor blog]() Topics: [experiments](), [Statistics](), [Data Science]() Tags: [analysis](), [experimentation](), [experiments](), [product](), [statistics](), [testing]() ### AI overview This article explains why summing the observed lift from winning experiments overstates cumulative impact because of the winner's curse. It presents two established approaches: a randomized holdout that measures the combined effect directly, and a statistical correction that aggregates existing experiment estimates. Datadog's Cumulative Impact feature applies the correction without requiring a quarter-long holdout and can analyze an entire experimentation program or a filtered team subset. ### Source excerpt Summing individual wins overstates true impact. See two accurate methods, holdouts and Datadog's Cumulative Impact, and how to choose between them. ## LaunchDarkly is now available on the Vercel Marketplace DevFeed: [LaunchDarkly is now available on the Vercel Marketplace]() Original publisher: [Read original article]() Author: Sam Halstead Published: 2026-08-11T00:00:00Z Content type: release Language: en Sources: [Vercel News]() Topics: [feature flags](), [Vercel](), [experiments](), [SDKs]() Tags: [agent](), [experiments](), [feature-flags](), [sdk](), [vercel]() ### AI overview LaunchDarkly is now available through the Vercel Marketplace, providing feature flags with local evaluation, release targeting, experiments, metrics, rollback controls, and Vercel Toolbar management. ### Source excerpt LaunchDarkly is now available on the Vercel Marketplace, allowing you to quickly get started with feature flags without additional setup. You can: Sync flags into Global Config and evaluate them locally Target releases by user, attribute, or segment Run experiments with metrics, and roll back with a kill switch View and override flags from the Vercel Toolbar with the Flags Explorer To get started, run vercel install launchdarkly, add the @flags-sdk/launchdarkly adapter, and declare a flag with the Flags SDK: Using a coding agent? Hand it this prompt: Add LaunchDarkly from the Vercel Marketplace, or read the adapter docs. Read more ## Tailscale's Remy Guercio on what comes after token maxing DevFeed: [Tailscale's Remy Guercio on what comes after token maxing]() Original publisher: [Read original article]() Author: WorkOS Published: 2026-08-05T23:49:35Z Content type: article Language: en Sources: [WorkOS Blog]() Topics: [Artificial Intelligence](), [AI-assisted coding](), [experiments]() Tags: [agents](), [ai](), [ai-gateway](), [analysis](), [cost](), [experiments](), [model](), [tailscale](), [token]() ### AI overview An interview with Tailscale's Remy Guercio examines the shift from maximizing token usage to maximizing return on investment in AI development. It covers coding-agent adoption, rising bills across multiple AI tools, cost per task, model experimentation, and the visibility an AI gateway can provide. ### Source excerpt Tailscale's Remy Guercio tells Michael Grinich why the next 12 months of AI are ROI maxing: cost per task, model experiments, and what a gateway can see. ## How Speechify serves 500,000 dynamic pages to 60 million users on Vercel DevFeed: [How Speechify serves 500,000 dynamic pages to 60 million users on Vercel]() Original publisher: [Read original article]() Author: Susan Aziz Published: 2026-07-15T04:00:00Z Content type: article Language: en Sources: [Vercel News]() Topics: [Vercel](), [Caching](), [Next.js](), [Cache](), [Security](), [incident]() Tags: [cache](), [experiments](), [incident](), [infrastructure](), [next-js](), [scale](), [security](), [vercel]() ### AI overview A case study of how Speechify rebuilt its infrastructure on Next.js and Vercel to serve more than 500,000 dynamic pages across 40-plus languages to 60 million users. It describes caching, auto-scaling, cost reduction, uptime, and Instant Rollbacks for safer frequent deployments. ### Source excerpt Speechify on Vercel 500,000+ pages served across 40+ languages Cut costs 50% by auto-scaling with Fluid compute Zero user impact on bad deploys with Instant Rollbacks Speechify started as a tool for people with dyslexia. Cliff Weitzman, Founder & CEO, built it because reading was challenging and audio made it much easier. That initial use case led Speechify to tens of millions of users and an Apple Design Award for inclusion. The product has since grown into something much larger: an AI work platform where 60 million people listen to documents, delegate tasks to agents, and complete work entirely through voice. But the journey wasn't easy. Early on, they got hacked, and for half a day, every visitor was redirected to a casino website. After that, Denis Chernobai, Head of Growth Engineering, knew it was time to re-evaluate their infrastructure stack. He decided to rebuild from scratch on Next.js and Vercel, and they ended up serving 40 times more pages, reaching a global audience three times larger, and achieving a 50% cost reduction. Since migrating to Vercel, Speechify has maintained 99.99% uptime and hasn't had a single security incident. The cost of serving 500,000 dynamic pages Speechify's website is its primary growth engine, with 10,000 base pages translated into 40+ languages across onboarding funnels, localized landing pages, and pricing experiments that change constantly. Static generation isn't an option because the content changes too frequently. But serving everything dynamically means every visitor is a potential database read, which compounds fast at hundreds of thousands of visits every day. Vercel's Data Cache, ISR, and Next.js Cache Components resolve both problems at once. Pages render dynamically on first visit, cache immediately, and serve from the closest point of presence until something changes. The result is a global growth engine that scales without the infrastructure costs scaling with it. The risk of shipping to 60 million usersInstant Rol ## GLM-5.2: Considerations for enterprise teams starting out with open-weight models DevFeed: [GLM-5.2: Considerations for enterprise teams starting out with open-weight models]() Original publisher: [Read original article]() Author: Robat Williams Published: 2026-07-08T10:00:00Z Content type: article Language: en Sources: [Scott Logic]() Topics: [Agentic development](), [developer setup](), [AI Development](), [experiments](), [Benchmark](), [Open Source](), [Claude]() Tags: [agent](), [agentic](), [agentic-development](), [artificial-intelligence](), [benchmark](), [claude](), [developer-setup](), [experiments](), [open-source]() ### AI overview The article describes how a small team prepared an open-weight agentic development setup for AI productivity experiments. It explains the team's model-selection criteria, distinguishes open-weight models from locally run models, and reports choosing GLM-5.2 after comparing DeepSeek V4, Kimi K-2.6, and other models using benchmarks and practical trials. ### Source excerpt Setting yourself up to try out open-weight models for agentic development isn't difficult, but it isn't as straightforward as downloading a coding agent from one of the handful of well-known AI vendors. In preparation for the latest round of our AI productivity experiments, we've recently been through this process. Read on for the choices we made, the considerations at play, and what made our situation unusual. ## The effect distribution: The missing piece in experimentation programs DevFeed: [The effect distribution: The missing piece in experimentation programs]() Original publisher: [Read original article]() Author: Tyler Buffington Published: 2026-07-02T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [experiments]() Tags: [a-b-testing](), [experiment](), [experimentation](), [experiments](), [testing]() ### AI overview The article explains why effect distributions are essential for interpreting results across experimentation programs. It shows how statistically significant results can all be false positives when true effects are concentrated at zero, and introduces the challenge of estimating true effects from noisy observed effects. ### Source excerpt Learn about the importance of considering the effect distribution when running experiments. ## Variance Reduction Below the Randomization Grain DevFeed: [Variance Reduction Below the Randomization Grain]() Original publisher: [Read original article]() Author: Tilman Drerup Published: 2026-07-01T16:28:36Z Content type: article Language: en Sources: [Instacart]() Topics: [experiments]() Tags: [capacity](), [causal-inference](), [economics](), [engineering](), [experiment](), [experimentation](), [experiments](), [machine-learning](), [marketplaces](), [metrics](), [reduce](), [science](), [techniques](), [variance]() ### AI overview This article explains how marketplace experiments can reduce metric variance below the level at which treatment is randomized. It describes cluster-level randomization for containing interference and shows how fine-grained outcome predictability can improve statistical power and reduce experimentation time. ### Source excerpt Sergio Camelo, Caitlin Kearns, Matias Cersosimo, and Tilman Drerup As artificial intelligence increases the velocity of engineering and science teams, experimental throughput is set to become a bottleneck for many product decisions. Many companies can now build faster than they can experiment, with queues of good ideas running the risk of not being tested because of lack of experimental capacity. This problem is particularly severe in marketplaces, where the presence of spillover and cannibalization effects between experimental units requires cluster-level randomization techniques. That randomization, in turn, has the unfortunate tendency to substantially reduce statistical power and slow down experimentation. In this post, we show that the predictability of outcomes at fine grains can be exploited to reduce the variance of aggregate metrics, even when experiments themselves are run at a coarse level. Since statistical power depends on metric variability, this yields considerable reductions in experimentation time. The Interference Problem In marketplace settings, behavior and outcomes for individual participants are inherently intertwined. In a delivery marketplace like Instacart, for example, the dispatch system solves a bipartite matching problem between shoppers and customer orders. Since assignments are global and interdependent, matching an order to one shopper means that the same order cannot be matched to another shopper. As a result, changing the handling for a single order creates ripples that affect the orders around it. If an experimenter were to assign a treatment intervention to one of these orders while leaving neighboring orders as controls, the latter would evidently be contaminated. A common response to this problem is to randomize treatments at the level of a cluster, chosen so that interference can stay within it. In food and grocery delivery, that cluster is typically a geographical region. Since every order within a region sees the same treatme ## Outlier Handling at Scale in Experimentation DevFeed: [Outlier Handling at Scale in Experimentation]() Original publisher: [Read original article]() Author: Margarida Moreira da Silva Published: 2026-07-01T13:44:26Z Content type: article Language: en Sources: [Booking.com Data Science]() Topics: [experiments](), [data](), [Simulation](), [Ground truth / benchmark quality](), [plotting]() Tags: [data](), [experimentation](), [experiments](), [false-positive](), [outlier-detection](), [simulation](), [variance]() ### AI overview The article examines how extreme values affect experimentation at Booking.com. It describes permutation tests and simulated A/A experiments for diagnosing distorted p-value distributions, and reports that increasing outlier magnitude and frequency can cause test failures. ### Source excerpt At Booking.com, thousands of experiments run simultaneously across highly heterogeneous users, from individual travellers to large travel agencies. This means our experiment data regularly contains legitimate but extreme values. When these go unhandled, they distort the statistical conclusions we draw, leading us to scale ideas that don't create value, or to discard ones that do. So, we need outlier handling methods that are reliable, automated, and applicable across diverse metrics without manual intervention. The Problem When extreme values are present in experiment data, they can compromise the estimation of average treatment effects (ATE), leading to unreliable test results and reduced statistical power. Even a single observation can inflate variance enough to mask a real effect or produce a spurious one. In practice, this means we risk shipping changes that appear positive but are not, or killing promising features because noise masked their real effect. At Booking.com's scale, this increase in false conclusions quickly compounds into a meaningful impact on customer experience and business outcomes. A Diagnostic Tool: the Permutation Test One way to assess whether extreme values are distorting results is the permutation test. By permuting over experiment data, we generate hundreds of simulated AA experiments where we know the ground truth: there is no real effect. Plotting the resulting p-values, we expect a uniform distribution. If it instead looks skewed, the underlying data distribution is compromising the validity of results. Plot 1: P-value distributions from simulated A/A tests. Clean normally-distributed estimated effects produce a uniform distribution (left), while the presence of extreme outliers results in skewed p-values (right), indicating a distorted false positive rate.Simulation Evidence: What Drives Failure? We ran AA permutation tests across a range of simulated data distributions to understand when they fail (i.e. not show a uniform p-value di [Next page]()