# experimentation Published articles for experimentation. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## How to Implement Feature Flags for Safe and Gradual Rollouts DevFeed: [How to Implement Feature Flags for Safe and Gradual Rollouts]() Original publisher: [Read original article]() Author: Abisoye Alli-Balogun Published: 2026-09-14T14:58:55Z Content type: article Language: en Sources: [freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More]() Topics: [feature flags](), [CI/CD](), [Deployment](), [Development](), [Tech Debt](), [Testing]() Tags: [architecture](), [best-practices](), [ci-cd](), [deployment](), [development](), [experimentation](), [feature-flags](), [how-to](), [tech-debt](), [testing]() ### AI overview This article explains how to implement feature flags for safe, gradual rollouts. It covers separating deployment from release, boolean toggles, percentage-based rollouts, user segmentation, experimentation, kill switches, lifecycle management, and common anti-patterns. ### Source excerpt Feature flags are one of the most powerful tools in a team's deployment arsenal. They decouple deployment from release, meaning your CI/CD pipeline can push code to production servers on every merge, ## Analyze your experiments in ChatGPT with the Datadog Experiments plugin DevFeed: [Analyze your experiments in ChatGPT with the Datadog Experiments plugin]() Original publisher: [Read original article]() Author: Jonathan Fulton; Amy Zhou; Uday Tennety Published: 2026-09-11T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [experiments](), [data](), [Latency]() Tags: [ai](), [analysis](), [chatgpt](), [experimentation](), [experiments](), [openai]() ### AI overview Datadog introduces a ChatGPT Work plugin that lets teams query current experiment results in plain language while retaining Datadog's statistical methods, guardrails, and connected context. ### Source excerpt Learn how the Datadog Experiments OpenAI Data plugin lets your team read, question, and act on experiment results directly in ChatGPT. ## How we built Datadog Experiments DevFeed: [How we built Datadog Experiments]() Original publisher: [Read original article]() Author: Chas DeVeas; Aaron Silverman; Tyler Buffington; Jonathan Fulton; Taylor Overturf Published: 2026-09-10T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [experiments](), [real user monitoring]() Tags: [a-b-testing](), [acquisition](), [data-analytics](), [data-engineering](), [experimentation](), [experiments](), [metrics](), [observability](), [product-analytics](), [real-user-monitoring]() ### AI overview Datadog describes rebuilding its experimentation platform to speed up confident A/B-test decisions. The article explains a flexible CUPED approach that reduces metric variance and can be applied to segments. ### Source excerpt Datadog Experiments shortens the time from result to decision with CUPED on percentiles, verifiable warehouse results, and near real-time RUM metrics. ## Why Spotify Is Not Using Bayesian A/B Testing DevFeed: [Why Spotify Is Not Using Bayesian A/B Testing]() Original publisher: [Read original article]() Author: Spotify Engineering Published: 2026-09-08T13:18:44Z Content type: article Language: en Sources: [Spotify Engineering](), [Spotify Engineering Blog]() Topics: [Data Science](), [Tooling]() Tags: [a-b-testing](), [data](), [data-science](), [experimentation](), [inference](), [statistics](), [testing](), [tooling]() ### AI overview Spotify argues that Bayesian A/B testing can be oversimplified and that its claimed advantages require careful configuration. The article compares Bayesian and frequentist approaches and explains why Spotify does not currently need Bayesian inference alongside its frequentist tooling. ### Source excerpt Clearing the confusion about what Bayesian A/B testing is. The post Why Spotify Is Not Using Bayesian A/B Testing appeared first on Spotify Engineering. ## Coordinate product launches with Datadog DevFeed: [Coordinate product launches with Datadog]() Original publisher: [Read original article]() Author: Milene Darnis; Adam Virani Published: 2026-09-08T00:00:00Z Content type: tutorial Language: en Sources: [Datadog | The Monitor blog]() Topics: [Instrumentation](), [experiments]() Tags: [ai](), [bits-ai](), [digital-experience-monitoring](), [experimentation](), [experiments](), [feature-flags](), [instrumentation](), [launch](), [product-analytics](), [session-replay]() ### AI overview A tutorial on using Datadog Product Analytics Launches to plan product releases, define measurement questions, create tracking plans, and identify missing events and properties before rollout. ### Source excerpt Learn how to turn a product brief and feature flag into a connected launch workflow for instrumentation, experimentation, QA, and reporting. ## How we built a benchmarking framework to horizontally accelerate transaction model research DevFeed: [How we built a benchmarking framework to horizontally accelerate transaction model research]() Original publisher: [Read original article]() Author: Nubank Editorial Published: 2026-09-03T13:53:30Z Content type: article Language: en Sources: [Nubank]() Topics: [benchmarking](), [Framework](), [Transformer](), [data]() Tags: [architectures](), [automated](), [benchmarking](), [data](), [data-science-machine-learning](), [experimentation](), [framework](), [model](), [performance](), [pipelines](), [training]() ### AI overview Nubank built an automated benchmarking framework for horizontally evaluating transformer-based transaction representation models across multiple downstream tasks and trials. The framework made experimentation reproducible and statistically rigorous, helping the team identify improvements that generalize across applications. It increased the team's capacity to run experiments by roughly five times per month while reducing operational overhead. ### Source excerpt The framework that transformed weeks of manual experimentation into automated pipelines for horizontal transaction model research The post How we built a benchmarking framework to horizontally accelerate transaction model research appeared first on Building Nubank. ## How AI-native companies turn workflows into operating capability DevFeed: [How AI-native companies turn workflows into operating capability]() Original publisher: [Read original article]() Published: 2026-09-01T17:00:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [Artificial Intelligence]() Tags: [account-management](), [agents](), [ai](), [ai-adoption](), [ai-agents](), [codex](), [developer](), [experimentation](), [integration](), [onboarding](), [tools](), [workflows]() ### AI overview The article explains how Basis, Clay, and Exa Labs embed AI agents into onboarding, account management, and developer integrations. It presents a progression from teaching agents stable processes to giving them persistent context and enabling tested action, with emphasis on repeatability, measurement, trust, and continuous improvement. ### Source excerpt Basis, Clay, and Exa Labs use AI agents to improve onboarding, account management, and developer integrations. See what enterprise leaders can apply. ## PyTorch vs. TensorFlow: Differences, Performance, and How to Choose DevFeed: [PyTorch vs. TensorFlow: Differences, Performance, and How to Choose]() Original publisher: [Read original article]() Author: NICOLAS PIRO, DATA SCIENTIST AND AI DEVELOPER @ TOPTAL Published: 2026-08-27T04:00:00Z Content type: comparison Language: en Sources: [Toptal Blog]() Topics: [Training AI Models]() Tags: [deep-learning](), [deployment](), [experimentation](), [infrastructure](), [performance](), [pytorch](), [tensorflow](), [tooling]() ### AI overview A comparison of PyTorch and TensorFlow for deep-learning experimentation, model design, production workflows, tooling, and infrastructure considerations. ### Source excerpt This comprehensive guide explores how PyTorch and TensorFlow shape deep-learning work in 2026, from experimentation and model design to production workflows, ecosystem tooling, and infrastructure considerations. ## Responsible AI adoption needs developer workflow design DevFeed: [Responsible AI adoption needs developer workflow design]() Original publisher: [Read original article]() Author: Dr. Gleb Tsipursky Published: 2026-08-24T14:00:00Z Content type: article Language: en Sources: [Stack Overflow Blog]() Topics: [responsible-ai](), [shadow AI](), [AI Strategy](), [AI Platforms/Deployment](), [Tooling](), [Stack Overflow](), [Microsoft]() Tags: [ai](), [ai-adoption](), [article](), [cc-by-sa](), [compliance](), [developer](), [devex](), [experimentation](), [microsoft](), [platforms](), [responsible-ai](), [se-stackoverflow](), [se-tech](), [shadow-ai](), [tools](), [workflow]() ### AI overview The article argues that responsible AI adoption depends on designing developer workflows, not merely publishing policies. It presents shadow AI as a signal that approved tools or processes are too slow, vague, or disconnected from engineering work, and recommends investigating workflow friction while providing monitored gateways and approved AI platforms that support visibility and experimentation. ### Source excerpt Organizations cannot solve shadow AI with a document employees read once. They need to make responsible use easier than improvised use. ## Two ways to measure the cumulative impact of experiments DevFeed: [Two ways to measure the cumulative impact of experiments]() Original publisher: [Read original article]() Author: Lukas Goetz-Weiss; Eddie Cai Published: 2026-08-18T00:00:00Z Content type: opinion Language: en Sources: [Datadog | The Monitor blog]() Topics: [experiments](), [Statistics](), [Data Science]() Tags: [analysis](), [experimentation](), [experiments](), [product](), [statistics](), [testing]() ### AI overview This article explains why summing the observed lift from winning experiments overstates cumulative impact because of the winner's curse. It presents two established approaches: a randomized holdout that measures the combined effect directly, and a statistical correction that aggregates existing experiment estimates. Datadog's Cumulative Impact feature applies the correction without requiring a quarter-long holdout and can analyze an entire experimentation program or a filtered team subset. ### Source excerpt Summing individual wins overstates true impact. See two accurate methods, holdouts and Datadog's Cumulative Impact, and how to choose between them. ## What building an AI-native finance function taught me DevFeed: [What building an AI-native finance function taught me]() Original publisher: [Read original article]() Published: 2026-08-10T17:00:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [Artificial Intelligence](), [OpenAI](), [real-time](), [data]() Tags: [ai](), [business](), [company](), [experimentation](), [finance](), [openai](), [real-time](), [tools](), [workflow]() ### AI overview OpenAI CFO Sarah Friar describes lessons from building an AI-native finance function, including zero-day financial close, continuously updated forecasting, broad access to AI, structured experimentation, workflow accountability, and measuring AI's dependable work for ROI. ### Source excerpt OpenAI CFO Sarah Friar shares five lessons for building an AI-native finance function, from automated forecasting to stronger controls and AI ROI. ## Evolving AI maturity for our user experience team DevFeed: [Evolving AI maturity for our user experience team]() Original publisher: [Read original article]() Author: info@1password.com (Matt Davey) Published: 2026-08-07T00:00:00Z Content type: article Language: en Sources: [Blog on 1Password Blog]() Topics: [AI Chat]() Tags: [ai](), [building-1password](), [community](), [design](), [experimentation](), [prototypes](), [user-experience](), [workflow]() ### AI overview The article presents an AI maturity model for user experience teams, arguing that mature adoption improves decisions, collaboration, and product experiences rather than merely accelerating work. ### Source excerpt AI is changing how products get made. For user experience teams, that means the very shape of the work is changing. There are two key elements of user experience design. On one side is craft: the interaction, nuance, visual judgement, emotional texture, and other qualities that make a product feel considered. On the other side are systems, strategy and behavioural thinking: journeys, concepts, mental models, product architecture, behavioural patterns, and the shared systems and languages that help teams make better products. AI tooling has created opportunities to deepen both of these skillsets. Designers can now get closer to the front-end experience using real components, real data, and realistic prototypes, instead of hoping that important details survive the process. At the same time, we now have more ability to work upstream, shaping product decisions at the strategy level. Now, rather than strategy and execution conflicting with each other, they can harmonize more closely, held together by a team that can think clearly and ship responsibly. The challenge of AI is that working faster simply produces more work; it doesn't always mean that work is better. For UX and design teams, AI maturity is not simply about whether a team uses AI, but whether it improves the quality of our decisions, our collaboration, and the experiences we ship. An AI maturity model for user experience teams In the early stages of adoption, AI use tends to be experimentation without much structure. A designer might use it to generate a few rough ideas or make an impressive prototype, but it falls apart when the team asks how it would actually work. To avoid the pitfalls of confusing AI enthusiasm with maturity, 1Password has been investing in AI fluency across the company. To make that progress visible and chart a path to impact, we have developed a simple maturity model for design teams, which charts AI use from limited, reactive, developing, embedded, and finally through to leading. Head ## Correlation Lied to Us: Rethinking Product Impact with Causal Inference DevFeed: [Correlation Lied to Us: Rethinking Product Impact with Causal Inference]() Original publisher: [Read original article]() Author: Enderson Santos Published: 2026-08-04T15:31:01Z Content type: article Language: en Sources: [OLX]() Topics: [data]() Tags: [causal-inference](), [data](), [data-science](), [experimentation](), [machine-learning](), [metrics](), [performance](), [product-analytics]() ### AI overview The article examines why higher-tier OLX exposure packages appeared to perform worse than cheaper packages in some cases. It explains that sellers self-select packages, making direct package-level comparisons potentially misleading, and introduces causal inference as an approach to separate correlation from causation. ### Source excerpt Introduction At OLX, professional sellers pay for higher-tier packages because they promise more exposure. More visibility, and, in theory, better results. But when we looked at the data, something unexpected happened. In some cases, ads published with premium packages appeared to perform worse than ads using cheaper packages. That raised an uncomfortable question: If higher-tier packages provide more exposure, shouldn't they consistently perform better? At first glance, there were several possible explanations. Perhaps the extra visibility weren't creating as much value as we expected. Perhaps ranking dynamics were offsetting the additional exposure. Or perhaps the package itself wasn't the real driver of performance. It was then that we started asking a different question: Were we measuring this correctly? More specifically, were the ads across different packages actually comparable in the first place? Answering that question turned out to be far more important than comparing package-level metrics. It forced us to rethink how we measure product impact in a marketplace environment and ultimately led us to a causal inference approach designed to separate correlation from causation. In this article, I'll walk through how we approached that problem, what we learned, and how comparing similar ads changed our understanding of the true value created by exposure products. Problem Definition To understand the challenge, it's important to first understand how package exposure works at OLX. Professional sellers self select into a package when publishing their ads. The difference between packages is largely defined by how many boosts an ad receives during its lifetime. For example, in the picture below we can see that Package 1 includes 1 boost on the period of 30 days, package 2 includes 2 boosts, package 3 includes 3 boosts, and package 4 includes 4 boosts all in the same period of 30 days. The business expectation is straightforward: more boosts should create more visibili ## Open Sauce and GPS time were my summer AI Antiseptics DevFeed: [Open Sauce and GPS time were my summer AI Antiseptics]() Original publisher: [Read original article]() Author: jeff@jeffgeerling.com (Jeff Geerling) Published: 2026-07-22T14:00:00Z Content type: opinion Language: en Sources: [Jeff Geerling]() Topics: [Artificial Intelligence](), [Data visualization](), [MicroPython](), [Claude](), [datasets](), [Code]() Tags: [ai](), [badge](), [claude](), [coding](), [data-visualization](), [electronics](), [experimentation](), [gps](), [micropython](), [open-sauce](), [projects](), [time](), [videos](), [visualization]() ### AI overview The author describes using Claude to build a 2,141-line MicroPython Badgeware app for a Pimoroni Tufty 2350 and Adafruit PA1010D GPS module. The GPS Time Badge uses GPS data to discipline the badge's internal real-time clock, although timing accuracy was limited by the hardware and BadgeOS architecture. The author also logged GPS satellite and fix-quality data and used Claude to create an interactive visualization, highlighting data visualization as a practical use of AI tools. ### Source excerpt In the midst of our AI slop revolution, traveling to the West coast for Open Sauce this past weekend was the perfect antiseptic for rising costs, summer heat, and online divisiveness. It's ironic, then, that I used Claude to vibe code my Tufty GPS Time Badge. Partly due to time constraints, and partly because I wanted to see if I could complete a personal project end-to-end, without editing a line of code, I throw my requirements at Claude and ultimately came up with this 2141-line MicroPython app for Pimoroni's Badgeware ecosystem. ## Harness AI Configs for Runtime Controls of AI Behavior DevFeed: [Harness AI Configs for Runtime Controls of AI Behavior]() Original publisher: [Read original article]() Author: Nico Zelaya Published: 2026-07-21T00:00:00Z Content type: article Language: en Sources: [Harness Blog]() Topics: [Artificial Intelligence](), [configuration](), [feature flags](), [experiments](), [Model Routing](), [Inference]() Tags: [ai](), [blog](), [config](), [experimentation](), [inference](), [policy](), [routing]() ### AI overview Harness AI Config Management provides a governed runtime configuration layer for changing prompts, models, routing, inference parameters, and other AI behavior without redeploying code. It supports targeting, experimentation, approvals, policy controls, versioning, and audit trails. ### Source excerpt Harness AI Config Management helps teams change prompts, models, and AI behavior at runtime with targeting, experimentation, approvals, policy, and audit trails | Blog ## A Recap of the 2026 Experimentation Conference at Booking.com DevFeed: [A Recap of the 2026 Experimentation Conference at Booking.com]() Original publisher: [Read original article]() Author: Mel JI Mueller Published: 2026-07-16T08:18:03Z Content type: article Language: en Sources: [Booking.com Data Science]() Topics: [experiments](), [decision-making](), [Artificial Intelligence](), [Tooling]() Tags: [2025](), [ai](), [analysis](), [data-science](), [events](), [experimentation](), [organizational](), [parallel](), [recap](), [themes]() ### AI overview A recap of Booking.com's 2026 Experimentation Conference, which brought together more than 150 experimentation practitioners from 49 companies. The article summarizes survey findings, conference themes, and sessions on AI-assisted experimentation, experimentation quality and velocity, and organizational culture. ### Source excerpt By Kevin Anderson, Angelica Goetzen, Jorden Lentze, and Melanie Mueller On May 18, 2026, we hosted the third annual Experimentation Conference at Booking.com on our Amsterdam campus. What started in 2024 as an experiment itself -- would large-scale experimentation practitioners come together to learn from each other? -- has grown into an event which brings together over 150 practitioners from 49 companies which run experiments at scale. About one third of attendees came back a second or third time. The room collectively ran 56,000 experiments per year. It's a unique crowd, and that's exactly the point. The day opened with sharing the results of the survey data we collected from the participating companies on the state of experimentation across the room, revealing some interesting findings: most teams operate a centre of excellence model, roughly a third release over 90% of features through controlled experiments, and the top challenges are scaling, coordination, platform tooling, and culture. These shared experiences helped shape the programme. We had three sessions, grouped by the three conference themes: AI and experimentation: AI-assisted analysis, no-code experimentation Quality / velocity tradeoff: High-quality vs high-speed experimentation Experimentation culture: Build organizational buy-in and data-driven decision-making Each session followed the same format: two talks, then a panel discussion on the same topic. We closed with nine parallel breakout groups for deeper conversation. Below is a recap of the key sessions. Read the recap of 2025 | Read the recap of 2024 Session 1: AI and experimentation The conference started off with the hot topic of AI in experimentation. AI is changing how we experiment and how we support experimenters. How Experimentation Protects Decisions in an AI-Written World -- Marcel Toben Marcel Toben, Head of Engineering at Zalando, opened with a provocation he'd recently heard from software engineers in Berlin: nobody on his team had wr ## Your AI is only as responsible as you are DevFeed: [Your AI is only as responsible as you are]() Original publisher: [Read original article]() Author: Phoebe Sajor Published: 2026-07-14T07:40:00Z Content type: article Language: en Sources: [Stack Overflow Blog]() Topics: [AI Chat]() Tags: [agentic](), [ai](), [ai-governance](), [design](), [experimentation](), [microsoft](), [podcast](), [reduce](), [responsible-ai](), [se-stackoverflow](), [se-tech](), [workflow](), [workflows]() ### AI overview A podcast conversation with Microsoft's Sarah Bird on responsible AI, the NIST approach, the risks of impact-free experimentation, and human/AI workflow design that reduces unnecessary escalation. ### Source excerpt Recorded at Microsoft Build, Ryan welcomes Sarah Bird, Microsoft's Chief Product Officer for Responsible AI, about how we can build and use AI responsibly with the NIST approach, why most irresponsible AI comes from experimentation without thought of impact, and how Microsoft is researching thoughtful human/AI workflow design to reduce unnecessary escalation. ## Eliminate Reliability Blind Spots in AWS, Azure, and GCP DevFeed: [Eliminate Reliability Blind Spots in AWS, Azure, and GCP]() Original publisher: [Read original article]() Author: Andre Newman Published: 2026-07-14T00:00:00Z Content type: article Language: en Sources: [Gremlin Blog]() Topics: [Cloud](), [Kubernetes](), [Resilience](), [Amazon Web Services](), [Azure](), [Chaos Engineering]() Tags: [availability](), [aws](), [azure](), [containers](), [deployment](), [experimentation](), [features](), [gcp](), [gremlin](), [kubernetes](), [orchestration](), [outages](), [resilience](), [uptime]() ### AI overview Gremlin's expanded Detected Risks feature automatically identifies high-priority reliability risks across AWS, Azure, GCP, and Kubernetes environments. It is designed to reveal issues such as misconfigured deployments, crash-looping containers, missing readiness probes, and poor pod distribution before they cause outages. ### Source excerpt Discover reliability risks without running a single test. See how Gremlin identifies high-priority reliability risks across AWS, Azure, and GCP to prevent outages. ## Designing For Distressed Users: Why Mental Health Apps Shouldn't Follow Every UI Fashion DevFeed: [Designing For Distressed Users: Why Mental Health Apps Shouldn't Follow Every UI Fashion]() Original publisher: [Read original article]() Author: hello@smashingmagazine.com (Kat Homan) Published: 2026-07-09T15:00:00Z Content type: article Language: en Sources: [Articles on Smashing Magazine -- For Web Designers And Developers]() Topics: [App](), [User interface design](), [Accessibility]() Tags: [cognitive-load](), [design](), [evaluation](), [experimentation](), [health](), [innovation](), [mental-health](), [research](), [retention](), [ui](), [ux]() ### AI overview This article argues that mental health apps should prioritize simplicity, reduced cognitive strain, accessibility, trust, and retention over fashionable interface trends. It presents trend-driven design as a potential source of friction for distressed or overwhelmed users and introduces an evaluation framework grounded in research, product audits, and app-store evidence. ### Source excerpt Many UI trends are designed to capture attention and signal innovation, but those goals often conflict with the needs of mental health apps: reducing cognitive strain, fostering trust, and providing a sense of refuge. Kat Homan introduces an evaluation framework that helps designers assess whether trendy visual and interaction patterns support or undermine the unique goals of mental health experiences. ## The effect distribution: The missing piece in experimentation programs DevFeed: [The effect distribution: The missing piece in experimentation programs]() Original publisher: [Read original article]() Author: Tyler Buffington Published: 2026-07-02T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [experiments]() Tags: [a-b-testing](), [experiment](), [experimentation](), [experiments](), [testing]() ### AI overview The article explains why effect distributions are essential for interpreting results across experimentation programs. It shows how statistically significant results can all be false positives when true effects are concentrated at zero, and introduces the challenge of estimating true effects from noisy observed effects. ### Source excerpt Learn about the importance of considering the effect distribution when running experiments. ## Variance Reduction Below the Randomization Grain DevFeed: [Variance Reduction Below the Randomization Grain]() Original publisher: [Read original article]() Author: Tilman Drerup Published: 2026-07-01T16:28:36Z Content type: article Language: en Sources: [Instacart]() Topics: [experiments]() Tags: [capacity](), [causal-inference](), [economics](), [engineering](), [experiment](), [experimentation](), [experiments](), [machine-learning](), [marketplaces](), [metrics](), [reduce](), [science](), [techniques](), [variance]() ### AI overview This article explains how marketplace experiments can reduce metric variance below the level at which treatment is randomized. It describes cluster-level randomization for containing interference and shows how fine-grained outcome predictability can improve statistical power and reduce experimentation time. ### Source excerpt Sergio Camelo, Caitlin Kearns, Matias Cersosimo, and Tilman Drerup As artificial intelligence increases the velocity of engineering and science teams, experimental throughput is set to become a bottleneck for many product decisions. Many companies can now build faster than they can experiment, with queues of good ideas running the risk of not being tested because of lack of experimental capacity. This problem is particularly severe in marketplaces, where the presence of spillover and cannibalization effects between experimental units requires cluster-level randomization techniques. That randomization, in turn, has the unfortunate tendency to substantially reduce statistical power and slow down experimentation. In this post, we show that the predictability of outcomes at fine grains can be exploited to reduce the variance of aggregate metrics, even when experiments themselves are run at a coarse level. Since statistical power depends on metric variability, this yields considerable reductions in experimentation time. The Interference Problem In marketplace settings, behavior and outcomes for individual participants are inherently intertwined. In a delivery marketplace like Instacart, for example, the dispatch system solves a bipartite matching problem between shoppers and customer orders. Since assignments are global and interdependent, matching an order to one shopper means that the same order cannot be matched to another shopper. As a result, changing the handling for a single order creates ripples that affect the orders around it. If an experimenter were to assign a treatment intervention to one of these orders while leaving neighboring orders as controls, the latter would evidently be contaminated. A common response to this problem is to randomize treatments at the level of a cluster, chosen so that interference can stay within it. In food and grocery delivery, that cluster is typically a geographical region. Since every order within a region sees the same treatme ## Outlier Handling at Scale in Experimentation DevFeed: [Outlier Handling at Scale in Experimentation]() Original publisher: [Read original article]() Author: Margarida Moreira da Silva Published: 2026-07-01T13:44:26Z Content type: article Language: en Sources: [Booking.com Data Science]() Topics: [experiments](), [data](), [Simulation](), [Ground truth / benchmark quality](), [plotting]() Tags: [data](), [experimentation](), [experiments](), [false-positive](), [outlier-detection](), [simulation](), [variance]() ### AI overview The article examines how extreme values affect experimentation at Booking.com. It describes permutation tests and simulated A/A experiments for diagnosing distorted p-value distributions, and reports that increasing outlier magnitude and frequency can cause test failures. ### Source excerpt At Booking.com, thousands of experiments run simultaneously across highly heterogeneous users, from individual travellers to large travel agencies. This means our experiment data regularly contains legitimate but extreme values. When these go unhandled, they distort the statistical conclusions we draw, leading us to scale ideas that don't create value, or to discard ones that do. So, we need outlier handling methods that are reliable, automated, and applicable across diverse metrics without manual intervention. The Problem When extreme values are present in experiment data, they can compromise the estimation of average treatment effects (ATE), leading to unreliable test results and reduced statistical power. Even a single observation can inflate variance enough to mask a real effect or produce a spurious one. In practice, this means we risk shipping changes that appear positive but are not, or killing promising features because noise masked their real effect. At Booking.com's scale, this increase in false conclusions quickly compounds into a meaningful impact on customer experience and business outcomes. A Diagnostic Tool: the Permutation Test One way to assess whether extreme values are distorting results is the permutation test. By permuting over experiment data, we generate hundreds of simulated AA experiments where we know the ground truth: there is no real effect. Plotting the resulting p-values, we expect a uniform distribution. If it instead looks skewed, the underlying data distribution is compromising the validity of results. Plot 1: P-value distributions from simulated A/A tests. Clean normally-distributed estimated effects produce a uniform distribution (left), while the presence of extreme outliers results in skewed p-values (right), indicating a distorted false positive rate.Simulation Evidence: What Drives Failure? We ran AA permutation tests across a range of simulated data distributions to understand when they fail (i.e. not show a uniform p-value di ## Introducing GeneBench-Pro DevFeed: [Introducing GeneBench-Pro]() Original publisher: [Read original article]() Published: 2026-06-30T00:00:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [AI research agents](), [AI, ML & Data Engineering]() Tags: [agents](), [ai](), [ai-agents](), [analysis](), [benchmark](), [complexity](), [data](), [datasets](), [diagnostics](), [experimentation](), [explore](), [model](), [models](), [performance](), [research](), [skills](), [support](), [testing](), [workflow]() ### AI overview GeneBench-Pro is a research-level benchmark for evaluating whether AI agents can make judgment-heavy decisions during computational-biology analysis. It uses messy datasets and downstream decision targets to assess ambiguity handling, analytical-path selection, iterative experimentation, and revision of initial plans. ### Source excerpt Introducing GeneBench-Pro, a new benchmark testing AI performance in genomics, biology, and scientific research using complex, real-world datasets. ## Kotlin Multiplatform in Production: Two Real-World Use Cases from Booking.com DevFeed: [Kotlin Multiplatform in Production: Two Real-World Use Cases from Booking.com]() Original publisher: [Read original article]() Author: Diego Gómez Olvera Published: 2026-06-05T15:09:18Z Content type: article Language: en Sources: [Booking.com Development - Medium]() Topics: [Kotlin Multiplatform](), [compose-multiplatform](), [experiments](), [A/B Testing](), [Android](), [iOS](), [Design system](), [Mobile](), [Development]() Tags: [a-b-testing](), [android](), [booking](), [bookingcom](), [compose](), [compose-multiplatform](), [concepts](), [consistency](), [data](), [development](), [engineering](), [experiment](), [experimentation](), [experiments](), [ios](), [java](), [jetpack-compose](), [kotlin](), [kotlin-multiplatform](), [mobile](), [multiplatform](), [objective-c]() ### AI overview This article describes two Booking.com engineering use cases for Kotlin Multiplatform and Compose Multiplatform: a shared experimentation library for consistent experiment assignments across Android and iOS, and hosting an Android design system in a web browser. ### Source excerpt Introduction For the majority of Booking.com travelers, mobile is the primary channel for researching, planning, and booking trips. Recent data shows that over 80% of travelers rely on a mobile app during the research phase, with more than half of all bookings occurring on mobile devices. Consequently, the Android and iOS platforms are critical to the company's product strategy; engineering choices made here have significant repercussions for the entire organisation. To maintain agility at this scale, two elements must function in unison: Strict decision validation: At any time, Booking.com manages over 1,000 simultaneous experiments across its product suite, with hundreds active on mobile. Every minor adjustment undergoes A/B testing via our proprietary experimentation library before reaching the user. A unified design system ensures product consistency and makes design goals transparent to all contributors, not just maintenance engineers. This article examines two specific engineering challenges solved using Kotlin Multiplatform (KMP) and Compose Multiplatform (CMP): Developing a shared experimentation library to ensure uniform experiment assignments across Android and iOS. Using Compose Multiplatform to host our Android design system in a web browser, bridging the gap between design concepts and implementation. While both cases use the same underlying technology, each provides unique insights into multiplatform development. Use case 1: shared experimentation library on Android and iOSThe problem with two implementations Historically, our internal experimentation library, responsible for managing experiment assignments, evaluations, and tracking on mobile, was maintained as two distinct codebases: a mix of Java and Kotlin for Android and Objective-C for iOS. While intended to be identical, managing two languages with fluctuating team resources inevitably led to logic drift. Discrepancies in event-tracking and experiment-fetching behaviours emerged, though they wer [Next page]()