# ab-testing Published articles for ab-testing. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Automating feature-flag cleanup at Yazio: from a prompt to CI DevFeed: [Automating feature-flag cleanup at Yazio: from a prompt to CI]() Original publisher: [Read original article]() Author: Artem Zasypalov Published: 2026-11-04T08:29:23Z Content type: article Language: en Sources: [YAZIO]() Topics: [agentic workflows](), [CI/CD](), [Pull Request](), [jira]() Tags: [a-b-testing](), [ab-testing](), [agentic-workflow](), [agents](), [ai](), [ai-coding-agents](), [automation](), [autonomous](), [ci](), [cleanup](), [feature](), [flag](), [prompt](), [software-engineering](), [workflow-automation]() ### AI overview Yazio automated feature flag removal from a manually prepared engineering task to a GitHub CI workflow that handles cleanup through opening a PR, with human oversight retained for merging. A Product Manager's removal decision starts the process; the agent removes flags and prepares changes, while a review agent gives a preliminary review. Reliability changes let the workflow process multiple flags sequentially with a fresh agent context and separate commit for each. It reads the Jira decision alongside Growthbook results and skips a flag when they conflict. Failures, skipped flags, gate errors, and successful PR links are reported in Jira. Over almost a year, the workflow created 237 tasks across mobile squads. Median time from cleanup initiation to PR creation fell from 32 days before the local prompt, to 17 days with the local prompt or skill, and 5 days with CI. A successful CI run takes about 15 minutes and removes a median 243 lines across 7 files. The article says the latest reliability improvements had been running only about ten days, so their backlog effect was not yet clear. ### Source excerpt Photo by Franck V. on Unsplash Remember those days when we thought robots and AI were supposed to take care of tedious things nobody wanted to do? Like washing dishes, taking out the rubbish? My team and I have been working to give AI a less physical version of that job: cleaning up after A/B tests. With more things being automated, we are expected to accomplish more as automatable work is often considered at least half done. There is truth to it. But it also means our context switching becomes the norm. A truly useful automation should preserve the focus of engineers as long as possible, meaning it should be able to act trustworthily without approval for as long as possible within the workflow. At the same time, critical stages should still be approved by a human. The definition of the critical stage might change over the next years with further advancements in AI and autonomous software development. Right now at Yazio, we consider merging a PR a critical stage that should be done under a human's oversight. We started experimenting with automated workflows that include AI coding agents over a year ago. Feature flag cleanup was one of the first ones for us to automate. Over several iterations, this effort has grown from a small 64-line prompt into an autonomous workflow, with an engineer only at the critical stage. I'd like to share our journey. Where cleanup starts Our A/B testing lifecycle has 3 core stages: Preparation, Running, and Cleanup. The Cleanup stage begins when the Product Manager chooses one of the 3 options after the test is stopped: 📦 Keep the flag if it still serves a purpose, for example, a feature config. 🔄 Update the code default if we want to roll out a variation as the new baseline while keeping the flag for another iteration. 🗑 Remove the flag if we decide to roll out one variation permanently. The workflow we designed covers the third case. The old process required me or someone else to pick up the A/B ticket, create an engineering task to r ## A Date-Stamped Data Warehouse Setup DevFeed: [A Date-Stamped Data Warehouse Setup]() Original publisher: [Read original article]() Author: Sahar Massachi Published: 2025-10-24T21:03:30Z Content type: article Language: en Sources: [DataExpert.io Newsletter]() Topics: [data-engineering](), [data](), [Apache Iceberg]() Tags: [ab-testing](), [data](), [data-engineering](), [storage](), [warehouse]() ### AI overview The article presents date-stamping data as a simple organizing principle for building resilient data warehouses and pipelines. It explains how this approach can manage changes over time, support experimentation and metrics, and work with systems including Hive metastore, Iceberg, Delta, and Hudi. ### Source excerpt Storage is cheap, your time is not! ## Speed with Rigor: testing smarter with group sequential design DevFeed: [Speed with Rigor: testing smarter with group sequential design]() Original publisher: [Read original article]() Author: Gabriela Lewenfus Published: 2025-08-05T17:25:29Z Content type: article Language: en Sources: [OLX]() Topics: [sequential-testing](), [Software Testing](), [A/B Testing](), [product-experimentation]() Tags: [a-b-testing](), [ab-testing](), [challenges](), [costs](), [data-driven](), [decision-making](), [design](), [efficiency](), [experimentation](), [experiments](), [false-positive](), [false-positive-rate](), [false-positives](), [implementation](), [olx](), [power](), [sample](), [sequential-testing](), [speed](), [statistics](), [testing]() ### AI overview Group sequential testing (GST) lets experimenters check A/B test results at predefined interim stages and stop early while controlling the overall false-positive rate. Efficacy boundaries require especially strong evidence at early checks; futility boundaries can stop tests unlikely to show an effect, but overly aggressive futility rules risk ending tests prematurely. Compared with a fixed-horizon design, GST may reduce average sample use when boundaries are crossed, while its maximum sample size is higher to account for repeated analyses. The text gives an example where a 20-stage design's maximum can be nearly 25% larger than the fixed-horizon sample size. More frequent checks can increase early stopping opportunities but also affect that maximum. Accurate traffic estimates and planned analysis timing matter: unexpected traffic increases may compromise Type I error control, while drops can reduce sensitivity. The article describes OLX's move to GST but provides no specific implementation results in this excerpt. ### Source excerpt generated by chatgptIntroduction In the world of A/B testing and experimentation, it's tempting to check results frequently and stop tests earlier if they seem significant (a practice known as peeking). However, this practice dramatically inflates the false positive rate, leading to misleading conclusions. Imagine you're running an A/B test, and after just 10 days, the results look like a clear win. You might be tempted to end the experiment early and move forward with the release of the new feature. After all, why wait another two weeks as originally planned? But here's the catch: the more often you check the results, the higher the chance you'll spot a "win" that isn't real. Frequent peeking increases the risk of false positives -- in other words, drawing the wrong conclusion just by chance. To prevent peeking while ensuring statistical power, a common practice is to set a minimum sample size that must be reached before ending the experiment. However, this fixed horizon approach can extend the experiment's duration and delay decision-making. Group sequential testing (GST) provides a statistical framework to monitor results at predefined intervals while maintaining control over error rates. At OLX, we migrated from fixed horizon experiments to GST to achieve faster insights, lower costs, and greater efficiency, enabling our teams to make smarter, data-driven decisions with speed and confidence. In this post, I'll explain how GST works, how it can accelerate experiments while still preventing the pitfalls of peeking, and how we can use it even to stop ineffective tests early. Finally, I'll discuss real-world trade-offs and implementation challenges. Peeking is a trap Before we dive into GST, let's first understand what "peeking" is and why it's so concerning. Peeking occurs when the experimenter repeatedly checks an experiment's results and finishes it once a statistically significant effect appears. This inflates the false positive rate, leading to unreliable conclu ## WISE: Skyscanner's Bayesian AB experimentation library and decision engine DevFeed: [WISE: Skyscanner's Bayesian AB experimentation library and decision engine]() Original publisher: [Read original article]() Author: Skyscanner Engineering Published: 2024-04-25T06:22:08Z Content type: article Language: en Sources: [Skyscanner]() Topics: [product-experimentation](), [A/B Testing](), [Data analysis](), [decision-making]() Tags: [ab-testing](), [analytics](), [bayesian](), [bayesian-statistics](), [best-practices](), [compare](), [consistency](), [data](), [data-science](), [data-scientists](), [designing](), [engine](), [engineering](), [experiment](), [experimentation](), [experiments](), [feature](), [features](), [jekyll](), [library](), [metrics](), [practices](), [product](), [product-managers]() ### AI overview WISE is Skyscanner's centralized Bayesian Python library that analyzes A/B experiments and recommends which variant to ship, reducing manual analysis and standardizing decisions across its Dr Jekyll platform and notebook analyses. Teams preregister one primary metric and guardrail metrics; WISE estimates conversion rates with a beta-binomial model and revenue per user with a hurdle gamma-exponential model, then evaluates relative-lift distributions using 90% highest-density intervals and a simulation-derived stopping-width rule. It labels metrics as beating, worse than, or inconclusive against control, or "keep testing" when the stopping threshold is unmet. A winner must beat control on the primary metric and not be worse on any guardrail; monitoring metrics do not determine the recommendation. The article also describes sample-ratio checks and long-term revenue estimates. It notes that Bayesian stopping and decision methods lack standardization, stakeholder education takes effort, and adding metrics can require custom models. ### Source excerpt By Dhanush Kishore, with Jose Parreño, May Alexander, Robert Shepherd, and the Skyscanner experimentation squad At Skyscanner, product decisions are driven by experimentation. As a global leader in travel, with 110 million users every month, we rely on a data-driven approach to growing and improving our product. Product managers, engineers, and data scientists run hundreds of experiments on our in-house AB experimentation platform, Dr Jekyll, to help make decisions about what features to ship for travelers and partners. When a team develops a new feature, the next step is to run a randomized controlled experiment. Every eligible user is randomly assigned either to the test variant B, where they interact with the new feature, or to the control variant A. After running the experiment for a certain duration, we test key metrics in variant B against variant A to determine whether the feature will be beneficial if rolled out to all users. This ensures that we follow an objective, scientifically rigorous approach to making product decisions. Although we already had a strong experimentation culture and an excellent experimentation platform, we identified the following needs as we scaled our experimentation program: Reduce manual effort in analyzing experiment results: While our experimentation platform, Dr Jekyll, reported test results for various metrics in an experiment, our data analytics team had to spend a considerable amount of time interpreting the results and making a decision on what variant to ship, which was becoming a bottleneck to scaling our experimentation program Need for standardization: Different teams at Skyscanner used different approaches to AB experimentation, and inconsistencies in methodologies meant it was difficult to compare results and share learnings across teams Consistency in adherence to experimentation best practices: While many teams maintained high standards of statistical rigor, others required additional support in designing and interpr