# A/B Testing A/B testing compares two or more versions or configurations by assigning participants to variants and analyzing results to determine which performs better against a defined goal. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Optimizing agent system prompts with Amazon Bedrock AgentCore DevFeed: [Optimizing agent system prompts with Amazon Bedrock AgentCore]() Original publisher: [Read original article]() Author: Han Ding Published: 2026-09-16T15:47:39Z Content type: article Language: en Sources: [Artificial Intelligence]() Topics: [Amazon Bedrock AgentCore](), [Prompt optimization](), [Prompt Engineering](), [Traces](), [Benchmark](), [A/B Testing](), [Automated prompt engineering]() Tags: [a-b-testing](), [amazon-bedrock](), [amazon-bedrock-agentcore](), [benchmark](), [best-practices](), [evaluation](), [intermediate-200](), [open-source](), [prompt-optimization](), [system-prompts](), [traces]() ### AI overview This technical companion explains how Amazon Bedrock AgentCore's system prompt optimizer uses production agent traces and reward signals to propose revised system prompts. It describes the reflector engine, recommendation explanations, offline batch evaluation, online A/B testing, and promotion workflow, and reports benchmark results for Single Agent Reflector and the experimental open source Sub-Agent Reflector. ### Source excerpt AgentCore optimization turns production traces into proposed configuration changes, then validates them before promotion. This technical companion to the launch post explains how the system prompt optimizer's reflector engine works and shares benchmark results for the Single Agent and Sub-Agent Reflectors. ## MAPS: Netflix's Multimodal Asset Personalization at Scale DevFeed: [MAPS: Netflix's Multimodal Asset Personalization at Scale]() Original publisher: [Read original article]() Author: Netflix Technology Blog Published: 2026-08-28T16:01:02Z Content type: article Language: en Sources: [Netflix](), [Netflix TechBlog - Medium]() Topics: [Netflix](), [personalization](), [Embeddings](), [multimodal](), [recommendation systems](), [data](), [A/B Testing]() Tags: [data](), [embedding](), [embeddings](), [machine-learning](), [model](), [multimodal](), [netflix](), [personalization](), [production](), [testing]() ### AI overview Netflix describes MAPS, a multimodal asset-personalization system that uses embeddings from artwork and video previews to address cold-start problems and personalize recommendations soon after a title launches. ### Source excerpt By Emma Yanyang Kong, Aditya Deshpande, Asad Abbasi, Bowei Yan, David Fagnan, Ashish Rastogi, Dhaval Patel, Ray Zhang Introduction The Netflix experience is a journey of discovery. Every visual cue, from the artwork on a title to the video previews that autoplay while you browse, is there to connect you with a story you will love. We call these visual cues assets, and choosing the right one for each member is a personalization problem of its own. But which image or video preview of Squid Game should we show you? And what do we do right after a title launches, when there's far too little interaction data to know which asset we should recommend to each member? For years, our models answered the first question well and the second poorly. They learned which assets members interacted with, but treated every asset as an opaque ID, blind to what was actually in the artwork or video preview. Right after a title launched, its assets had no history, so we dialed up exploration on its assets to gather interaction data, and otherwise fell back to popularity heuristics that ignore your taste. Only once enough interactions had piled up could personalization take over. This is the classic cold-start problem. This post shares how multimodal embeddings let our models see and hear the assets they recommend, so personalization can kick in far sooner, close to a title's launch. Because a new asset arrives with its embedding the model already understands, that embedding carries member taste signals from related assets immediately. Consequently, the model needs far less interaction history before it can personalize. We cover three production systems, artwork personalization, query-aware artwork ranking, and video preview personalization, plus a cheap trick for choosing new embeddings before committing to full end-to-end integration and A/B testing. Artwork Personalization A single image is often a member's first touchpoint with a title, so we create a diverse set of artworks for each tit ## When Can LLMs Replace Humans in A/B Tests? DevFeed: [When Can LLMs Replace Humans in A/B Tests?]() Original publisher: [Read original article]() Author: Spotify Engineering Published: 2026-08-13T18:57:22Z Content type: article Language: en Sources: [Spotify Engineering](), [Spotify Engineering Blog]() Topics: [Large Language Model](), [A/B Testing](), [datasets](), [data](), [experiments]() Tags: [a-b-testing](), [analysis](), [data](), [data-science](), [experiment](), [gpt](), [llm](), [llms](), [model](), [testing]() ### AI overview The article examines whether large language model predictions can replace human outcomes in A/B tests. Using the Upworthy dataset, it finds that calibrated predictions can recover treatment effects under specific assumptions, while raw predictions recovered only 39% of the observed human effect. ### Source excerpt TL;DR: LLM predictions can stand in for human outcomes in A/B tests, but only by assumption, not by design.... The post When Can LLMs Replace Humans in A/B Tests? appeared first on Spotify Engineering. ## Measuring time savings from Figma Make DevFeed: [Measuring time savings from Figma Make]() Original publisher: [Read original article]() Author: Remy Stewart Published: 2026-08-11T19:59:00Z Content type: article Language: en Sources: [Figma Blog]() Topics: [Figma](), [Data Science](), [Artificial Intelligence](), [A/B Testing]() Tags: [a-b-testing](), [ai](), [ai-tools](), [data-science](), [figma](), [productivity](), [research](), [testing]() ### AI overview Figma's Data Science team describes a randomized controlled trial measuring how Figma Make affects users' design work. Across 100 participants, design work was 20% faster and 16% easier overall, while PMs saw tasks become 23% faster and 37% easier. The article explains why confounders make AI time savings difficult to measure and discusses the limitations of online A/B testing and causal inference using log data. ### Source excerpt The Figma Data Science team assumed that AI saves users time--but quantifying it required a new approach to research design. ## How to Build an A/B Testing Framework for Modern CI/CD Pipelines DevFeed: [How to Build an A/B Testing Framework for Modern CI/CD Pipelines]() Original publisher: [Read original article]() Author: Aaron Newcomb Published: 2026-08-10T00:00:00Z Content type: tutorial Language: en Sources: [Harness Blog]() Topics: [A/B Testing](), [CI/CD](), [CI/CD Pipeline](), [feature flags](), [progressive delivery](), [Monitoring]() Tags: [a-b-testing](), [ci-cd](), [feature-flags](), [how-to](), [monitoring](), [progressive-delivery](), [rollback](), [verification]() ### AI overview This tutorial explains how to integrate A/B testing into CI/CD pipelines using feature flags, progressive delivery, real-time metrics, automated verification, and rollback controls. It argues that deployment should be decoupled from release so teams can validate changes before full rollout. ### Source excerpt Learn how to build an A/B testing framework in your CI/CD pipeline. Accelerate software delivery, reduce risk, and boost innovation. Start now! | Blog ## Datadog acquires Adaptive ML DevFeed: [Datadog acquires Adaptive ML]() Original publisher: [Read original article]() Author: Alexis Lê-Quôc Published: 2026-06-30T00:00:00Z Content type: news Language: en Sources: [Datadog | The Monitor blog]() Topics: [Artificial Intelligence](), [agent observability](), [Frontier AI](), [Reinforcement learning](), [Synthetic Data Generation](), [A/B Testing](), [Deployment](), [observability](), [Security](), [real-time]() Tags: [a-b-testing](), [acquisition](), [ai](), [ai-agents](), [autonomous](), [data](), [deployment](), [frontier-ai](), [models](), [observability](), [platform](), [production](), [real-time](), [reinforcement-learning](), [security](), [synthetic-data-generation]() ### AI overview Datadog announces the acquisition of Adaptive ML, whose Adaptive Engine helps enterprises build, own, and deploy specialized AI agents and models. The platform supports fine-tuning open models with reinforcement learning and synthetic data, evaluating them with AI judges and A/B testing, and using production signals to improve subsequent training. ### Source excerpt Datadog has acquired Adaptive ML, a platform for building, owning, and deploying specialized AI agents and models. ## See your serverless: introducing the Headlamp plugin for Knative DevFeed: [See your serverless: introducing the Headlamp plugin for Knative]() Original publisher: [Read original article]() Author: Mudit Maheshwari; Kahiro Okina Craftsman Software; Inc Published: 2026-06-25T18:00:00Z Content type: article Language: en Sources: [Kubernetes Blog]() Topics: [Serverless](), [Kubernetes](), [autoscaling](), [Routing (disambiguation)](), [A/B Testing](), [Open Source]() Tags: [a-b-testing](), [autoscaling](), [cli](), [kubernetes](), [open-source](), [routing](), [serverless]() ### AI overview This article introduces a Headlamp plugin for Knative that provides a unified interface for exploring and managing serverless workloads on Kubernetes. It covers resource relationships, KService editing, traffic splitting across revisions, autoscaling configuration, logs, redeployments, pod restarts, and permission-aware actions. ### Source excerpt Headlamp is an open-source, extensible Kubernetes SIG UI project designed to let you explore, manage, and debug cluster resources. Knative brings serverless workloads to Kubernetes, handling traffic routing, autoscaling, and revision management so teams can deploy and iterate without fighting infrastructure. But operating Knative workloads day-to-day can be difficult, there's still a lot of jumping between the kn CLI, kubectl, and the Kubernetes UI to get a full picture of what's running. We built the Headlamp Knative plugin to bridge that very gap, allowing operators to inspect, understand and act on their workloads all from a single place. This plugin was built as part of the LFX mentorship. Here's a tour of what we shipped. Here is a short walkthrough of the Knative plugin for Headlamp: Integrating Knative resources with Headlamp's map view Headlamp's resource mapping works for Knative CRDs too. You can see how KServices, Revisions, and DomainMappings relate to each other in a single graph view. KService management: edit traffic splits, restart pods, and view logs A KService is the top-level resource in Knative: it manages the lifecycle of Routes, Configurations, Revisions, and everything needed to run and expose your application. The plugin gives KServices a full detail view with an Edit Mode toggle for making live changes to traffic splits, autoscaling annotations, and more. Common actions like viewing the YAML, opening logs, triggering a redeploy, or restarting backing pods are surfaced in the header, gated by your current RBAC permissions. Traffic splitting: route across revisions for gradual rollouts and testing Knative makes it possible to route traffic across multiple Revisions of the same service. This is useful for canary releases, gradual rollouts, tagged preview URLs, and A/B testing. The plugin shows the traffic assigned to each Revision, the latest ready Revision, readiness status, age, and configured tags. In edit mode, you can adjust percentages an ## Gamification 2.0. Beyond Points and Badges: Designing for Players, Not Metrics. Conclusion DevFeed: [Gamification 2.0. Beyond Points and Badges: Designing for Players, Not Metrics. Conclusion]() Original publisher: [Read original article]() Author: Montgomery Singman Published: 2026-06-16T05:02:36Z Content type: opinion Language: en Sources: [UX Magazine]() Topics: [User experience (UX)](), [App](), [A/B Testing]() Tags: [a-b-testing](), [app](), [games](), [gaming](), [ux]() ### AI overview This conclusion to a gamification series argues that product teams should design genuinely valuable and enjoyable experiences for users rather than applying game mechanics solely to improve engagement metrics. It recommends identifying the game genre that naturally fits an app and learning from that genre's underlying psychology. ### Source excerpt Part 6 of the "Gamification Series." Design for players, not metrics Through decades of watching games evolve from 8-bit sprites to photorealistic 3D, from arcade cabinets to cloud gaming, from niche hobby to global culture, one principle has remained constant: people play games because games are worth playing. Not because games award points. Not because The post Gamification 2.0. Beyond Points and Badges: Designing for Players, Not Metrics. Conclusion appeared first on UX Magazine. ## Kotlin Multiplatform in Production: Two Real-World Use Cases from Booking.com DevFeed: [Kotlin Multiplatform in Production: Two Real-World Use Cases from Booking.com]() Original publisher: [Read original article]() Author: Diego Gómez Olvera Published: 2026-06-05T15:09:18Z Content type: article Language: en Sources: [Booking.com Development - Medium]() Topics: [Kotlin Multiplatform](), [compose-multiplatform](), [experiments](), [A/B Testing](), [Android](), [iOS](), [Design system](), [Mobile](), [Development]() Tags: [a-b-testing](), [android](), [booking](), [bookingcom](), [compose](), [compose-multiplatform](), [concepts](), [consistency](), [data](), [development](), [engineering](), [experiment](), [experimentation](), [experiments](), [ios](), [java](), [jetpack-compose](), [kotlin](), [kotlin-multiplatform](), [mobile](), [multiplatform](), [objective-c]() ### AI overview This article describes two Booking.com engineering use cases for Kotlin Multiplatform and Compose Multiplatform: a shared experimentation library for consistent experiment assignments across Android and iOS, and hosting an Android design system in a web browser. ### Source excerpt Introduction For the majority of Booking.com travelers, mobile is the primary channel for researching, planning, and booking trips. Recent data shows that over 80% of travelers rely on a mobile app during the research phase, with more than half of all bookings occurring on mobile devices. Consequently, the Android and iOS platforms are critical to the company's product strategy; engineering choices made here have significant repercussions for the entire organisation. To maintain agility at this scale, two elements must function in unison: Strict decision validation: At any time, Booking.com manages over 1,000 simultaneous experiments across its product suite, with hundreds active on mobile. Every minor adjustment undergoes A/B testing via our proprietary experimentation library before reaching the user. A unified design system ensures product consistency and makes design goals transparent to all contributors, not just maintenance engineers. This article examines two specific engineering challenges solved using Kotlin Multiplatform (KMP) and Compose Multiplatform (CMP): Developing a shared experimentation library to ensure uniform experiment assignments across Android and iOS. Using Compose Multiplatform to host our Android design system in a web browser, bridging the gap between design concepts and implementation. While both cases use the same underlying technology, each provides unique insights into multiplatform development. Use case 1: shared experimentation library on Android and iOSThe problem with two implementations Historically, our internal experimentation library, responsible for managing experiment assignments, evaluations, and tracking on mobile, was maintained as two distinct codebases: a mix of Java and Kotlin for Android and Objective-C for iOS. While intended to be identical, managing two languages with fluctuating team resources inevitably led to logic drift. Discrepancies in event-tracking and experiment-fetching behaviours emerged, though they wer ## The Causality Gap: Measuring the True Impact of Voluntary Adoption in Digital Marketplaces DevFeed: [The Causality Gap: Measuring the True Impact of Voluntary Adoption in Digital Marketplaces]() Original publisher: [Read original article]() Author: Lin Jia Published: 2026-05-22T08:05:43Z Content type: article Language: en Sources: [Booking.com Data Science]() Topics: [A/B Testing](), [doubleml](), [Data Science](), [Tool]() Tags: [a-b-testing](), [causal-inference](), [causal-machine-learning](), [causality](), [data-science](), [doubleml](), [e-commerce](), [experimentation](), [featured](), [measuring]() ### AI overview This article explains why standard A/B tests can mismeasure features that require voluntary adoption in digital marketplaces. It presents Randomized Encouragement Design combined with Double Machine Learning as a way to estimate both causal impact among adopters and overall rollout impact. ### Source excerpt by Lin Jia, Kexin Fei The Content of this post has been presented at Pydata Amsterdam 2026 and the slides can be found here TL; DR Whenever a feature requires voluntary adoption, standard A/B testing breaks: low adoption flattens the topline, and self-selection makes adopters incomparable to non-adopters. Combining Randomized Encouragement Design (RED) with Double Machine Learning (DoubleML) recovers two answers -- the causal lift for adopters and the rollout's overall impact. When they diverge, the gap turns an ambiguous topline into a sharp product decision -- build a better product, or build a better adoption funnel. 1. The Opt-In Barrier Across Demand and Supply Across the tech industry, many platform features rely on voluntary adoption. A customer chooses whether to claim a promotional discount. A traveller opts into a loyalty program. An e-commerce seller enables a smart-pricing tool. In every case, the platform cannot force adoption -- and the feature's true impact becomes hard to measure, on both demand and supply. At Booking.com this challenge spans both sides of the marketplace -- travellers choosing to log in, partners choosing to adopt new features. Unlike a search-ranking change that applies to 100% of traffic, opt-in features create a "trilemma" for Product Data Science: Voluntary Adoption (the "Opt-In" Barrier): Users must actively enable the feature. A standard A/B test cannot separate the product's effect from the motivation that drove users to adopt it. Extreme Heterogeneity: Travellers range from once-a-year holidaymakers to travel agencies booking thousands of nights; partners range from single-apartment hosts to hotel chains. This variance is noise on both sides. Finite Sample Sizes: Opt-in features target a finite sub-segment, so we cannot simply "run the test longer" to gain power. When these stack up, a flat topline can hide a strong product behind a weak adoption funnel. Genuinely strong features get killed, and resources flow into the wrong int ## Kubernetes v1.36: New Metric for Route Sync in the Cloud Controller Manager DevFeed: [Kubernetes v1.36: New Metric for Route Sync in the Cloud Controller Manager]() Original publisher: [Read original article]() Author: Lukas Metzner Published: 2026-05-15T18:35:00Z Content type: article Language: en Sources: [Kubernetes Blog]() Topics: [Kubernetes](), [Cloud](), [A/B Testing](), [API]() Tags: [a-b-testing](), [api](), [article](), [cloud](), [github](), [kubernetes](), [slack](), [testing]() ### AI overview Kubernetes v1.36 adds the alpha route_controller_route_sync_total metric to the Cloud Controller Manager. The metric helps operators evaluate watch-based route reconciliation, which syncs routes when nodes change instead of on a fixed interval. ### Source excerpt This article was originally published with the wrong date. It was later republished, dated the 15th of May 2026. Kubernetes v1.36 introduces a new alpha counter metric route_controller_route_sync_total to the Cloud Controller Manager (CCM) route controller implementation at k8s.io/cloud-provider. This metric increments each time routes are synced with the cloud provider. A/B testing watch-based route reconciliation This metric was added to help operators validate the CloudControllerManagerWatchBasedRoutesReconciliation feature gate introduced in Kubernetes v1.35. That feature gate switches the route controller from a fixed-interval loop to a watch-based approach that only reconciles when nodes actually change. This reduces unnecessary API calls to the infrastructure provider, lowering pressure on rate-limited APIs and allowing operators to make more efficient use of their available quota. To A/B test this, compare route_controller_route_sync_total with the feature gate disabled (default) versus enabled. In clusters where node changes are infrequent, you should see a significant drop in the sync rate with the feature gate turned on. Example: expected behavior With the feature gate disabled (the default fixed-interval loop), the counter increments steadily regardless of whether any node changes occurred: # After 10 minutes with no node changes route_controller_route_sync_total 60 # After 20 minutes, still no node changes route_controller_route_sync_total 120 With the feature gate enabled (watch-based reconciliation), the counter only increments when nodes are actually added, removed, or updated: # After 10 minutes with no node changes route_controller_route_sync_total 1 # After 20 minutes, still no node changes -- counter unchanged route_controller_route_sync_total 1 # A new node joins the cluster -- counter increments route_controller_route_sync_total 2 The difference is especially visible in stable clusters where nodes rarely change. Where can I give feedback? If you ## The Illusion of Choice: How Micro-Decisions Guide Macro-Control DevFeed: [The Illusion of Choice: How Micro-Decisions Guide Macro-Control]() Original publisher: [Read original article]() Author: Tushar Deshmukh Published: 2026-04-21T06:23:51Z Content type: opinion Language: en Sources: [UX Magazine]() Topics: [User experience (UX)](), [Usability](), [User interface design](), [A/B Testing]() Tags: [a-b-testing](), [article](), [design-patterns](), [interfaces](), [usability](), [ux]() ### AI overview This article examines how small interface decisions can influence user behavior and create an illusion of choice. It argues that choice architecture, including button prominence, wording, navigation order, and default states, can steer users toward predetermined outcomes, and calls for UX practices that balance usability with responsibility. ### Source excerpt Part 8 of the "Ethical UX Series." "The greatest enemy of freedom is a happy slave." -- Friedrich von Schiller What is "ethical UX," and why this series matters In the design world, we often speak about empathy, innovation, and delight. But underneath the surface of many successful interfaces lies a subtle, often unchecked force The post The Illusion of Choice: How Micro-Decisions Guide Macro-Control appeared first on UX Magazine. ## Beyond A/B Testing: Using Surrogacy and Region-Splits to Measure Long-Term Effects in Marketplaces DevFeed: [Beyond A/B Testing: Using Surrogacy and Region-Splits to Measure Long-Term Effects in Marketplaces]() Original publisher: [Read original article]() Author: Iraklikhorguani Published: 2026-03-25T13:56:39Z Content type: article Language: en Sources: [Lyft Engineering - Medium]() Topics: [A/B Testing](), [App](), [Optimization]() Tags: [a-b-testing](), [app](), [cost](), [drivers](), [driving](), [growth](), [optimization](), [payments](), [pricing](), [resources](), [revenue](), [testing]() ### AI overview Lyft describes why standard A/B tests may not capture the long-term and market-mediated effects of pricing, payments, and incentive decisions in its multi-sided marketplace. The article introduces surrogacy and region splits as approaches for measuring those effects. ### Source excerpt Image generated with Gemini 3 Pro (Google), 2026. Written by Amber Wang and Yoonji Kim at Lyft. Background Whenever you use the Lyft app, there is a complex balancing act happening behind the scenes. Various levers are used to keep the marketplace running smoothly; Base prices and coupons for riders affect demand, while driver pay and bonuses impact the level of available supply. Since every change to prices and payments impacts Lyft's costs and revenue, they lead to key optimization problems, such as: How should we allocate budget between driver incentives and rider incentives? How do we invest resources to achieve x% rides growth, and how much does it cost in terms of short term profit? These are the questions the Foundational Models team at Lyft tries to answer in a systematic way. A key ingredient is understanding the effects of different types of investments -- for instance, what will happen if we increase the total budget for driver incentives by x%? What will happen if we increase the rider price of all rides by y%? It's worth noting that the long term effects of such decisions tend to dominate the short term effects: we may earn more short term profit from a ride if we charge riders more and pay drivers less, but lose riders and drivers in the long run. Estimating the long term effects of resource allocation decisions is challenging in a multi-sided marketplace such as Lyft. Because these decisions tend to be consequential, their effects go beyond first order effects on directly affected users. For example, if we increase driver incentive spending by x% in week 1, drivers will drive more in week 1 (short term effect), and may return to drive a bit more in the following weeks (direct long term effects). But this is not the full picture: in week 1, when there is a positive increase in driver hours as the result of more incentives, riders will enjoy better experiences (e.g. less surge pricing, shorter wait times) and may want to return to Lyft in the future. How ## Firebase A/B Testing is now available for the web DevFeed: [Firebase A/B Testing is now available for the web]() Original publisher: [Read original article]() Author: Sumit Chandel; Athira Manu Published: 2026-03-04T00:00:00Z Content type: release Language: en Sources: [Firebase Blog]() Topics: [A/B Testing](), [Firebase](), [Web](), [Website](), [Google Analytics]() Tags: [a-b-testing](), [announce](), [firebase](), [firebase-remote-config](), [google-analytics](), [react](), [release](), [remote-config](), [typescript](), [web](), [web-apps]() ### AI overview Firebase A/B Testing is now available for web apps, extending its existing mobile app experimentation capabilities. The feature uses Google Analytics and Firebase Remote Config to run and analyze experiments, track metrics such as retention, revenue, and engagement, and identify statistically significant variations. ### Source excerpt From Mobile to Web: Easily run, analyze, and scale product experiments to drive real impact using Firebase A/B Testing now available for your web apps! ## Chargebee Review 2026: Pricing, Features and Fit DevFeed: [Chargebee Review 2026: Pricing, Features and Fit]() Original publisher: [Read original article]() Author: Joshua D'Costa Published: 2025-12-16T00:00:00Z Content type: comparison Language: en Sources: [Dodo Payments Blog]() Topics: [Software as a service](), [A/B Testing]() Tags: [2026](), [a-b-testing](), [alternatives](), [automated](), [billing](), [checkout](), [customer](), [enterprise](), [experiments](), [features](), [payments](), [paypal](), [retry](), [review](), [saas](), [self-service](), [subscription](), [teams]() ### AI overview A review of Chargebee's pricing, subscription management features, payment integrations, trial workflows, and fit for SaaS and digital businesses. It highlights Chargebee's enterprise focus, broad capabilities, and steeper learning curve for smaller teams. ### Source excerpt Comprehensive Chargebee review covering pricing tiers, subscription management features, enterprise focus, and which SaaS teams benefit most in 2026. ## Speed with Rigor: testing smarter with group sequential design DevFeed: [Speed with Rigor: testing smarter with group sequential design]() Original publisher: [Read original article]() Author: Gabriela Lewenfus Published: 2025-08-05T17:25:29Z Content type: article Language: en Sources: [OLX]() Topics: [Testing](), [A/B Testing](), [experiments](), [data](), [implementation]() Tags: [a-b-testing](), [ab-testing](), [data](), [efficiency](), [experiment](), [experimentation](), [false-positive](), [implementation](), [pitfalls](), [sequential-testing](), [statistics](), [testing]() ### AI overview The article explains how group sequential testing can replace fixed-horizon A/B tests with monitoring at predefined intervals while controlling error rates. It discusses how this approach addresses peeking, may allow ineffective tests to stop early, and involves practical trade-offs and implementation challenges. ### Source excerpt generated by chatgptIntroduction In the world of A/B testing and experimentation, it's tempting to check results frequently and stop tests earlier if they seem significant (a practice known as peeking). However, this practice dramatically inflates the false positive rate, leading to misleading conclusions. Imagine you're running an A/B test, and after just 10 days, the results look like a clear win. You might be tempted to end the experiment early and move forward with the release of the new feature. After all, why wait another two weeks as originally planned? But here's the catch: the more often you check the results, the higher the chance you'll spot a "win" that isn't real. Frequent peeking increases the risk of false positives -- in other words, drawing the wrong conclusion just by chance. To prevent peeking while ensuring statistical power, a common practice is to set a minimum sample size that must be reached before ending the experiment. However, this fixed horizon approach can extend the experiment's duration and delay decision-making. Group sequential testing (GST) provides a statistical framework to monitor results at predefined intervals while maintaining control over error rates. At OLX, we migrated from fixed horizon experiments to GST to achieve faster insights, lower costs, and greater efficiency, enabling our teams to make smarter, data-driven decisions with speed and confidence. In this post, I'll explain how GST works, how it can accelerate experiments while still preventing the pitfalls of peeking, and how we can use it even to stop ineffective tests early. Finally, I'll discuss real-world trade-offs and implementation challenges. Peeking is a trap Before we dive into GST, let's first understand what "peeking" is and why it's so concerning. Peeking occurs when the experimenter repeatedly checks an experiment's results and finishes it once a statistically significant effect appears. This inflates the false positive rate, leading to unreliable conclu ## Deploying a model serving endpoint DevFeed: [Deploying a model serving endpoint]() Original publisher: [Read original article]() Author: Başak Tuğçe Eskili Published: 2025-08-02T11:00:01Z Content type: tutorial Language: en Sources: [MarvelousMLOps]() Topics: [databricks](), [MLOps](), [model-serving](), [model-deployment](), [Deployment](), [autoscaling](), [API](), [A/B Testing]() Tags: [a-b-testing](), [api](), [article](), [autoscaling](), [databricks](), [deployment](), [latency](), [mlops](), [model-serving]() ### AI overview Lecture 6 in an MLOps with Databricks course explains how to deploy registered MLflow models as Databricks Model Serving REST APIs. It covers serverless deployment, automatic scaling, monitoring, Unity Catalog integration, workload sizing, and limitations such as restricted runtime and cluster control. ### Source excerpt Lecture 6 of MLOps with Databricks course ## Framer upgrades user-facing analytics to real-time with Tinybird DevFeed: [Framer upgrades user-facing analytics to real-time with Tinybird]() Original publisher: [Read original article]() Author: Tinybird Published: 2025-07-01T00:00:00Z Content type: article Language: en Sources: [Tinybird]() Topics: [real-time](), [No-code](), [A/B Testing](), [Website]() Tags: [a-b-testing](), [analytics](), [customer-stories](), [infrastructure](), [no-code](), [real-time]() ### AI overview This article explains how Framer upgraded its user-facing analytics from basic metrics to real-time analytics with Tinybird, adding features such as A/B testing and funnels and enabling new revenue streams. ### Source excerpt Learn how the popular no-code website builder transformed their analytics offering from basic metrics to advanced features like A/B testing and funnels, enabling new revenue streams with Tinybird's real-time analytics infrastructure. ## How GrowthBook and ClickHouse make enterprise-grade A/B testing easy DevFeed: [How GrowthBook and ClickHouse make enterprise-grade A/B testing easy]() Original publisher: [Read original article]() Author: ClickHouse Published: 2025-06-11T11:19:41Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [A/B Testing](), [clickhouse](), [Open Source](), [Netflix]() Tags: [2025](), [a-b-testing](), [clickhouse](), [development](), [enterprise](), [experimentation](), [features](), [meetup](), [netflix](), [open-source](), [presentation](), [product-development](), [testing]() ### AI overview The article explains how GrowthBook and ClickHouse support scalable, enterprise-grade A/B testing and experimentation. It argues that aggregate product metrics can mislead teams, while controlled experiments provide clearer evidence about whether new features work. The article also describes the challenges of scaling experimentation as products mature. ### Source excerpt "Our goal, together with ClickHouse, is to bend that curve, so you can get really sophisticated, enterprise-grade experimentation without requiring much effort at all." Graham McNicholl, Co-founder and CEO ## Deliveroo experimentation principles DevFeed: [Deliveroo experimentation principles]() Original publisher: [Read original article]() Author: Ella Johnson-Watts Published: 2025-02-06T00:00:00Z Content type: article Language: en Sources: [Deliveroo]() Topics: [A/B Testing](), [experiments](), [Testing](), [Tooling]() Tags: [a-b-testing](), [algorithm](), [development](), [testing](), [tooling](), [ui]() ### AI overview Deliveroo describes how its experimentation programme evolved from ad hoc practices into a mature platform and culture. The article introduces principles such as defining a clear hypothesis and success criteria before running an experiment. ### Source excerpt Experimentation at Deliveroo has significantly boosted our innovation as a company. It has allowed us to explore new ideas, protected us from unforeseen challenges by preventing wasted time and resources on ineffective initiatives, and promoted intellectual honesty by validating our assumptions. Over time, the breadth of our experimentation programme, our tooling and our culture have undergone significant development. Each year, we run hundreds of experiments, testing a wide array of features. These experiments range from simple UI modifications to complex algorithm adjustments. We employ a variety of methods, from standard A/B testing to advanced techniques like interleaving, multi-armed bandits and switchbacks. We've moved from teams using ad-hoc methods for deploying and analysing experiments to a mature experimentation platform, which standardises and automates much of the experiment process. But successful experimentation requires more than just technical expertise and tooling. It requires an experimentation "culture" - a shared set of principles and values that embeds experiments in the broader organisational context, and ensures that experimentation is done in a way that is maximally impactful. As we have matured technically and increasingly democratised our technical capabilities, we have also evolved a set of such principles, which codifies that cultural side. These enable us to uphold high-quality experimentation whilst achieving our current goal of broadening the pool of Deliveroo employees capable of running experiments. Principle 1: Every experiment starts with a clear hypothesis and success criteria. Experiments work best when they are specific. We formulate our hypothesis based on past experiments, anecdotes, user research, and competitive analysis - leveraging existing learnings helps us design better experiments. We avoid experimenting aimlessly just to 'see what happens', risking learning nothing. Experiments are a highly valuable resource and unst ## Making Smarter A/B Testing Decisions with Event Tracking and Session Replays DevFeed: [Making Smarter A/B Testing Decisions with Event Tracking and Session Replays]() Original publisher: [Read original article]() Author: Ryan Barker Published: 2024-09-12T23:31:33Z Content type: article Language: en Sources: [Kogan.com]() Topics: [A/B Testing](), [data](), [User experience (UX)](), [Website]() Tags: [a-b-testing](), [conversion](), [data](), [e-commerce](), [sensitive-data](), [tracking](), [ux]() ### AI overview The article explains how event tracking and session replays can improve A/B testing decisions in e-commerce. Event tracking shows user actions and variant behavior, while session replays provide context about friction points, drop-offs, and usability issues. ### Source excerpt A/B tests are theoretically simple but sometimes offer challenges when insufficient data or events are tracked. In e-commerce, conversion rates are often highlighted as a key metric but what causes it and why. By leveraging additional tools that enable event tracking and session replays such as FullStory we can attach context and understand what users are actually doing, allowing us to make data-driven decisions which is crucial in modern business. Event Tracking: What Happened When it comes to e-commerce, every click matters, which is why we need to have an event-tracking mechanism. Event-tracking enables tracking user actions across your website, such as adding an item to the cart, hitting a call to action button, or proceeding to checkout. This data is essential in understanding the different behaviors that exist among your A/B test variants. For example, if you're testing two versions of a product page, event tracking helps you see: Click Rate: Which of the two gets more clicks on the 'Add to Cart' button? Engagement: How long do users spend on the page and how many elements such as images, product descriptions, or reviews do they interact with? Form Submissions: What user feedback method leads to the best uptake for optional benefits? This approach is beneficial because it allows you to understand what's happening behind the scenes, beyond just looking at the final sales numbers. Session Replays: Why it Happened While event tracking shows what happened, session replays reveal the why. Watching a replay of a customer's experience (with sensitive data masked) often uncovers behaviors and friction points you, as the developer, didn't anticipate or encounter during testing. It's an insight you simply can't get from final sales numbers, and it's invaluable when trying to identify behavioral patterns or usability issues. For example, if event tracking shows a significant drop-off with Variant A users who aren't reaching the checkout page, session replays might reveal ## Making Feature-Gated Code Easier to Remove DevFeed: [Making Feature-Gated Code Easier to Remove]() Original publisher: [Read original article]() Published: 2024-07-25T13:07:35Z Content type: opinion Language: en Sources: [Blogs on Ryan Harter]() Topics: [Code](), [Development](), [legacy](), [implementation](), [App](), [A/B Testing]() Tags: [a-b-testing](), [app](), [code](), [development](), [implementation](), [legacy]() ### AI overview This commentary explains the benefits and cleanup costs of feature gates. It argues that planning cleanup and minimizing branching can make legacy gated code easier and safer to remove after a rollout. ### Source excerpt I've been using feature gates for a long time. Having the ability to conditionally alter the behavior of an app unlocks so many benefits that I wouldn't go back. I love being able to have a consistent release train by keeping in-progress features locked behind a gate, being able to validate the utility and impact of the features I work on using A/B testing, and being able to mitigate the risk of rolling out new features. But feature gates also come with downsides, especially as a codebase and organization ages. ## Introducing frequentist inference in Firebase A/B Testing DevFeed: [Introducing frequentist inference in Firebase A/B Testing]() Original publisher: [Read original article]() Author: Stephanie Madison; Jon Mensing Published: 2023-11-21T00:00:00Z Content type: release Language: en Sources: [Firebase Blog]() Topics: [A/B Testing](), [Firebase](), [Inference](), [Statistics]() Tags: [a-b-testing](), [firebase](), [inference](), [news](), [remote-config](), [statistics](), [updates]() ### AI overview Firebase A/B Testing is replacing its previous Bayesian analysis methodology with a frequentist method. The article explains that the new approach exposes input values for independent verification and reports the leading variant as soon as it is detected rather than waiting a fixed number of days. ### Source excerpt News, tutorials, and updates from the Firebase team. ## Accelerating experimentations through Simulations DevFeed: [Accelerating experimentations through Simulations]() Original publisher: [Read original article]() Author: Rishav Jayswal Published: 2023-11-20T00:00:00Z Content type: article Language: en Sources: [Trivago]() Topics: [Simulation](), [Testing](), [A/B Testing](), [Java](), [reactive](), [Development]() Tags: [a-b-testing](), [backend](), [blog-post](), [data-science](), [development](), [java](), [reactive](), [testing]() ### AI overview This blog post describes a Java-based reactive tool that simulates production requests to provide faster indications of how application changes may affect key metrics. It presents the tool as a way to support hypothesis evaluation and potentially reduce the time required for A/B testing. ### Source excerpt During the development of customer-facing applications, time is crucial, especially when it comes to testing and analyzing changes before accepting them in production. This blog post explores ho... [Next page]()