# product-experimentation

Product experimentation is the practice of testing product changes and variants to measure their impact on user behavior and product metrics.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## When Both Sides Adapt: The Challenges of Evaluating Interactive LLM Systems

DevFeed: [When Both Sides Adapt: The Challenges of Evaluating Interactive LLM Systems](<https://devfeed.tech/articles/when-both-sides-adapt-the-challenges-of-evaluating-interactive-llm-systems-132324.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17239>)

Author: Hannah Li

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>)

Tags: [a-b-testing](<https://devfeed.tech/tags/a-b-testing.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [effect](<https://devfeed.tech/tags/effect.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [hacking](<https://devfeed.tech/tags/hacking.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [linear](<https://devfeed.tech/tags/linear.md>), [live](<https://devfeed.tech/tags/live.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llms](<https://devfeed.tech/tags/llms.md>), [measuring](<https://devfeed.tech/tags/measuring.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [offline](<https://devfeed.tech/tags/offline.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>)

### AI overview

Users adapt their prompts and tasks in response to model outputs, so evaluation can change the prompt distribution as well as measure model performance. The paper distinguishes the common-distribution effect (comparing models on the same prompt distribution) from the induced-distribution effect (comparing each on prompts generated through interaction). Standard offline benchmark difference-in-means estimates target the former; live A/B test estimates target the latter. These can rank models differently and change deployment recommendations. The authors characterize when rankings reverse and give a sufficient condition for agreement as a heuristic for judging whether offline evaluation can reproduce a live decision. In a stylized linear interaction model, user adaptivity can also increase estimation variance. The paper connects adaptivity to challenges including reward hacking and novelty effects, and argues that evaluation should measure both model performance and user behavior induced by the model.

### Source excerpt

arXiv:2611.17239v1 Announce Type: cross Abstract: In interactive systems where users engage with LLMs, users are adaptive and, after observing LLM responses, can change their prompting strategy, as well as the type and difficulty of subsequent tasks. Thus model responses can influence the set of prompts on which the model is subsequently evaluated. We study the implications of this user adaptivity and prompt endogeneity for the evaluation of LLMs. We define two classes of estimands for comparing models: the common-distribution effect, which compares model performance on the same distribution of prompts, and the induced-distribution effect, which compares each model on the prompt distribution generated through user interaction. We show that offline benchmark evaluation and live A/B testing target different quantities, with the standard difference-in-means estimators for the two methods converging to the common-distribution effect and the induced-distribution effect, respectively. This distinction can affect decision-making: the two comparisons can rank models differently and lead to discrepancies in deployment recommendations. We characterize when such ranking reversals occur and provide a sufficient condition under which the rankings are guaranteed to agree. This result provides a practical heuristic for assessing whether offline evaluation is sufficient to reproduce the decision suggested by a live experiment. Third, we show, in a stylized linear model of interactions, that user adaptivity can increase variance in estimation. Finally, we discuss implications of user adaptivity for the interpretation of experiments and show that adaptivity creates a common mechanism leading to common evaluation challenges, such as reward hacking and novelty effects. Together, this work highlights the necessity of measuring both model performance as well as changes in the user behavior that the model induces.

## When Does a Recommendation Spark Curiosity? Theory, Measurement, and Field Evidence

DevFeed: [When Does a Recommendation Spark Curiosity? Theory, Measurement, and Field Evidence](<https://devfeed.tech/articles/when-does-a-recommendation-spark-curiosity-theory-measurement-and-field-evidence-132936.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.14938>)

Author: Pan Li, Alexander Tuzhilin

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [personalization](<https://devfeed.tech/topics/personalization.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [asia](<https://devfeed.tech/tags/asia.md>), [based](<https://devfeed.tech/tags/based.md>), [build](<https://devfeed.tech/tags/build.md>), [can](<https://devfeed.tech/tags/can.md>), [canonical](<https://devfeed.tech/tags/canonical.md>), [click](<https://devfeed.tech/tags/click.md>), [company](<https://devfeed.tech/tags/company.md>), [consumer](<https://devfeed.tech/tags/consumer.md>), [content](<https://devfeed.tech/tags/content.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [curiosity](<https://devfeed.tech/tags/curiosity.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [design](<https://devfeed.tech/tags/design.md>), [econ-gn](<https://devfeed.tech/tags/econ-gn.md>), [field](<https://devfeed.tech/tags/field.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [q-fin-ec](<https://devfeed.tech/tags/q-fin-ec.md>), [recommendation](<https://devfeed.tech/tags/recommendation.md>), [spark](<https://devfeed.tech/tags/spark.md>), [theory](<https://devfeed.tech/tags/theory.md>)

### AI overview

The paper proposes a computational measure of how much curiosity a recommendation evokes, derived from five axioms formalizing information-gap theory. In a survey, the measure matched consumers' self-reported curiosity and outperformed alternative proxy metrics. A recommender incorporating the measure into its utility function outperformed state-of-the-art methods in offline tests on three datasets. In a controlled experiment at a major Asian video-streaming platform, it increased click-through rate, video views, and time spent compared with the platform's latest production system; the authors report that it was subsequently deployed and remains in operation. The supplied text gives no details on the measure's formula, experiment effect sizes, or deployment context.

### Source excerpt

arXiv:2611.14938v1 Announce Type: cross Abstract: Curiosity, the drive to close the gap between what one knows and what one wants to know, pulls consumers beyond their familiar content, sustains engagement, and breaks them out of the filter bubbles of repetitive, relevance-only recommendations. Despite its importance, curiosity is hardly incorporated into modern recommender systems for lack of a formal measure. The information-gap theory of curiosity provides no formal quantitative model that a recommender system can operate on, and existing proxy metrics do not accurately measure the curiosity that a recommended item evokes. To fill this research gap, we formalize the information-gap theory into five axioms of curiosity and construct the canonical formulation of a computational measure of the curiosity that a recommended item evokes in a consumer. We validate our curiosity measure through a survey-based study, in which it matches consumers' self-reported curiosity and outperforms alternative proxy metrics. We subsequently build a curiosity-aware recommender system that incorporates this measure into the utility function design, and we show through offline experiments on three real-world datasets that it significantly outperforms the state-of-the-art methods. Furthermore, we conduct a large-scale online controlled experiment at a major video-streaming platform in Asia, where we show that our method significantly increases click-through rate, video views, and time spent over the latest production system, leading to tangible economic impact for the company. Our method has since been deployed on the platform and remains in operation. Our findings establish consumer curiosity as a measurable and actionable objective for recommender systems, with important managerial implications for digital platforms to recommend what consumers wonder about.

## FIRE: Failure Isolation and Reliability for Production-Connected Experiments at Scale

DevFeed: [FIRE: Failure Isolation and Reliability for Production-Connected Experiments at Scale](<https://devfeed.tech/articles/fire-failure-isolation-and-reliability-for-production-connected-experiments-at-scale-131527.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.14962>)

Author: Shripad Nadgowda, Gaurav Mitra, Sing Sing Ma, Vipul Chaskar, Pradipta Ghosh, Canturk Isci, Onkar Deshpande, Edgardo Aviles-Lopez, Ryan Feng, J\\'an Ondru\\v{s}ek

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [staging](<https://devfeed.tech/topics/staging.md>)

Tags: [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [failure](<https://devfeed.tech/tags/failure.md>), [guardrails](<https://devfeed.tech/tags/guardrails.md>), [isolation](<https://devfeed.tech/tags/isolation.md>), [monitors](<https://devfeed.tech/tags/monitors.md>), [production](<https://devfeed.tech/tags/production.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [recommendation](<https://devfeed.tech/tags/recommendation.md>), [record-and-replay](<https://devfeed.tech/tags/record-and-replay.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [scale](<https://devfeed.tech/tags/scale.md>)

### AI overview

FIRE supports production-connected experiments at scale by combining three guardrails: isolated tiers redirect replay traffic away from shared dependencies where replication is practical; dependency monitors track downstream health and experiment-attributed errors, stopping experiments after sustained violations; and traffic pinning confines calls to selected downstream workers when full replication would use scarce capacity. In a large-scale recommendation system, isolated ranking tiers handled billions of replayed requests daily with under 1% of corresponding serving capacity. Across 90 days, more than 200,000 abnormal worker exits in those tiers did not directly affect production-serving workers. In a separate three-month period, monitors cancelled about 5% of experiments for the measured ranking service and accounted for 87% of its cancellations. Dependency-aware triggering reduced code-review experiment executions by 25%.

### Source excerpt

arXiv:2611.14962v1 Announce Type: cross Abstract: Pre-production changes are usually tested in isolated environments to prevent their side effects from reaching production. Some record-and-replay platforms allow a production-connected mode: candidate changes receive recorded production requests while selected downstream calls reach production dependencies. This improves fidelity before deployment, but it also introduces reliability risks. At hyperscale, experimental traffic can fan out across shared services, trigger latent defects, or send malformed input to systems serving user requests. In this paper, we show how production-connected experimentation can operate reliably in a large-scale recommendation serving system. To contain downstream effects without sacrificing realistic dependency behavior, the platform uses three complementary guardrails. Isolated tiers redirect replay traffic away from shared dependencies where replication is practical. Dependency monitors observe downstream health and experiment-attributed errors and terminate experiments after sustained violations. Traffic pinning limits experimental calls to a designated subset of downstream workers when full replication would require scarce capacity. Isolated ranking tiers process billions of replayed requests per day using under 1% of the corresponding serving capacity. Over 90 days, these tiers experienced more than 200,000 abnormal worker exits without directly affecting production-serving workers. During a separate three-month window, dependency monitors cancelled approximately 5% of experiments for the measured ranking service and accounted for 87% of its cancellations. Dependency-aware triggering also reduced code-review experiment executions by 25%. These results show that high-fidelity, production-connected experimentation can operate at scale when downstream effects are isolated where possible, bounded where necessary, and continuously monitored and attributed.

## FME Metric Check Step: Gate Rollouts on Live Metrics

DevFeed: [FME Metric Check Step: Gate Rollouts on Live Metrics](<https://devfeed.tech/articles/fme-metric-check-step-gate-rollouts-on-live-metrics-132617.md>)

Original publisher: [Read original article](<https://www.harness.io/blog/fme-metric-check-step-gate-rollouts>)

Author: Ryan Vila

Published: 2026-11-13T00:00:00Z

Content type: tutorial

Language: en

Sources: [Harness](<https://devfeed.tech/sources/harness-blog.md>)

Topics: [progressive delivery](<https://devfeed.tech/topics/progressive-delivery.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [account](<https://devfeed.tech/tags/account.md>), [advanced](<https://devfeed.tech/tags/advanced.md>), [after](<https://devfeed.tech/tags/after.md>), [alert](<https://devfeed.tech/tags/alert.md>), [alerting](<https://devfeed.tech/tags/alerting.md>), [also](<https://devfeed.tech/tags/also.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [api](<https://devfeed.tech/tags/api.md>), [app](<https://devfeed.tech/tags/app.md>), [apply](<https://devfeed.tech/tags/apply.md>), [fail](<https://devfeed.tech/tags/fail.md>), [feature](<https://devfeed.tech/tags/feature.md>), [flag](<https://devfeed.tech/tags/flag.md>), [harness](<https://devfeed.tech/tags/harness.md>), [metric](<https://devfeed.tech/tags/metric.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [pass](<https://devfeed.tech/tags/pass.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>)

### AI overview

Harness FME's Metric Check step gates a rollout by evaluating flag metrics against JEXL failure conditions during a pipeline run. It measures users who received the flag treatment, then passes or fails; the configured pipeline failure strategy can stop or roll back the rollout. Add it to a Custom stage, choose the environment, flag, lookback window, and metrics, and define thresholds such as error-rate mean or latency p95. The lookback must contain events recorded after users' impressions, and metric and treatment names must match; missing data or names produce errors. Results appear in execution details and pipeline outputs. Metric Check uses fixed thresholds for fast release decisions, not statistical significance or causal experiment analysis. It is Limited GA, enabled per account, and requires pipeline view, edit, and execute permissions. FME environment-level approvals do not apply to pipeline runs, so approvals must be built into the pipeline.

### Source excerpt

The Metric Check step lets a Harness pipeline read a feature flag metric, compare it to a threshold you define, and pass or fail the rollout automatically. | Blog

## How Amplitude ships product improvements in 1-2 days with OpenAI

DevFeed: [How Amplitude ships product improvements in 1-2 days with OpenAI](<https://devfeed.tech/articles/how-amplitude-ships-product-improvements-in-1-2-days-with-openai-131631.md>)

Original publisher: [Read original article](<https://openai.com/business/customer-stories/amplitude>)

Published: 2026-11-12T20:00:00Z

Content type: news

Language: en

Sources: [OpenAI News](<https://devfeed.tech/sources/openai-news.md>)

Topics: [AI-enhanced product development](<https://devfeed.tech/topics/ai-enhanced-product-development.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [product analytics](<https://devfeed.tech/topics/product-analytics.md>), [OpenAI](<https://devfeed.tech/topics/openai.md>), [codex](<https://devfeed.tech/topics/codex.md>), [Product Management](<https://devfeed.tech/topics/product-management.md>), [implementation](<https://devfeed.tech/topics/implementation.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [amplitude](<https://devfeed.tech/tags/amplitude.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [days](<https://devfeed.tech/tags/days.md>), [impact](<https://devfeed.tech/tags/impact.md>), [improvements](<https://devfeed.tech/tags/improvements.md>), [models](<https://devfeed.tech/tags/models.md>), [openai](<https://devfeed.tech/tags/openai.md>), [openai-models](<https://devfeed.tech/tags/openai-models.md>), [proactive](<https://devfeed.tech/tags/proactive.md>), [product](<https://devfeed.tech/tags/product.md>), [ship](<https://devfeed.tech/tags/ship.md>), [teams](<https://devfeed.tech/tags/teams.md>)

### AI overview

Amplitude's Wave analyzes product analytics, session replays, customer feedback, and code to find improvement opportunities, prepare implementation plans and acceptance criteria, and track results. Teams can route work to coding agents such as Codex, with human review varying by change type. On Amplitude's homepage, Wave proposed personalizing suggestion cards after finding that 80 of 100 sampled users typed their own questions while 12 chose a suggestion; a team-reviewed, A/B-tested change increased card-driven sends by 160% versus control after one day. Amplitude assigns models by task: GPT-6.1 Sol for deeper investigations, GPT-6 Luna for frequent lower-cost checks, and GPT-Image-2.5 Flare for mockups. Evaluations report GPT-6.1 Sol leading Amplitude's Opportunity Investigator comparison, while its Data Science Agent score was slightly below another model's (67.5 vs. 69.3) at lower mean model cost per dispatch ($1.28 vs. $2.63). Amplitude estimates Wave can help ship improvements in 1-2 days rather than roughly a month; among customer-launched work, one quarter went live the same day an opportunity surfaced, half within four days, and two thirds within a week.

### Source excerpt

Built primarily on OpenAI models, Amplitude's proactive product agent, Wave, identifies what to build, helps teams ship changes with coding agents, and measures the impact.

## What will you build with infinite engineers?

DevFeed: [What will you build with infinite engineers?](<https://devfeed.tech/articles/what-will-you-build-with-infinite-engineers-130652.md>)

Original publisher: [Read original article](<https://amplitude.com/blog/wave-launch>)

Author: Darshil Gandhi

Published: 2026-11-12T00:00:00Z

Content type: release

Language: en

Sources: [Amplitude](<https://devfeed.tech/sources/amplitude.md>)

Topics: [Product Management](<https://devfeed.tech/topics/product-management.md>), [product analytics](<https://devfeed.tech/topics/product-analytics.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ambition](<https://devfeed.tech/tags/ambition.md>), [amplitude](<https://devfeed.tech/tags/amplitude.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analyst](<https://devfeed.tech/tags/analyst.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [approach](<https://devfeed.tech/tags/approach.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [area](<https://devfeed.tech/tags/area.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [customer](<https://devfeed.tech/tags/customer.md>), [decision](<https://devfeed.tech/tags/decision.md>), [improvements](<https://devfeed.tech/tags/improvements.md>), [next](<https://devfeed.tech/tags/next.md>), [product](<https://devfeed.tech/tags/product.md>), [results](<https://devfeed.tech/tags/results.md>), [ship](<https://devfeed.tech/tags/ship.md>)

### AI overview

Amplitude Wave connects product investigation and impact measurement to AI accelerated building. Teams set a business goal; Wave analyzes behavioral data, session replays, and customer feedback to identify Opportunities, recommend changes and test hypotheses, then measure results. Teams retain control over priorities and what reaches customers, while using product judgment to adapt recommendations. In examples, PixAI adapted a suggestion to guide new users to its Missions feature and reported higher Missions opens, publishing, and Day 1 retention. Amplitude used replay analysis to identify generic prompts that users rarely selected, then tested personalized suggestions, reporting about 160% more suggestion sends. DigitalOcean uses ranked Opportunities with experiment plans and can create Jira tickets from Wave. Wave is a paid offering for Growth and Enterprise customers; getting started includes assessing analytics and replay data. The excerpt presents these as company-reported examples and results.

### Source excerpt

Turn customer behavior into your next product decision with Amplitude Wave. Investigate opportunities, ship improvements, and measure results.

## Superhuman's AI advantage: measuring the business value of AI adoption

DevFeed: [Superhuman's AI advantage: measuring the business value of AI adoption](<https://devfeed.tech/articles/superhuman-s-ai-advantage-measuring-the-business-value-of-ai-adoption-127729.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/migration-and-modernization/superhumans-ai-advantage-measuring-the-business-value-of-ai-adoption/>)

Author: Artur Chiriac

Published: 2026-11-10T17:28:23Z

Content type: article

Language: en

Sources: [Migration & Modernization](<https://devfeed.tech/sources/migration-modernization.md>)

Topics: [AI Strategy](<https://devfeed.tech/topics/ai-strategy.md>), [developer-productivity](<https://devfeed.tech/topics/developer-productivity.md>), [internal developer platform automation](<https://devfeed.tech/topics/internal-developer-platform-automation.md>), [platform engineering automation](<https://devfeed.tech/topics/platform-engineering-automation.md>), [Developer experience](<https://devfeed.tech/topics/developer-experience.md>), [workflow automation](<https://devfeed.tech/topics/workflow-automation.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [alert triage](<https://devfeed.tech/topics/alert-triage.md>)

Tags: [adoption](<https://devfeed.tech/tags/adoption.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-coding](<https://devfeed.tech/tags/agentic-coding.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-adoption](<https://devfeed.tech/tags/ai-adoption.md>), [ai-assistant](<https://devfeed.tech/tags/ai-assistant.md>), [ai-assistants](<https://devfeed.tech/tags/ai-assistants.md>), [ai-code](<https://devfeed.tech/tags/ai-code.md>), [ai-code-review](<https://devfeed.tech/tags/ai-code-review.md>), [ai-economics](<https://devfeed.tech/tags/ai-economics.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [autonomous](<https://devfeed.tech/tags/autonomous.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [business-value](<https://devfeed.tech/tags/business-value.md>), [case-study](<https://devfeed.tech/tags/case-study.md>), [developer](<https://devfeed.tech/tags/developer.md>), [enterprise-strategy](<https://devfeed.tech/tags/enterprise-strategy.md>), [experimentation](<https://devfeed.tech/tags/experimentation.md>), [innovation-and-reinvention](<https://devfeed.tech/tags/innovation-and-reinvention.md>), [measuring](<https://devfeed.tech/tags/measuring.md>), [platform](<https://devfeed.tech/tags/platform.md>), [pull-requests](<https://devfeed.tech/tags/pull-requests.md>), [responsible-ai](<https://devfeed.tech/tags/responsible-ai.md>), [team](<https://devfeed.tech/tags/team.md>), [thought-leadership](<https://devfeed.tech/tags/thought-leadership.md>)

### AI overview

Superhuman reports that making AI adoption a platform-wide effort helped its platform team increase merged pull requests per engineer by 60% between H1 2025 and H1 2026, while keeping headcount constant. Weekly active use of its coding agent rose from 51% to 89% after company-wide workshops, an "AI Friday" sharing ritual, governed connectors, self-service prototyping, and easier tool access. Autonomous agents produced over 1,300 pull requests across 90+ repositories; engineers merged about two-thirds. The agents also handled deferred maintenance and support tasks, while engineers retained review and merge control. Output gains did not shorten median pull request cycle time, and 28% of pull requests required rework, which took 5.7 times longer. Superhuman says it is investing in validation, including AI review and quality gates. It also reports 3x as many experiments per growth engineer per week and roughly doubled migration output per engineer in one Amazon EKS project, while emphasizing that coordination and verification still required engineering work.

### Source excerpt

This post shares quantified results of the company Superhuman, which made AI to be the default way of work for their platform team. As a result, developer output rose 60%, 1,300 autonomous pull requests shipped in a single half, and experimentation rates tripled. This post looks at how the Superhuman's platform team did it, what [...]

## From Research to Prototype: Testing the New Desktop Settings

DevFeed: [From Research to Prototype: Testing the New Desktop Settings](<https://devfeed.tech/articles/from-research-to-prototype-testing-the-new-desktop-settings-127356.md>)

Original publisher: [Read original article](<https://blog.thunderbird.net/2026/11/from-research-to-prototype-testing-the-new-desktop-settings/>)

Author: Aly Blenkin

Published: 2026-11-10T15:10:23Z

Content type: news

Language: en

Sources: [The Thunderbird Blog](<https://devfeed.tech/sources/the-thunderbird-blog.md>)

Topics: [human centric design](<https://devfeed.tech/topics/human-centric-design.md>), [inclusive design](<https://devfeed.tech/topics/inclusive-design.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>)

Tags: [adopting](<https://devfeed.tech/tags/adopting.md>), [can](<https://devfeed.tech/tags/can.md>), [community](<https://devfeed.tech/tags/community.md>), [content](<https://devfeed.tech/tags/content.md>), [copy](<https://devfeed.tech/tags/copy.md>), [desktop](<https://devfeed.tech/tags/desktop.md>), [ease](<https://devfeed.tech/tags/ease.md>), [first](<https://devfeed.tech/tags/first.md>), [information](<https://devfeed.tech/tags/information.md>), [language](<https://devfeed.tech/tags/language.md>), [product](<https://devfeed.tech/tags/product.md>), [prototype](<https://devfeed.tech/tags/prototype.md>), [research](<https://devfeed.tech/tags/research.md>), [settings](<https://devfeed.tech/tags/settings.md>), [structured](<https://devfeed.tech/tags/structured.md>), [team](<https://devfeed.tech/tags/team.md>), [technical](<https://devfeed.tech/tags/technical.md>), [testing](<https://devfeed.tech/tags/testing.md>), [thunderbird](<https://devfeed.tech/tags/thunderbird.md>), [update](<https://devfeed.tech/tags/update.md>), [updates](<https://devfeed.tech/tags/updates.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Thunderbird's tree test found that some settings categories were easy to locate, but naming and organization caused confusion: participants struggled with the split between global and account settings, overlapping appearance and theme categories, and labels such as "New Features." The study gathered responses from 381 community members across 42 countries; the text cautions that a text-only tree test cannot assess visual cues or search, which many participants use as their primary way to find settings. The team used the findings to revise category organization in a lightweight interactive prototype, which lacks full functionality and operating-system-specific wording. Account and global settings remain separate for this iteration because combining them requires deeper architectural changes. The team plans incremental changes, with opportunities for feedback before features move to beta and stable releases.

### Source excerpt

In the last update, the team shared ongoing work around removing technical jargon and adopting plain language so everyone can navigate settings with ease. Those initial content refinements have rolled out into Thunderbird already (v160). Following those copy updates, the team ran research to evaluate how settings information is structured. Below is a breakdown of [...] The post From Research to Prototype: Testing the New Desktop Settings appeared first on The Thunderbird Blog.

## Query- and Candidate-Aware Long User Sequence Modeling for Sponsored Product Ranking in Search

DevFeed: [Query- and Candidate-Aware Long User Sequence Modeling for Sponsored Product Ranking in Search](<https://devfeed.tech/articles/query-and-candidate-aware-long-user-sequence-modeling-for-sponsored-product-ranking-in-search-125427.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11675>)

Author: Dongyue Xie, Sergey Kolbin, Manavender Malgireddy, Iurii Shcherbak, Weixing Tang, Arushi Jain, Wanchen Shao, Kurt Zimmer, Raja Hafiz Affandi, Matt Lindsay

Published: 2026-11-10T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [information retrieval](<https://devfeed.tech/topics/information-retrieval.md>), [personalization](<https://devfeed.tech/topics/personalization.md>), [ann](<https://devfeed.tech/topics/ann.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>)

Tags: [actions](<https://devfeed.tech/tags/actions.md>), [advertising](<https://devfeed.tech/tags/advertising.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [context](<https://devfeed.tech/tags/context.md>), [conversion](<https://devfeed.tech/tags/conversion.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [framework](<https://devfeed.tech/tags/framework.md>), [fusion](<https://devfeed.tech/tags/fusion.md>), [history](<https://devfeed.tech/tags/history.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [model](<https://devfeed.tech/tags/model.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [nearest-neighbor](<https://devfeed.tech/tags/nearest-neighbor.md>), [product](<https://devfeed.tech/tags/product.md>), [query](<https://devfeed.tech/tags/query.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [search](<https://devfeed.tech/tags/search.md>), [sequence](<https://devfeed.tech/tags/sequence.md>), [sponsored](<https://devfeed.tech/tags/sponsored.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [user](<https://devfeed.tech/tags/user.md>)

### AI overview

The framework ranks sponsored products by combining recent behavior with two long-term history retrieval channels: actions similar to the current candidate and actions similar to the current query. It encodes retrieved actions with query and candidate context tokens, and injects candidate context into each history token through early fusion. Offline ablations report improved major ranking metrics over the production baseline. An online A/B test on sponsored-product search traffic reports increases in CTR, attributed conversion rate, and orders per impression; the excerpt gives no effect sizes or implementation details.

### Source excerpt

arXiv:2611.11675v1 Announce Type: cross Abstract: User sequence modeling is widely used in recommendation and advertising. Search ranking presents an additional challenge: the current query expresses immediate intent, while a candidate product determines which parts of a user's long-term history are relevant. We present a production sequence framework for sponsored-product ranking that augments recent behavior with two retrieval channels over long-term history: candidate-nearest-neighbor retrieval and query-nearest-neighbor retrieval. The selected actions are encoded together with dedicated query and candidate context tokens, while candidate-aware early fusion injects the candidate into every history token. Offline ablations show that dual retrieval improves major ranking metrics relative to the production baseline. In an online A/B test on sponsored-product search traffic, the model increased CTR, attributed conversion rate, and orders per impression.

## What Consumer Goods Taught Me About Building Tech Products

DevFeed: [What Consumer Goods Taught Me About Building Tech Products](<https://devfeed.tech/articles/what-consumer-goods-taught-me-about-building-tech-products-125029.md>)

Original publisher: [Read original article](<https://sdtimes.com/software-development/what-consumer-goods-taught-me-about-building-tech-products/>)

Author: Sahit Garapati

Published: 2026-11-09T20:18:54Z

Content type: opinion

Language: en

Sources: [SD Times](<https://devfeed.tech/sources/sd-times.md>)

Topics: [product mindset](<https://devfeed.tech/topics/product-mindset.md>), [Business, Industry, Usability, User Experience, User Research, Web Strategy](<https://devfeed.tech/topics/business-industry-usability-user-experience-user-research-web-strategy.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [User Experience](<https://devfeed.tech/topics/user-experience.md>), [personalization](<https://devfeed.tech/topics/personalization.md>)

Tags: [advertising-technology](<https://devfeed.tech/tags/advertising-technology.md>), [ai](<https://devfeed.tech/tags/ai.md>), [consumer-behavior](<https://devfeed.tech/tags/consumer-behavior.md>), [consumer-goods](<https://devfeed.tech/tags/consumer-goods.md>), [customer-trust](<https://devfeed.tech/tags/customer-trust.md>), [customers](<https://devfeed.tech/tags/customers.md>), [decision-rules](<https://devfeed.tech/tags/decision-rules.md>), [empathy](<https://devfeed.tech/tags/empathy.md>), [engagement](<https://devfeed.tech/tags/engagement.md>), [experimentation](<https://devfeed.tech/tags/experimentation.md>), [guest-view](<https://devfeed.tech/tags/guest-view.md>), [hypothesis](<https://devfeed.tech/tags/hypothesis.md>), [portfolio](<https://devfeed.tech/tags/portfolio.md>), [portfolio-management](<https://devfeed.tech/tags/portfolio-management.md>), [product-management](<https://devfeed.tech/tags/product-management.md>)

### AI overview

The author argues that consumer goods product management transfers to advertising technology and AI through customer empathy, explicit hypotheses, and portfolio thinking, while tech requires faster release and learning cycles. At Kimberly-Clark, costly tests took six to nine months and required a one-page hypothesis; the author keeps that discipline for faster, cheaper tech experiments. In an advertiser workflow, a simpler default with advanced controls performed better than adding options. Portfolio rules helped focus resources where a minority of SKUs drove most profit; in ad tech, durable advertiser segments can guide scalable experiences without bespoke features for every customer. Other examples include simplifying a Huggies onboarding question and explaining an unchanged bidding feature in user-centered language, which roughly doubled adoption. The author concludes that technology needs consumer understanding and trust alongside model performance, and recommends that CPG professionals retain those instincts while adopting smaller releases and faster learning.

### Source excerpt

Most of what I rely on daily as a product leader in advertising technology, I first learned selling diapers. That probably sounds like a punchline, but the instincts CPG (consumer packaged goods) brand management builds--consumer empathy, hypothesis-driven experimentation, and portfolio thinking--transfer to AI product work in ways that surprised me. When people ask about the... ... continue reading The post What Consumer Goods Taught Me About Building Tech Products appeared first on SD Times.

## How I use Wave to find Amplitude's next product improvement

DevFeed: [How I use Wave to find Amplitude's next product improvement](<https://devfeed.tech/articles/how-i-use-wave-to-find-amplitude-s-next-product-improvement-126468.md>)

Original publisher: [Read original article](<https://amplitude.com/blog/wave-product-improvements>)

Author: Jacob Newman

Published: 2026-11-09T00:00:00Z

Content type: article

Language: en

Sources: [Amplitude](<https://devfeed.tech/sources/amplitude.md>)

Topics: [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [UI/UX Design](<https://devfeed.tech/topics/ui-ux-design.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [AI-driven information retrieval](<https://devfeed.tech/topics/ai-driven-information-retrieval.md>), [implementation](<https://devfeed.tech/topics/implementation.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [activation](<https://devfeed.tech/tags/activation.md>), [added](<https://devfeed.tech/tags/added.md>), [administrator](<https://devfeed.tech/tags/administrator.md>), [after](<https://devfeed.tech/tags/after.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [amplitude](<https://devfeed.tech/tags/amplitude.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [can](<https://devfeed.tech/tags/can.md>), [i](<https://devfeed.tech/tags/i.md>), [more](<https://devfeed.tech/tags/more.md>), [next](<https://devfeed.tech/tags/next.md>), [perspectives](<https://devfeed.tech/tags/perspectives.md>), [product](<https://devfeed.tech/tags/product.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [team](<https://devfeed.tech/tags/team.md>), [users](<https://devfeed.tech/tags/users.md>)

### AI overview

Amplitude uses Wave to connect replay and analytics evidence to product opportunities, proposed changes, implementation, and follow-up measurement; teams review the proposals before shipping. In a 100-replay analysis, users typed questions in 80 sessions while only 12 selected generic suggestions. Personalized cards based on recent chart or dashboard activity recorded 2.43 suggestion sends per 100 exposed users versus 0.94 for generic cards in an initial test, which still needs validation. Other shipped changes added paths from useful answers to Custom Agents or saved Notebook content, surfaced agent outputs through MCP, improved AI Feedback setup guidance, and clarified administrator prerequisites for Intercom and Zendesk connections. Reported outcomes vary: setup connection completion was 69.4%, while an HTTP API onboarding recommendation showed a directional, statistically inconclusive activation result. Wave also informed agent workflow changes and a docs search fix using a three-character minimum and about 200 ms debounce. Some proposals remain unconfirmed, and attribution or metric definitions still need work to assess impact.

### Source excerpt

See how the Amplitude team uses Wave to improve our AI agents, from more relevant suggestions to repeatable tasks and answers users can keep.

## How Wave makes me a better growth PM

DevFeed: [How Wave makes me a better growth PM](<https://devfeed.tech/articles/how-wave-makes-me-a-better-growth-pm-126467.md>)

Original publisher: [Read original article](<https://amplitude.com/blog/wave-product-friction>)

Author: Cassie Peretore

Published: 2026-11-09T00:00:00Z

Content type: article

Language: en

Sources: [Amplitude](<https://devfeed.tech/sources/amplitude.md>)

Topics: [UI/UX Design](<https://devfeed.tech/topics/ui-ux-design.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>)

Tags: [amplitude](<https://devfeed.tech/tags/amplitude.md>), [customers](<https://devfeed.tech/tags/customers.md>), [dashboard](<https://devfeed.tech/tags/dashboard.md>), [funnel](<https://devfeed.tech/tags/funnel.md>), [growth](<https://devfeed.tech/tags/growth.md>), [investigation](<https://devfeed.tech/tags/investigation.md>), [investigations](<https://devfeed.tech/tags/investigations.md>), [perspectives](<https://devfeed.tech/tags/perspectives.md>)

### AI overview

Amplitude Wave connects related customer feedback in Slack over time, keeping customer details together and preparing a problem summary, supporting evidence, and suggested solution outline so product teams can assess and act on friction without reconstructing reports themselves. The excerpt gives two examples: starting session replays about two seconds before a relevant funnel moment was followed by a 15% rise in average daily replay launches from those investigations over 30 days, sustained beyond launch; and improving Slack dashboard subscription setup with in-window reconnection, private-channel guidance, and manual channel ID entry led users to create 14.5% more subscriptions. These are reported outcomes from the author's examples, not claims about every use of Wave.

### Source excerpt

See how Cassie uses Amplitude Wave to fix replay, onboarding, and dashboard friction, and measure what changes for our customers.

## Flipsnack grows product signups with conversion optimization

DevFeed: [Flipsnack grows product signups with conversion optimization](<https://devfeed.tech/articles/flipsnack-grows-product-signups-with-conversion-optimization-120963.md>)

Original publisher: [Read original article](<https://openai.com/ads/customers/flipsnack>)

Published: 2026-11-05T00:45:15Z

Content type: news

Language: en

Sources: [OpenAI News](<https://devfeed.tech/sources/openai-news.md>)

Topics: [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [Product Management](<https://devfeed.tech/topics/product-management.md>)

Tags: [ads](<https://devfeed.tech/tags/ads.md>), [ads-manager](<https://devfeed.tech/tags/ads-manager.md>), [advertising](<https://devfeed.tech/tags/advertising.md>), [bidding](<https://devfeed.tech/tags/bidding.md>), [budget](<https://devfeed.tech/tags/budget.md>), [business-applications](<https://devfeed.tech/tags/business-applications.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [chatgpt-ads](<https://devfeed.tech/tags/chatgpt-ads.md>), [clicks](<https://devfeed.tech/tags/clicks.md>), [communications](<https://devfeed.tech/tags/communications.md>), [conversion](<https://devfeed.tech/tags/conversion.md>), [conversion-optimization](<https://devfeed.tech/tags/conversion-optimization.md>), [digital-experiences](<https://devfeed.tech/tags/digital-experiences.md>), [enablement](<https://devfeed.tech/tags/enablement.md>), [hospitality](<https://devfeed.tech/tags/hospitality.md>), [landing-pages](<https://devfeed.tech/tags/landing-pages.md>), [product](<https://devfeed.tech/tags/product.md>)

### AI overview

Flipsnack shifted five US ChatGPT Ads campaigns from a clicks objective with fixed bids to a conversions objective with bidding set to maximize conversions, measuring completed registrations as the first meaningful step in its signup flow. Across the campaign pairs, registrations rose about 310% and cost per registration fell about 75%; daily budgets were unchanged, while spend increased about 1.3%. The comparison covers August 25-31 versus September 2-8, 2026, uses 30-day click attribution, and excludes views. Because objectives and bidding changed together, the excerpt does not isolate the effect of either change.

### Source excerpt

Flipsnack focused its ChatGPT Ads campaigns on getting people to try its product, growing registrations while lowering the cost per signup.

## Learnings and Pitfalls from AI-Assisted A/B Experimentation at Scale

DevFeed: [Learnings and Pitfalls from AI-Assisted A/B Experimentation at Scale](<https://devfeed.tech/articles/learnings-and-pitfalls-from-ai-assisted-a-b-experimentation-at-scale-117499.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.04515>)

Author: Brett Daley, Ilya Gorbachev, Anubhav Natani, Nicholas Brown, Athena Dai, Rachel Song, Marcin Pekalski, Josip Domazet, Anchal Choubey, Daniel Ting

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>)

Tags: [a-b-testing](<https://devfeed.tech/tags/a-b-testing.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-assisted](<https://devfeed.tech/tags/ai-assisted.md>), [automate](<https://devfeed.tech/tags/automate.md>), [autonomous](<https://devfeed.tech/tags/autonomous.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [experimentation](<https://devfeed.tech/tags/experimentation.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [learnings](<https://devfeed.tech/tags/learnings.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llms](<https://devfeed.tech/tags/llms.md>), [pitfalls](<https://devfeed.tech/tags/pitfalls.md>), [scale](<https://devfeed.tech/tags/scale.md>)

### AI overview

The agent bounds LLM nondeterminism by letting the model select and sequence procedures while deterministic code computes every reported quantity. The authors report three lessons: withholding information improved answers; users relied more on routine retrieval than anticipated autonomous decisions; and the agent's eagerness to help could threaten statistical validity. The supplied text is an abstract, so it does not provide implementation details or evidence beyond these reported findings.

### Source excerpt

arXiv:2611.04515v1 Announce Type: new Abstract: A/B testing underpins product decisions by providing trustworthy, reproducible quantification of proposed changes' effects. Growing large language model (LLM) capabilities have motivated efforts to automate experimentation workflows. Yet LLMs are nondeterministic and prone to hallucination, properties that conflict with the trust experimentation systems must provide. Experiments are also operationally messy: diagnosing anomalies, debugging configurations, and conducting bespoke follow-up analyses require flexible reasoning and manual effort. These tasks let agents accelerate experimentation without replacing its deterministic statistical foundations. We report lessons from building and operating such an agent. Our central finding is that this tension is resolved not by making the model deterministic, but by bounding where its nondeterminism can reach; the model selects and sequences procedures, while deterministic code computes every quantity it reports. Three consequences surprised us: withholding information from the model improved its answers, demand concentrated on mundane retrieval rather than the autonomous decisions we anticipated, and the agent's eagerness to help threatened statistical validity.

## Compare AI agent changes with Online Experiments in Agent Observability

DevFeed: [Compare AI agent changes with Online Experiments in Agent Observability](<https://devfeed.tech/articles/compare-ai-agent-changes-with-online-experiments-in-agent-observability-118927.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/online-experiments-agent-observability/>)

Author: Alex Barksdale; Yakov Shapiro; Demetri Pananos; Miguel Tulla Lizardi

Published: 2026-11-04T00:00:00Z

Content type: article

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog-the-monitor-blog.md>)

Topics: [experiments](<https://devfeed.tech/topics/experiments.md>), [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [observability](<https://devfeed.tech/topics/observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-observability](<https://devfeed.tech/tags/agent-observability.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [can](<https://devfeed.tech/tags/can.md>), [changes](<https://devfeed.tech/tags/changes.md>), [compare](<https://devfeed.tech/tags/compare.md>), [cost](<https://devfeed.tech/tags/cost.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [errors](<https://devfeed.tech/tags/errors.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [latency](<https://devfeed.tech/tags/latency.md>), [learn](<https://devfeed.tech/tags/learn.md>), [observability](<https://devfeed.tech/tags/observability.md>), [online](<https://devfeed.tech/tags/online.md>), [production](<https://devfeed.tech/tags/production.md>), [quality](<https://devfeed.tech/tags/quality.md>), [test](<https://devfeed.tech/tags/test.md>), [testing](<https://devfeed.tech/tags/testing.md>), [traffic](<https://devfeed.tech/tags/traffic.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Datadog Online Experiments compares AI agent variants on real production traffic, pairing evaluation scores with operational measures such as latency, cost, token use, and errors. Unlike offline experiments built on predefined datasets, online evaluations can surface unanticipated requests and behaviors; random assignment against a control helps compare variants under the same period of usage. Feature Flags can adjust traffic exposure without a new deployment, and traces let teams inspect tool calls and responses behind metric changes. In Datadog's internal comparison, GPT-6 Luna's estimated cost was 44.3% lower than GPT-5.6 Luna, while mean quality scores were close (3.86 vs. 3.84); those averages did not establish equivalent quality. Duration and workflow-error estimates were higher but uncertain, and trace review found more invalid tool parameters in GPT-6 Luna, usually corrected on retry. The team addressed that issue before proceeding and identified failures as candidates for offline regression tests. The article presents online experiments as complementary to offline testing, with results also informing future evaluation datasets.

### Source excerpt

Learn how Datadog Online Experiments, available through Agent Observability, enables you to test AI agent changes against real production traffic so you can compare response quality, cost, latency, and errors.

## Difference-in-Differences vs Lagged-Outcome Regression

DevFeed: [Difference-in-Differences vs Lagged-Outcome Regression](<https://devfeed.tech/articles/difference-in-differences-vs-lagged-outcome-regression-106580.md>)

Original publisher: [Read original article](<https://towardsdatascience.com/difference-in-differences-vs-lagged-outcome-regression/>)

Author: Ryan OSullivan

Published: 2026-10-29T14:00:00Z

Content type: comparison

Language: en

Sources: [Towards Data Science](<https://devfeed.tech/sources/towards-data-science.md>)

Topics: [product analytics](<https://devfeed.tech/topics/product-analytics.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [doubleml](<https://devfeed.tech/topics/doubleml.md>)

Tags: [data-science](<https://devfeed.tech/tags/data-science.md>), [difference-in-differences](<https://devfeed.tech/tags/difference-in-differences.md>), [editor-s-picks](<https://devfeed.tech/tags/editor-s-picks.md>), [product-analytics](<https://devfeed.tech/tags/product-analytics.md>), [product-management](<https://devfeed.tech/tags/product-management.md>), [regression](<https://devfeed.tech/tags/regression.md>), [simulation](<https://devfeed.tech/tags/simulation.md>)

### AI overview

The study's choice between lagged-outcome regression and difference-in-differences (DiD) depends on how users self-select into treatment: lagged-outcome regression assumes no unobserved confounding after conditioning on outcome history and covariates, while DiD assumes parallel trends. In simulations of weekly engagement, DiD handled stable unobserved level differences but overstated effects when treated users opted in during a temporary dip; lagged-outcome regression handled the dip but was biased by stable unobserved confounding. A pre-trend check modeled on DiD detected the simulated dips and supports choosing DiD when flat or lagged-outcome regression when it dips, though a flat check does not prove parallel trends and can miss violations outside its window or in noisy, small samples. The authors advise reporting both estimates when the check dips, with conclusions treated cautiously. The results come from one simulated setting and do not establish how common these patterns are in real products; DiD is also not directly suitable for absorbing outcomes such as churn.

### Source excerpt

A simulation study and a pre-period check for choosing the right design for self-selected treatments in product analytics. The post Difference-in-Differences vs Lagged-Outcome Regression appeared first on Towards Data Science.

## How Fanatics and OpenAI streamline the fan experience

DevFeed: [How Fanatics and OpenAI streamline the fan experience](<https://devfeed.tech/articles/how-fanatics-and-openai-streamline-the-fan-experience-101255.md>)

Original publisher: [Read original article](<https://openai.com/index/fanatics-streamline-fan-experience>)

Published: 2026-10-26T07:00:00Z

Content type: news

Language: en

Sources: [OpenAI News](<https://devfeed.tech/sources/openai-news.md>)

Topics: [OpenAI](<https://devfeed.tech/topics/openai.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [UI/UX Design](<https://devfeed.tech/topics/ui-ux-design.md>), [codex](<https://devfeed.tech/topics/codex.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [chatgpt](<https://devfeed.tech/tags/chatgpt.md>), [chatgpt-work](<https://devfeed.tech/tags/chatgpt-work.md>), [codex](<https://devfeed.tech/tags/codex.md>), [collectibles](<https://devfeed.tech/tags/collectibles.md>), [commerce](<https://devfeed.tech/tags/commerce.md>), [corporate](<https://devfeed.tech/tags/corporate.md>), [daily](<https://devfeed.tech/tags/daily.md>), [data](<https://devfeed.tech/tags/data.md>), [engagement](<https://devfeed.tech/tags/engagement.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [experience](<https://devfeed.tech/tags/experience.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [help](<https://devfeed.tech/tags/help.md>), [look](<https://devfeed.tech/tags/look.md>), [openai](<https://devfeed.tech/tags/openai.md>), [products](<https://devfeed.tech/tags/products.md>), [shop](<https://devfeed.tech/tags/shop.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

Fanatics' OpenAI-powered Shop the Look recommends coordinated, in-stock merchandise on team jersey product pages. OpenAI models use enriched product, team and inventory data to assemble looks, while Fanatics' merchandising rules keep selections compatible and relevant; recommendations refresh daily as catalogs change. Codex helped the teams build tools to enrich catalog data, find compatible items, evaluate recommendations and generate looks. They tested 50 products in under a day, iterated on Fanatics' feedback, and approved a minimum viable product that week. The approach went into production and scaled across thousands of products and Fanatics network sites. Testing across roughly 30,000 MLB products showed 10% higher engagement than the previous Shop the Look experience.

### Source excerpt

Fanatics uses OpenAI to help fans complete their game-day outfits with Shop the Look. Testing across roughly 30,000 MLB products showed 10% higher engagement.

## Best practices for scaling A/B testing with experiment protocols

DevFeed: [Best practices for scaling A/B testing with experiment protocols](<https://devfeed.tech/articles/best-practices-for-scaling-a-b-testing-with-experiment-protocols-101076.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/best-practices-for-scaling-ab-testing-with-experiment-protocols/>)

Author: Eddie Cai; Lukas Goetz-Weiss

Published: 2026-10-26T00:00:00Z

Content type: tutorial

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog-the-monitor-blog.md>)

Topics: [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [experiments](<https://devfeed.tech/topics/experiments.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>)

Tags: [a-b-testing](<https://devfeed.tech/tags/a-b-testing.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [protocols](<https://devfeed.tech/tags/protocols.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [teams](<https://devfeed.tech/tags/teams.md>)

### AI overview

Datadog Experiments protocols turn an approved experiment methodology into reusable defaults for metrics, randomization, exposure, duration, analysis, and notifications. Published protocols prefill new experiment drafts; copied settings remain editable, and later protocol changes affect only new experiments. Define a primary outcome and secondary guardrails before launch, set acceptable thresholds, and choose stable subject identifiers and consistent assignment inputs across services. Staged exposure can limit risk, while duration should span a full business cycle. Review diagnostics before interpreting results: traffic imbalance is a blocker, mixed assignments are excluded from analysis, and dimensional or segment warnings can reveal localized problems. Agree on the analysis plan in advance: sequential frequentist analysis supports repeated checks, fixed-sample analysis requires a preset sample or duration, and Bayesian analysis needs its own decision criteria. CUPED can reduce variance when representative pre-experiment data is informative, but it does not correct biased randomization. Use separate protocols when recurring designs or decision processes differ.

### Source excerpt

Learn how to standardize A/B testing at scale with experiment protocols. Define metrics, randomization, exposure, and analysis, then reuse them across teams.

## Amplitude + GitHub Agents: Closing the Product Learning Loop

DevFeed: [Amplitude + GitHub Agents: Closing the Product Learning Loop](<https://devfeed.tech/articles/amplitude-github-agents-closing-the-product-learning-loop-90087.md>)

Original publisher: [Read original article](<https://amplitude.com/blog/amplitude-github-learning-loop>)

Author: Christina Nguyen

Published: 2026-10-19T00:00:00Z

Content type: article

Language: en

Sources: [Amplitude](<https://devfeed.tech/sources/amplitude.md>)

Topics: [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [GitHub Copilot](<https://devfeed.tech/topics/github-copilot.md>), [product analytics](<https://devfeed.tech/topics/product-analytics.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [amplitude](<https://devfeed.tech/tags/amplitude.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [copilot](<https://devfeed.tech/tags/copilot.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [feature](<https://devfeed.tech/tags/feature.md>), [github](<https://devfeed.tech/tags/github.md>), [github-copilot](<https://devfeed.tech/tags/github-copilot.md>), [github-issue](<https://devfeed.tech/tags/github-issue.md>), [guardrails](<https://devfeed.tech/tags/guardrails.md>), [ideas](<https://devfeed.tech/tags/ideas.md>), [impact](<https://devfeed.tech/tags/impact.md>), [integration](<https://devfeed.tech/tags/integration.md>), [product](<https://devfeed.tech/tags/product.md>)

### AI overview

The article describes a product learning loop connecting Amplitude Wave, GitHub Copilot, and Statsig. Wave uses behavioral analytics and session replays to identify onboarding friction and propose a fix; Copilot turns the proposal into a pull request for engineering review; and Statsig feature gates and randomized experiments measure the change before broader rollout.

### Source excerpt

How teams are building the right thing at AI speed with GitHub Copilot, Amplitude Wave, and Statsig

## Measuring the Impact of Personalized Recommendations

DevFeed: [Measuring the Impact of Personalized Recommendations](<https://devfeed.tech/articles/measuring-the-impact-of-personalized-recommendations-87084.md>)

Original publisher: [Read original article](<https://netflixtechblog.com/measuring-the-impact-of-personalized-recommendations-4c26be3a4d96?source=rss----2615bd06b42e---4>)

Author: Netflix Technology Blog

Published: 2026-10-14T19:57:43Z

Content type: article

Language: en

Sources: [Netflix](<https://devfeed.tech/sources/netflix-techblog-medium.md>)

Topics: [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>)

Tags: [consumer-behavior](<https://devfeed.tech/tags/consumer-behavior.md>), [economics](<https://devfeed.tech/tags/economics.md>), [effect](<https://devfeed.tech/tags/effect.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [measuring](<https://devfeed.tech/tags/measuring.md>), [model](<https://devfeed.tech/tags/model.md>), [personalization](<https://devfeed.tech/tags/personalization.md>), [recommendation-system](<https://devfeed.tech/tags/recommendation-system.md>), [recommendations](<https://devfeed.tech/tags/recommendations.md>), [recommender-systems](<https://devfeed.tech/tags/recommender-systems.md>)

### AI overview

Netflix describes a method for measuring the impact of personalized recommendations. It combines a behavioral model of member choice with small randomized tests that validate predicted substitution patterns. Comparisons with random, popularity based, and matrix factorization recommendations suggest the current system increases engagement while maintaining broader viewing across the catalog; the model attributes much of its advantage to targeting titles to the members most likely to respond.

### Source excerpt

By Kevin Zielnicki, Guy Aridor, Aurélien Bibaut, Allen Tran, Winston Chou, and Nathan Kallus Introduction Personalization is everywhere on the internet. It shapes the products we discover, the stories we read, the music we hear, and the movies and TV shows we watch. At Netflix, personalization has been central to the member experience for decades -- from the Netflix Prize to our modern recommender system that helps members navigate a large and constantly changing catalog. But personalization is not only an algorithmic problem, it is also a measurement problem. How do we know how much value personalization creates? More precisely, how do we separate the appeal of a title from the value created by recommending that title to the right member at the right time? In a new article in the International Journal of Industrial Organization -- which we are also presenting at this year's ACM Conference on Economics & Computation (EC '26) -- we develop a model of member behavior, title choice, and personalized recommendations. The model lets us estimate how recommendations affect both whether members choose to watch and, if so, which titles they choose. In this post, we summarize our approach and main findings. The Measurement Challenge A natural first attempt at valuing recommendations is to ask: what share of viewing comes from titles that were recommended? This is an intuitive metric, but it fails to answer the causal question at hand. The problem is that recommendations are not randomly assigned. A title appears on a member's homepage precisely because the recommender system predicts that the member might want to watch it. If a member watches that title, was it because the recommendation caused the member to watch it? Or would she have watched it anyway because of her underlying preference for that kind of title? This is the core measurement challenge: recommendations are targeted to demand, so viewing a recommended title combines the effect of personalization with the member's

## You're measuring AI adoption. You should be measuring AI abandonment

DevFeed: [You're measuring AI adoption. You should be measuring AI abandonment](<https://devfeed.tech/articles/you-re-measuring-ai-adoption-you-should-be-measuring-ai-abandonment-84042.md>)

Original publisher: [Read original article](<https://www.cio.com/article/4232656/youre-measuring-ai-adoption-you-should-be-measuring-ai-abandonment.html>)

Author: Nishant Virmani

Published: 2026-10-09T10:00:00Z

Content type: opinion

Language: en

Sources: [CIO](<https://devfeed.tech/sources/cio.md>)

Topics: [AI Strategy](<https://devfeed.tech/topics/ai-strategy.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [dora metrics](<https://devfeed.tech/topics/dora-metrics.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [artificial-intelligence-roi-and-metrics](<https://devfeed.tech/tags/artificial-intelligence-roi-and-metrics.md>), [contributor](<https://devfeed.tech/tags/contributor.md>), [edition-us](<https://devfeed.tech/tags/edition-us.md>), [language-en](<https://devfeed.tech/tags/language-en.md>), [metric](<https://devfeed.tech/tags/metric.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [roi-and-metrics](<https://devfeed.tech/tags/roi-and-metrics.md>), [saas](<https://devfeed.tech/tags/saas.md>), [storytype-opinion](<https://devfeed.tech/tags/storytype-opinion.md>)

### AI overview

The article argues that enterprise AI adoption figures such as licenses, activated seats and prompt counts do not establish durable value. It recommends measuring abandonment through cohort retention, diagnosing where usage drops, and using that evidence in renewal decisions. It also cautions that retention should be paired with delivery measures such as rework, code churn and peer review cycles.

### Source excerpt

Every CIO can produce an AI adoption dashboard: licenses purchased, seats activated, prompts submitted per week. Boards ask for these numbers, vendors report them and they climb every quarter. What none of them show is whether the tool earned a durable place in anyone's work. A license counts distribution, not value. An activated seat tells you someone clicked through onboarding once. A prompt count rewards the person who tried a tool five times before giving up exactly as much as the person who uses it daily and depends on it. Most CIOs concede that much, yet little better has replaced it. Across the C-suite, only 31% of leaders expect to measure generative AI's return on investment in the next six months. In most boardrooms, adoption metrics are not supplementing ROI; they have replaced it. Worse, these are usually not your numbers. In most SaaS deployments, the vendor defines the usage metric, instruments it and reports it back in business reviews. That is standard commercial practice rather than bad faith, but it means the criteria behind the number are built to justify license utilization rather than audit your workflows. The metric that actually shows whether an AI tool works is the one missing from the dashboard: abandonment. It tells you who tried a tool and walked away, how quickly they stopped and where the workflow broke down. Abandonment is not a vanity metric in reverse. It is the one signal your people produce rather than your vendor. Every day someone decides whether to open the tool, paying in attention and habit. Continuing to pay that cost is a judgment; walking away is the opposite judgment. Nobody fakes that behavior for a dashboard. Three patterns, three problems Not all abandonment means the same thing, and treating it as one undifferentiated category of failure misses the fix each pattern actually needs. Never adopted is the cleanest signal: a license was provisioned and never meaningfully used. I see this most often when a tool was bought at

## Metrics Board: Building an Agent-ready Metrics Layer

DevFeed: [Metrics Board: Building an Agent-ready Metrics Layer](<https://devfeed.tech/articles/metrics-board-building-an-agent-ready-metrics-layer-65684.md>)

Original publisher: [Read original article](<https://medium.com/pinterest-engineering/metrics-board-building-an-agent-ready-metrics-layer-2c8fefe68756?source=rss----4c5a5f6279b6---4>)

Author: Pinterest Engineering

Published: 2026-10-06T16:01:02Z

Content type: article

Language: en

Sources: [Pinterest](<https://devfeed.tech/sources/pinterest-engineering-blog-medium.md>)

Topics: [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [building](<https://devfeed.tech/tags/building.md>), [business](<https://devfeed.tech/tags/business.md>), [company](<https://devfeed.tech/tags/company.md>), [data](<https://devfeed.tech/tags/data.md>), [data-governance](<https://devfeed.tech/tags/data-governance.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-scientists](<https://devfeed.tech/tags/data-scientists.md>), [delivering-value](<https://devfeed.tech/tags/delivering-value.md>), [design](<https://devfeed.tech/tags/design.md>), [ecosystem](<https://devfeed.tech/tags/ecosystem.md>), [engagement](<https://devfeed.tech/tags/engagement.md>), [engineer](<https://devfeed.tech/tags/engineer.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [executive](<https://devfeed.tech/tags/executive.md>), [layer](<https://devfeed.tech/tags/layer.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [metrics-layer](<https://devfeed.tech/tags/metrics-layer.md>), [pinterest](<https://devfeed.tech/tags/pinterest.md>)

### AI overview

Pinterest's Metrics Board turns a metric definition into scheduled computation, quality checks, and publishing across data products, giving people and analytics agents a shared, governed source. Metrics are authored through guided forms and stored as versioned YAML in Git; each generates an isolated Airflow DAG that compiles SQL for SparkSQL or PrestoSQL, waits for upstream data, supports backfills, and resumes after late inputs. Rule-based thresholds and ML anomaly detection produce alerts and per-metric quality reports. Definitions publish metadata to PinCat and Helium at commit time, then fresh data to Superset and StarRocks after runs. Metric certification grades and dashboard tiers signal trust; tiers also govern documentation, retention, and query routing. An MCP server lets LLM tools discover definitions, while Pinterest's Analytics Agent grounds answers in certified metrics and golden queries. The authors report nearly 150 new metrics monthly, over 98% of experimentation metrics using the platform, and substantial time savings, including roughly two engineering-months on one project.

### Source excerpt

Michele Ceccacci; Software Engineer I | Jason Coffman; Sr. Software Engineer | Colm O'Shaughnessy; Software Engineer II | Laura Palmer; Staff Product Manager | Adam Podraza; Manager, Engineering | Surya Karri; Manager, Engineering At Pinterest, reliable and trustworthy metrics are behind every decision: from measuring company-wide business performance to evaluating each feature experiment results. Hundreds of data producers -- analysts, data scientists, and engineers across Pinterest -- create thousands of metrics from our petabyte scale data lake. A metrics ecosystem this size can't be ad hoc; it has to be organized, scalable, and resilient. So we built Metrics Board: an end-to-end platform that automates metric creation, enforces quality and enables agent-driven analytics. In this post we walk through the product experience, architecture, integrations and explain some of the design choices we made to make metrics work at planet scale. Why Metrics Matter at Pinterest Most organizations rely on a familiar set of metrics -- revenue, growth, engagement -- to steer the business. Metrics are how we understand whether Pinterest is delivering value to users and advertisers. Measures related to areas such as revenue, growth, and engagement turn the complexity of user behavior and Pinterest performance into a shared view of what is working, where we are falling short, and what matters most. These metrics drive the decisions our product, finance, and executive leaders make every day, from what to build to how we forecast and steer company strategy. Metrics also power Pinterest's product experimentation practice, which runs at massive scale -- thousands of experiments are active at any given time. Those experiments are used as gates for each new feature launch. For results to be fair, comparable, and statistically significant, metrics must be clearly defined, reliable, and reusable. Beyond common core metrics, individual experiments often need bespoke metrics scoped to a single mo

## How to choose your first Genie Agents for maximum impact

DevFeed: [How to choose your first Genie Agents for maximum impact](<https://devfeed.tech/articles/how-to-choose-your-first-genie-agents-for-maximum-impact-64032.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/how-choose-your-first-genie-agents-maximum-impact>)

Author: Nasir Dakri

Published: 2026-10-02T16:15:00Z

Content type: tutorial

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Retool](<https://devfeed.tech/topics/retool.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>)

Tags: [adoption](<https://devfeed.tech/tags/adoption.md>), [agents-in-production](<https://devfeed.tech/tags/agents-in-production.md>), [ai](<https://devfeed.tech/tags/ai.md>), [all](<https://devfeed.tech/tags/all.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [build](<https://devfeed.tech/tags/build.md>), [data-readiness](<https://devfeed.tech/tags/data-readiness.md>), [data-strategy](<https://devfeed.tech/tags/data-strategy.md>), [engineering](<https://devfeed.tech/tags/engineering.md>)

### AI overview

The article presents a five-criterion rubric--impact, demand, data readiness, scope, and governance--for prioritizing Genie Agent workflows. It recommends building when scores are high, shaping the workflow when scores are middling, and holding off when scores are low or there is no champion. Anonymized examples illustrate how data readiness, executive support, and existing demand affect adoption.

### Source excerpt

With more than 1 million Genie Agents created in 2026 alone, the question facing...

## Running production experiments with AWS AppConfig experimentation

DevFeed: [Running production experiments with AWS AppConfig experimentation](<https://devfeed.tech/articles/running-production-experiments-with-aws-appconfig-experimentation-62931.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/devops/running-production-experiments-with-aws-appconfig-experimentation/>)

Author: Aparna Krishnamoorthy

Published: 2026-09-30T19:45:35Z

Content type: tutorial

Language: en

Sources: [AWS DevOps & Developer Productivity Blog](<https://devfeed.tech/sources/aws-devops-developer-productivity-blog.md>)

Topics: [experiments](<https://devfeed.tech/topics/experiments.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [a-b-testing](<https://devfeed.tech/tags/a-b-testing.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-elastic-container-service](<https://devfeed.tech/tags/amazon-elastic-container-service.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cli](<https://devfeed.tech/tags/aws-cli.md>), [aws-systems-manager](<https://devfeed.tech/tags/aws-systems-manager.md>), [developer-tools](<https://devfeed.tech/tags/developer-tools.md>), [devops](<https://devfeed.tech/tags/devops.md>), [experimentation](<https://devfeed.tech/tags/experimentation.md>), [feature-flags](<https://devfeed.tech/tags/feature-flags.md>), [production](<https://devfeed.tech/tags/production.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

The tutorial explains how AWS AppConfig experimentation uses feature flags and A/B tests on controlled portions of production traffic. It walks through frontend and backend experiments, including testing a checkout button and comparing cache TTL settings, and covers assignment logging, health monitoring, result analysis, and promoting a winning treatment.

### Source excerpt

A redesigned checkout button is meant to lift sales. A longer cache time to live (TTL) is meant to cut backend load. But until you test a change against real production traffic, decisions come down to intuition and whoever argues hardest, not evidence. With AWS AppConfig experimentation, you can make data-driven calls instead: using A/B [...]

[Next page](<https://devfeed.tech/topics/product-experimentation.md?cursor=WyIyMDI2LTA5LTMwVDE5OjQ1OjM1KzAwOjAwIiwgIjhkODA5YzY1LTY1OWItNGJhOS1hZmZiLTk5YzVlNjQ5ZGQ2ZCJd>)