# A/B Testing

A/B testing compares two or more versions or configurations by assigning participants to variants and analyzing results to determine which performs better against a defined goal.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Variance-Optimal Counterfactual Policy Design for Sample-Efficient Online A/B Testing

DevFeed: [Variance-Optimal Counterfactual Policy Design for Sample-Efficient Online A/B Testing](<https://devfeed.tech/articles/variance-optimal-counterfactual-policy-design-for-sample-efficient-online-a-b-testing-143556.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24757>)

Author: Shashank Gupta, Subhadeep Maji

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [a-b-testing](<https://devfeed.tech/tags/a-b-testing.md>), [actions](<https://devfeed.tech/tags/actions.md>), [approach](<https://devfeed.tech/tags/approach.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [collection](<https://devfeed.tech/tags/collection.md>), [contributions](<https://devfeed.tech/tags/contributions.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [data-collection](<https://devfeed.tech/tags/data-collection.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [design](<https://devfeed.tech/tags/design.md>), [designs](<https://devfeed.tech/tags/designs.md>), [difference](<https://devfeed.tech/tags/difference.md>), [effect](<https://devfeed.tech/tags/effect.md>), [estimator](<https://devfeed.tech/tags/estimator.md>), [online](<https://devfeed.tech/tags/online.md>), [policy](<https://devfeed.tech/tags/policy.md>), [sample](<https://devfeed.tech/tags/sample.md>), [testing](<https://devfeed.tech/tags/testing.md>), [variance](<https://devfeed.tech/tags/variance.md>)

### AI overview

The paper proposes reducing online A/B test variance by jointly choosing which actions to sample and how observations estimate the treatment effect. It derives a closed-form, variance-optimal action-level logging policy for any fixed control variate, then uses historical logs to optimize the logging policy and control variate together. In simulated policy tests, the designs produced more accurate average treatment effect estimates than the evaluated variance-reduction methods in every setting, and generally improved statistical power and reduced selection of the worse policy. These results suggest lower traffic needs in the simulations; the supplied abstract does not establish performance in live deployments.

### Source excerpt

arXiv:2611.24757v1 Announce Type: new Abstract: Online A/B testing is the standard approach for evaluating changes to user-facing systems, such as a new recommendation or ranking policy, before deployment. While the difference-in-means estimator of a standard A/B test is unbiased, it often suffers from high variance. When the treatment and production policies choose the same actions with similar probabilities, the resulting observations can add noise while providing little information about the difference between the policies. Consequently, A/B tests can require large amounts of traffic and long durations to detect small treatment effects. Existing variance-reduction methods mainly improve the estimator after the data-collection scheme has been fixed, and recent overlap-aware designs only optimize a single traffic split between the two policies; thereby, they cannot direct traffic to the actions on which the policies disagree. In this paper, we jointly optimize how actions are sampled and how the resulting observations are used to estimate the treatment effect. Our contributions are two-fold. First, building on pairwise off-policy evaluation, we derive the variance-optimal action-level logging policy for estimating the average treatment effect between two policies, which has a closed-form solution for any fixed control variate. Second, we use historical logs to jointly optimize the logging policy and a control variate to further reduce variance. Our experimental results in simulated policy A/B tests show that our designs estimate the average treatment effect more accurately than existing variance reduction methods on every evaluated setting, while generally improving statistical power and reducing how often the worse policy is selected. As a result, our designs enable practitioners to reach reliable deployment decisions with considerably less online traffic.

## Datadog Brings Production Experimentation to AI Agent Observability

DevFeed: [Datadog Brings Production Experimentation to AI Agent Observability](<https://devfeed.tech/articles/datadog-brings-production-experimentation-to-ai-agent-observability-137152.md>)

Original publisher: [Read original article](<https://www.infoq.com/news/2026/11/datadog-ai-agent-observability/>)

Author: Craig Risi

Published: 2026-11-17T12:00:00Z

Content type: article

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [experiments](<https://devfeed.tech/topics/experiments.md>), [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [observability](<https://devfeed.tech/topics/observability.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-observability](<https://devfeed.tech/tags/agent-observability.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-agent-observability](<https://devfeed.tech/tags/ai-agent-observability.md>), [ai-coding-agents](<https://devfeed.tech/tags/ai-coding-agents.md>), [ai-ml-data-engineering](<https://devfeed.tech/tags/ai-ml-data-engineering.md>), [aiops](<https://devfeed.tech/tags/aiops.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [datadog-ai-agent-observability](<https://devfeed.tech/tags/datadog-ai-agent-observability.md>), [devops](<https://devfeed.tech/tags/devops.md>), [experimentation](<https://devfeed.tech/tags/experimentation.md>), [ml-data-engineering](<https://devfeed.tech/tags/ml-data-engineering.md>), [news](<https://devfeed.tech/tags/news.md>), [observability](<https://devfeed.tech/tags/observability.md>), [production](<https://devfeed.tech/tags/production.md>)

### AI overview

Datadog's Online Experiments lets teams compare AI agent versions on randomized real production traffic, pairing evaluator scores for response quality, accuracy, and completeness with latency, token use, estimated cost, and errors. Feature Flags support small, staged rollouts, while linked traces expose tool calls and results so teams can investigate differences and turn failures into offline regression tests. In the article's example, GPT-5.6 Luna and GPT-6 Luna received an even traffic split; their quality scores were close (3.84 and 3.86), and the newer model's estimated cost was 44.3% lower. Its execution duration and workflow errors appeared higher, but confidence intervals were too broad to establish regressions. Traces also revealed more invalid tool parameters, often recovered through retries, prompting tool-handling improvements before rollout. The approach complements offline evaluations, whose findings depend on how representative their datasets and scoring criteria are.

### Source excerpt

Datadog is extending its Agent Observability platform with Online Experiments, allowing engineering teams to compare AI agent versions against real production traffic rather than relying exclusively on predefined evaluation datasets. By Craig Risi

## From More to Better: Causal Inference and Digital Experimentation in the Age of AI

DevFeed: [From More to Better: Causal Inference and Digital Experimentation in the Age of AI](<https://devfeed.tech/articles/from-more-to-better-causal-inference-and-digital-experimentation-in-the-age-of-ai-132221.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16335>)

Author: Winston Chou, M{\\aa}rten Schultzberg, Lo-Hua Yuan, Wenjing Zheng, Avi Feller, David Holtz, Widad Machmouchi, Sven Schmit, Gagan Bhat, Simon Ejdemyr, Veronica Hannan, Amy Hartwig, Mahnaz Islam, Nathan

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [causality](<https://devfeed.tech/topics/causality.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [causal-inference](<https://devfeed.tech/tags/causal-inference.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [experimentation](<https://devfeed.tech/tags/experimentation.md>), [stat-ap](<https://devfeed.tech/tags/stat-ap.md>)

### AI overview

The article argues that as AI expands organizations' ability to produce and analyze data, data scientists' leverage increasingly lies in judging the credibility and importance of findings and whether evidence generalizes across contexts. Drawing on a roundtable of academic researchers and industry practitioners, it identifies three ways to turn greater analytical capacity into better decisions: preserve and accumulate learning from experimentation across individual tests; choose and combine experimental, observational, and synthetic evidence; and adapt evaluation as AI capabilities and uses evolve. For each area, it distinguishes established practice from open questions, and notes implications for data science training, practice, and organizations.

### Source excerpt

arXiv:2611.16335v1 Announce Type: new Abstract: Artificial intelligence (AI) is quickly expanding the capacity of organizations to produce and analyze data. We argue that this expansion shifts the leverage of data scientists from performing individual analyses to assessing the credibility and importance of findings and determining when evidence generalizes across contexts. Drawing on a roundtable of academic researchers and industry practitioners, we examine three ways in which data scientists who work on causal inference and experimentation can help organizations turn more analytical capacity into better decisions: preserving and accumulating the learning value of experimentation beyond individual tests; choosing and combining experimental, observational, and synthetic evidence; and ensuring evaluation keeps pace with AI as its capabilities and uses evolve. In each area, we distinguish established practice from open questions. We conclude with implications for the training and practice of data science and for organizations.

## Does paying more for a coding model deliver better performance? | Blog | Fivetran

DevFeed: [Does paying more for a coding model deliver better performance? | Blog | Fivetran](<https://devfeed.tech/articles/does-paying-more-for-a-coding-model-deliver-better-performance-blog-fivetran-136769.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/does-paying-more-for-a-coding-model-deliver-better-performance>)

Author: Aleksandar Kaplar

Published: 2026-11-11T16:05:32Z

Content type: news

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [OpenAI](<https://devfeed.tech/topics/openai.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>), [litellm](<https://devfeed.tech/topics/litellm.md>)

Tags: [claude-code](<https://devfeed.tech/tags/claude-code.md>), [coding](<https://devfeed.tech/tags/coding.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [openai-models](<https://devfeed.tech/tags/openai-models.md>), [performance](<https://devfeed.tech/tags/performance.md>), [study](<https://devfeed.tech/tags/study.md>), [value](<https://devfeed.tech/tags/value.md>), [workplace](<https://devfeed.tech/tags/workplace.md>)

### AI overview

In Fivetran's blinded two-week study, higher-priced OpenAI coding models showed only a modest, inconclusive preference edge in engineers' everyday work. Thirty-nine engineers were assigned GPT 5.6 Luna, Terra, or Sol through Claude Code; 27 surveyed after testing both assigned models. Sol won both matchups, but nearly half the preferences were weak, and the small vote counts did not establish a clear ranking. Typical weekly spend rose from about $5 on Luna to $41 on Terra and $168 on Sol; PR counts did not establish productivity differences. Feedback pointed to tradeoffs: deeper planning helped large refactors, while responsiveness and scope control mattered for focused tasks, where heavier models could overreach. The proxy setup also caused friction with non-Anthropic models, and each model handled the full agent loop at its own rates. The study did not measure potential savings from mixed-model workflows; Fivetran proposes a follow-up that holds execution models, tools, and budgets constant.

### Source excerpt

In a blinded workplace study, we tested 3 tiers of OpenAI models and learned that price is a poor proxy for value.

## Does paying more for a coding model deliver better performance? | Blog | Fivetran

DevFeed: [Does paying more for a coding model deliver better performance? | Blog | Fivetran](<https://devfeed.tech/articles/does-paying-more-for-a-coding-model-deliver-better-performance-blog-fivetran-123779.md>)

Original publisher: [Read original article](<https://www.fivetran.com/blog/does-paying-more-for-a-coding-model-deliver-better-performance>)

Author: Aleksandar Kaplar

Published: 2026-11-09T16:02:24Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>), [litellm](<https://devfeed.tech/topics/litellm.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [3](<https://devfeed.tech/tags/3.md>), [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [blog](<https://devfeed.tech/tags/blog.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [claude-sonnet](<https://devfeed.tech/tags/claude-sonnet.md>), [code](<https://devfeed.tech/tags/code.md>), [coding](<https://devfeed.tech/tags/coding.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [openai](<https://devfeed.tech/tags/openai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [proxy](<https://devfeed.tech/tags/proxy.md>), [study](<https://devfeed.tech/tags/study.md>), [value](<https://devfeed.tech/tags/value.md>), [workplace](<https://devfeed.tech/tags/workplace.md>)

### AI overview

In Fivetran's two-week blinded workplace study, a higher model price did not translate into a decisive day-to-day preference advantage. Thirty-nine engineers were assigned GPT 5.6 Luna, Terra, or Sol in Claude Code; 27 who used both assigned models completed the preference survey. Sol won both matchups, but nearly half of preferences were weak and the small vote counts did not establish a clear ranking. Typical weekly spend rose from about $5 on Luna to $41 on Terra and $168 on Sol. PR counts varied, but the study says they cannot establish productivity rankings because they depend on the work undertaken. Feedback pointed to tradeoffs: deeper planning helped large refactors, while speed and scope control mattered on focused tasks, where heavier models sometimes overreached. The proxy setup also introduced friction, and each model handled routine agent work at its own rates; the study did not measure potential savings from mixed-model routing. Its results apply to this setup and do not show that the models are equally capable.

### Source excerpt

In a blinded workplace study, we tested 3 tiers of OpenAI models and learned that price is a poor proxy for value.

## Verifier Noise Is a Temperature: Testing Tilt Predictions in Finite-Budget LLM-Guided Search

DevFeed: [Verifier Noise Is a Temperature: Testing Tilt Predictions in Finite-Budget LLM-Guided Search](<https://devfeed.tech/articles/verifier-noise-is-a-temperature-testing-tilt-predictions-in-finite-budget-llm-guided-search-123443.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09699>)

Author: Xiaomeng Li

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>)

Tags: [bin-packing](<https://devfeed.tech/tags/bin-packing.md>), [budget](<https://devfeed.tech/tags/budget.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [llm](<https://devfeed.tech/tags/llm.md>), [noise](<https://devfeed.tech/tags/noise.md>), [predictions](<https://devfeed.tech/tags/predictions.md>), [search](<https://devfeed.tech/tags/search.md>), [temperature](<https://devfeed.tech/tags/temperature.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

With both candidates rescored at each comparison, keep-if-better search has a tilted stationary law under stated assumptions; in a small-step diffusion limit, the tilt depends on the slope of the symmetric noise distribution at zero, and minibatch gates increase it with the square root of batch size. In LLM-guided program search, the predicted quality gain after 150 steps rose with minibatch size in six of seven settings but missed its magnitude on bin packing: search achieved 62-88% of the prediction when editing parent programs and 109-119% when writing new heuristics. The achieved fraction declined as unselected mutations had stronger parent-child quality correlation; a registered follow-up confirmed the direction but not corrected quantitative agreement. Good starting programs lost quality over 300 steps, and runs from different starts had not converged to a common level, leaving the stationary-law claim open. A registered prompt-optimization test on frozen task-model scores exceeded the tilt, and the bin-packing relation did not transfer.

### Source excerpt

arXiv:2611.09699v1 Announce Type: new Abstract: LLM-guided search loops propose a modified candidate, score it, and keep it if it scores better. When both candidates are re-scored at every comparison, as in the minibatch acceptance gates of prompt optimizers, keep-if-better is a reversible Markov chain if the score difference is logistic and the mutation operator is reversible, and its stationary law is the mutation equilibrium tilted by exp(beta f). For any continuous symmetric noise difference the same tilt is the stationary law of a small-step diffusion limit, with beta = 4F'(0); for a minibatch gate beta grows with the square root of the minibatch size. Caching the incumbent's score instead inflates the recorded score. We test the tilt as a prediction for LLM-guided program search, committing each prediction from calibration data before the search runs. The level after 150 steps rises with the minibatch size in six of seven settings, but its predicted value fails: on bin packing, search attains 62-88% of the predicted gain when the proposer edits the parent and 109-119% when it writes new heuristics. The attained fraction falls with the parent-child quality correlation of unselected mutation; a registered test on two new settings confirmed the direction, but not the quantitative agreement after a correction. Further registered tests found that runs started from a good program kept losing quality over 300 steps, more so from starts that search had selected, by a margin that passed its registered rule narrowly. Runs from different starts had not reached a common level, so whether the tilt describes the stationary law of LLM editing remains open. A registered test in prompt optimisation on frozen task-model scores failed: search exceeded the tilt, and the bin-packing relation did not transfer.

## How Far Can a Trainable Frequency Move? Resolution, Grids and Coupling in Trainable-FrequencyQuantum Models

DevFeed: [How Far Can a Trainable Frequency Move? Resolution, Grids and Coupling in Trainable-FrequencyQuantum Models](<https://devfeed.tech/articles/how-far-can-a-trainable-frequency-move-resolution-grids-and-coupling-in-trainable-frequencyquantum-models-119999.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.23409>)

Author: Michael Poppel, Markus Baumann, Sebastian W\\"olckert, Claudia Linnhoff-Popien, Jonas Stein

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [fourier transform](<https://devfeed.tech/topics/fourier-transform.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>)

Tags: [circuit](<https://devfeed.tech/tags/circuit.md>), [coupling](<https://devfeed.tech/tags/coupling.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-et](<https://devfeed.tech/tags/cs-et.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [develop](<https://devfeed.tech/tags/develop.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [gradient-flow](<https://devfeed.tech/tags/gradient-flow.md>), [grid](<https://devfeed.tech/tags/grid.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [models](<https://devfeed.tech/tags/models.md>), [quant-ph](<https://devfeed.tech/tags/quant-ph.md>), [quantum](<https://devfeed.tech/tags/quantum.md>), [series](<https://devfeed.tech/tags/series.md>)

### AI overview

Trainable-frequency models' movement depends strongly on their starting frequencies. The analysis finds that an isolated real frequency with an optimally adapting coefficient can reach its target under gradient flow when initialized within half a resolution step, while a fixed coefficient may encounter a phase-dependent barrier. A ternary prefactor grid places a model frequency within half a step across a band using logarithmically many encoding gates. In experiments, grid initialization improved synthetic-target fit success from 69% to 84% over the best random start. For targets with many frequencies, however, a fixed grid trained ten times longer outperformed training prefactors. On seven real series, trainable frequencies improved held-out R² by 0.010-0.045 on four, but a classical screen did not distinguish these gains from the others. A classical model with trainable frequencies showed the same dependence on initialization; the results concern training behavior, not a quantum-circuit advantage.

### Source excerpt

arXiv:2602.23409v4 Announce Type: replace-cross Abstract: Trainable-frequency quantum models train the prefactors of their encoding gates so that the circuit's frequencies can move onto the target's. From the unit start of prior work, training rarely reaches a target more than one resolution step away. We show that the target-driven part of the prefactor gradient is a weighted sum of slopes of the overlap $\sin u/u$, where $u$ is the model-target frequency distance scaled by the window half-width. This gives each frequency a reach: under gradient flow, an isolated real frequency whose coefficient adapts optimally reaches its target from within half a step, for every target at least one step from zero frequency; with a fixed coefficient, a phase-dependent barrier can stop it. A ternary prefactor grid places a model frequency within half a step of every band frequency, with logarithmically many encoding gates. Single-frequency circuits trained with Adam and jointly trained amplitudes behave as predicted: inside the first side lobe, a frequency starts toward its target exactly when in phase with it in 98% of runs, and only 11 of 768 frequencies starting beyond the lobe reached their target. With several frequencies, the grid start raised successful fits on synthetic targets from 69% to 84% against the best random start. With many target frequencies, least squares on the fixed grid fits near-perfectly, but the circuit does not reach the coefficients; training the prefactors narrows this shortfall, but with 8 and 24 target frequencies, training the fixed grid ten times longer fitted better. On real series, training the frequencies improved held-out $R^2$ at equal budget by 0.010--0.045 on four of seven series, but a classical screen did not separate these from the others. A classical model with independently trainable frequencies shows the same dependence on the start; our results describe how to train such models, not an advantage of the circuit.

## Personalized to Persuade: The Effects of Contextualization and Warmth on Trust and Reliance in Conversational AI

DevFeed: [Personalized to Persuade: The Effects of Contextualization and Warmth on Trust and Reliance in Conversational AI](<https://devfeed.tech/articles/personalized-to-persuade-the-effects-of-contextualization-and-warmth-on-trust-and-reliance-in-conversational-ai-117240.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.31275>)

Author: Mert Yazan, Suzan Verberne, Frederik Bungaran Ishak Situmeang

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Conversational AI](<https://devfeed.tech/topics/conversational-ai.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [trust](<https://devfeed.tech/topics/trust.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [human computer interaction](<https://devfeed.tech/topics/human-computer-interaction.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-assistant](<https://devfeed.tech/tags/ai-assistant.md>), [ai-literacy](<https://devfeed.tech/tags/ai-literacy.md>), [ai-personalization](<https://devfeed.tech/tags/ai-personalization.md>), [conversational-ai](<https://devfeed.tech/tags/conversational-ai.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-hc](<https://devfeed.tech/tags/cs-hc.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [research](<https://devfeed.tech/tags/research.md>), [trust](<https://devfeed.tech/tags/trust.md>)

### AI overview

In a fictional scenario with 407 participants who had no prior opinions or knowledge, a neutral, non-contextualized AI assistant arguing against expert recommendations was most persuasive. Contextualization and conversational warmth each reduced persuasiveness separately, while their combination increased it. Participants relied on the assistant in every condition, and neither cue detectably changed reliance. Trust predicted persuasion and reliance, but did not mediate effects of either cue. Higher self-reported AI literacy was associated with greater reliance without greater trust. The findings suggest that persuasion and reliance can diverge, and that users may defer to an AI assistant over an expert even without personalization. The evidence is limited to the described fictional experiment.

### Source excerpt

arXiv:2605.31275v2 Announce Type: replace-cross Abstract: Contextualization is a form of personalization in which the message is framed in a context meaningful to the recipient, using their interests and background. Personalization is a known persuasive strategy of artificial intelligence (AI) systems in politics and marketing. However, the persuasive effect of contextualization in everyday tasks, where users often lack prior knowledge, remains unclear. We ran a $2\times2$ between-subjects experiment ($N = 407$) in a fictional scenario where participants had no prior opinions or knowledge. We examined how contextualization, combined with conversational warmth, shapes the persuasive power of an AI assistant arguing against expert recommendations and users' reliance on it. Our findings show that the neutral, non-contextualized assistant was the most persuasive. Contextualization and warmth each reduced persuasiveness on their own, but their interaction increased it. Reliance was present in all conditions, and neither cue changed it detectably. Trust predicted persuasion and reliance, yet neither contextualization nor warmth had an indirect effect through trust. Participants with higher self-reported AI literacy relied more on the assistant without trusting it more. Our study extends AI personalization research from political opinions to everyday decisions made without prior knowledge. The results suggest that users are more likely to be persuaded by, and defer to, an AI assistant than an expert, even without personalization. AI persuasion experiments should therefore include untailored assistants and measure decisions; the cues that changed persuasion did not detectably change reliance.

## False Reassurance in LLM Severe-Weather Guidance: A Replayable Evaluation Against Real NWS Alerts

DevFeed: [False Reassurance in LLM Severe-Weather Guidance: A Replayable Evaluation Against Real NWS Alerts](<https://devfeed.tech/articles/false-reassurance-in-llm-severe-weather-guidance-a-replayable-evaluation-against-real-nws-alerts-117582.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05290>)

Author: Kadharmoideen Fadurudeen

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [alarm](<https://devfeed.tech/tags/alarm.md>), [alerts](<https://devfeed.tech/tags/alerts.md>), [claude](<https://devfeed.tech/tags/claude.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [cs-hc](<https://devfeed.tech/tags/cs-hc.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [disclosure](<https://devfeed.tech/tags/disclosure.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [guidance](<https://devfeed.tech/tags/guidance.md>), [llm](<https://devfeed.tech/tags/llm.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [regex](<https://devfeed.tech/tags/regex.md>), [token](<https://devfeed.tech/tags/token.md>), [warning](<https://devfeed.tech/tags/warning.md>), [weather](<https://devfeed.tech/tags/weather.md>)

### AI overview

In this replay study of 117 cases based on real NWS alerts, access to alert data eliminated observed false reassurance in the tested Claude-based pipeline: rates were 0/69 with alerts and a neutral prompt versus 26/69 (37.7%) when blind. Telling the blind model to disclose its limitation did not help (31/69, 44.9%); all false-reassurance responses were implicit. Alert access also had a tradeoff: under a neutral prompt, the model raised some alarm on 87.5% of no-alert controls, compared with 0% under the production prompt, and one production instruction explained part of that difference. An automated regex measure missed 53 of 57 labeled cases. The study also reports ingestion of NWS test products as real alerts and token-budget failures on complex cases. Findings are from the described replay evaluation; one rater labeled responses, and the authors disclose that the seven-category scheme followed an initial binary pass.

### Source excerpt

arXiv:2611.05290v1 Announce Type: cross Abstract: An LLM assistant that cannot see official severe-weather alerts may tell users that conditions are calm while a warning is active. We call this false reassurance and measure it by replaying 117 frozen cases built from real U.S. National Weather Service (NWS) alerts (69 alert cases, 32 no-alert controls, 16 non-U.S. locations) through five conditions of a Claude-based preparedness pipeline that differ in alert access and prompt design. One rater hand-labeled all 342 alert-case responses under a seven-category scheme written after an initial binary pass, which we disclose. False reassurance was 0.0% for the production pipeline (0/66, 95% CI 0.0-5.5) and for the same alerts under a neutral prompt (0/69), 37.7% for a blind neutral prompt (26/69), and 44.9% when the blind prompt also told the model to disclose its blind spot (31/69). Every case was implicit; no response explicitly denied an alert. Real alerts removed false reassurance (paired McNemar p < 0.0001); the disclosure instruction did not (p = 0.40). An automated regex measure suggested the opposite and missed 53 of 57 true cases, so we do not use it. Alert data alone was not enough for safe behavior: with a neutral prompt the model raised some alarm on 87.5% of no-alert controls, against 0% under the production prompt, and a preregistered single-sentence ablation showed that one production instruction explains part of that gap. We also report two failures common to similar systems, NWS test products ingested as real alerts and a token budget that fails on complex cases, and we release the dataset, labels and code.

## Usable String Matching by Humans through Overlaid Presentation

DevFeed: [Usable String Matching by Humans through Overlaid Presentation](<https://devfeed.tech/articles/usable-string-matching-by-humans-through-overlaid-presentation-109060.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.39084>)

Author: Enka Blanchard, Ted Selker, Alan T. Sherman, Levi Gabasova, Mehdi Bouhamidi, Ryan Robucci

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [human computer interaction](<https://devfeed.tech/topics/human-computer-interaction.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-hc](<https://devfeed.tech/tags/cs-hc.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [presentation](<https://devfeed.tech/tags/presentation.md>), [string-matching](<https://devfeed.tech/tags/string-matching.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

In two online experiments with 178 participants, overlay presentation made string comparisons substantially faster than adjacent presentation and improved accuracy for long strings. The researchers varied string length, whether pairs matched or differed, alignment, and presentation format. Despite the gains, mean error rates remained at least 4%, so the method may still be too inaccurate for some verification tasks.

### Source excerpt

arXiv:2610.39084v1 Announce Type: new Abstract: We explore the ability of humans to compare strings, an error-prone activity which occurs in critical contexts (e.g., bank wire transfers), where it is not always possible to trust a machine to help. We hypothesize that understanding human cognitive and perceptual limitations can improve how people verify whether two character strings are identical. We vary the strings' relative positions and presentation formats to measure comparison speed and accuracy. We present results from two online experiments with 178 human subjects who compared pairs of strings to determine whether the two strings were identical. The strings varied by length (5 characters to small paragraphs), difference (matching versus making a random change or a hard-to-detect one), alignment (vertical versus horizontal), and presentation (Overlay versus Adjacent). Our experiments test whether, by encoding and memory tasks into perceptual tasks, overlay formats can offer advantages in speed and accuracy. Results show that overlay offers a substantially faster method, with improved accuracy for long strings. Nevertheless, mean error rates were at least 4 percent, indicating that even the improved approach may be insufficiently accurate for many verification tasks.

## A Smart Instant Book Filter Model based on Dual-Objective Optimization

DevFeed: [A Smart Instant Book Filter Model based on Dual-Objective Optimization](<https://devfeed.tech/articles/a-smart-instant-book-filter-model-based-on-dual-objective-optimization-103110.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.36287>)

Author: Yi Hou, Li Fan, Huizhong Duan

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Operations research and optimization](<https://devfeed.tech/topics/operations-research-and-optimization.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>)

Tags: [a-b-testing](<https://devfeed.tech/tags/a-b-testing.md>), [adoption](<https://devfeed.tech/tags/adoption.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [booking](<https://devfeed.tech/tags/booking.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [filter](<https://devfeed.tech/tags/filter.md>), [instant](<https://devfeed.tech/tags/instant.md>), [model](<https://devfeed.tech/tags/model.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [production](<https://devfeed.tech/tags/production.md>), [search](<https://devfeed.tech/tags/search.md>), [system](<https://devfeed.tech/tags/system.md>)

### AI overview

The smart instant book filter aims to increase instant bookings while keeping overall booking rate from decreasing. The abstract says broadly promoting instant book listings could hurt bookings by surfacing low-quality listings, so the system applies the filter selectively to shift eligible request-to-book listings toward instant booking. In global-user A/B tests, it reportedly outperformed earlier human-expert-tuned models on both instant book rate and overall booking rate, and improved host instant book adoption. The system is described as deployed in production and serving millions of users daily; the excerpt provides no implementation details or numerical results.

### Source excerpt

arXiv:2610.36287v1 Announce Type: new Abstract: Hospitality service in online marketplaces has become increasingly popular over the past few years, not only because it helps people easily rent short-term lodging but also because it empowers travelers to connect with locals and feel belonging. A typical pattern of these industries is a central platform mediating the economic interactions between guests and hosts, where a successful booking experience is the key to user satisfaction. However, one of the biggest challenges is the possible delay of response and chance of rejection when guests try to book a request-to-book listing. Previously, we enabled the instant book feature to allow hosts to instantly accept qualified guests and guests to confirm bookings immediately. Yet the symptom has not been alleviated given mixed request-to-book and instant book listing supply in search results. One possible solution is to promote all instant book listings in search results; however, booking rate will be damaged due to some improperly promoted low quality listings. In this paper, we present a novel smart instant book filter system that applies the instant book filter intelligently and shifts request-to-book to instant book without hurting booking rate. Compared with traditional two-sided markets research that targets finding strategies to maximize a single overall social welfare goal, our proposed system aims to maximize the instant book rate with the constraint that booking rate should not decrease, forming a dual-objective optimization problem. We show A/B testing results for our system exposed to global users, demonstrating that our model performs better than previous human-expert-tuned models in terms of both instant book rate and overall booking rate. We also achieved improved metrics such as the host instant book adoption rate. The proposed system has been deployed to production and is serving millions of users every day.

## When Can a Learned Design Surrogate Be Trusted in a Self-Driving Microfluidics Laboratory? An Offline Closed-Loop Framework Built and Tested on Six Public Tables

DevFeed: [When Can a Learned Design Surrogate Be Trusted in a Self-Driving Microfluidics Laboratory? An Offline Closed-Loop Framework Built and Tested on Six Public Tables](<https://devfeed.tech/articles/when-can-a-learned-design-surrogate-be-trusted-in-a-self-driving-microfluidics-laboratory-an-offline-closed-loop-framework-built-and-tested-on-six-public-tables-103949.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.36395>)

Author: Zhiyuan Zhang

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [ML model versioning](<https://devfeed.tech/topics/ml-model-versioning.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>)

Tags: [3](<https://devfeed.tech/tags/3.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [acquisition](<https://devfeed.tech/tags/acquisition.md>), [adapting](<https://devfeed.tech/tags/adapting.md>), [all](<https://devfeed.tech/tags/all.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [beats](<https://devfeed.tech/tags/beats.md>), [better](<https://devfeed.tech/tags/better.md>), [boosting](<https://devfeed.tech/tags/boosting.md>), [budget](<https://devfeed.tech/tags/budget.md>), [can](<https://devfeed.tech/tags/can.md>), [chemistry](<https://devfeed.tech/tags/chemistry.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [design](<https://devfeed.tech/tags/design.md>), [driving](<https://devfeed.tech/tags/driving.md>), [framework](<https://devfeed.tech/tags/framework.md>), [laboratory](<https://devfeed.tech/tags/laboratory.md>), [offline](<https://devfeed.tech/tags/offline.md>), [public](<https://devfeed.tech/tags/public.md>), [self-driving](<https://devfeed.tech/tags/self-driving.md>), [tables](<https://devfeed.tech/tags/tables.md>)

### AI overview

Across six public droplet-microfluidic and liposome tables evaluated offline, warm-start residual boosting recovered surrogate performance within 20 target measurements in 94% of chemistry tiers, 90% of laboratory tiers, and 40% of device tiers; chemistry recovery fell to 27% under a strict noise-floor reading. In closed-loop simulations, proposing all at once and warm-starting after each measurement reached a 10% target in 2.80 measurements versus 3.31 for the reference, and in 5.72 versus 11.35 for random proposals on 301 hard units at 3% tolerance. These gains survived drift and shuffled-label controls and were concentrated in sparsely sampled tiers. No tested label-free acquisition rule survived shuffled-label controls, no interval-aware proposal or alternative model family performed better, and no diagnostic threshold beat continuing once escalation was priced. The work is entirely offline and claims no wet-lab validation; its results do not establish that any label-free statistic can certify individual candidates.

### Source excerpt

arXiv:2610.36395v1 Announce Type: new Abstract: Self-driving laboratories couple a learned design surrogate to an instrument, and on every new target domain it must answer four questions: when it can be trusted, how many target measurements it needs, which to make first, and when to stop adapting, switch models or escalate. We state a framework making each measurable (a competence ratio, a budget $k^*(\varepsilon)$, a time-to-target and a priced decision) and instantiate it without wet experiments on six public droplet-microfluidic and liposome tables as offline laboratories: 107 leave-one-group tiers whose rows are hidden labels. A sealed, pre-registered programme (two sealed negative validations and a pre-registered test on three more tables) and seven frozen closed-loop protocols, each run twice, yield 23 findings. Shift cost is stratified by shift type; no label-free statistic certifies individual candidates; random-split model-family rankings predicted neither the sealed nor the loop rankings. Warm-start residual boosting returns a surrogate to $1.25\times$ its development reference within 20 target rows in 94% of chemistry tiers (27% under a strict noise-floor reading), 90% of laboratory and 40% of device tiers, and a budget planned from the shift type on other tables is not measurably worse on a new one. In the loop, one gain survives drift and shuffled-label controls on all tiers, carried by the sparsely sampled ones: proposing at once and warm-starting on every measurement reaches a 10% target in 2.80 measurements against 3.31 for the reference, and in 5.72 against 11.35 for random proposals on 301 hard units at a 3% tolerance. No label-free acquisition rule survives the shuffled-label control, no interval-aware proposal or other model family does better, and no threshold rule on the tested diagnostics beats always continuing once escalation is priced. No wet-lab validation is claimed.

## 17 landing page examples to help you increase conversions

DevFeed: [17 landing page examples to help you increase conversions](<https://devfeed.tech/articles/17-landing-page-examples-to-help-you-increase-conversions-102445.md>)

Original publisher: [Read original article](<https://zapier.com/blog/landing-page-examples>)

Author: Katie Paterson

Published: 2026-10-26T05:00:00Z

Content type: article

Language: en

Sources: [Zapier](<https://devfeed.tech/sources/zapier.md>)

Topics: [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [User interface design](<https://devfeed.tech/topics/ui-design.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-tool](<https://devfeed.tech/tags/ai-tool.md>), [airbnb](<https://devfeed.tech/tags/airbnb.md>), [android](<https://devfeed.tech/tags/android.md>), [app](<https://devfeed.tech/tags/app.md>), [apple](<https://devfeed.tech/tags/apple.md>), [apple-pay](<https://devfeed.tech/tags/apple-pay.md>), [automate](<https://devfeed.tech/tags/automate.md>), [automation](<https://devfeed.tech/tags/automation.md>), [b2b-saas](<https://devfeed.tech/tags/b2b-saas.md>), [data](<https://devfeed.tech/tags/data.md>), [examples](<https://devfeed.tech/tags/examples.md>), [grow-your-business](<https://devfeed.tech/tags/grow-your-business.md>), [marketing](<https://devfeed.tech/tags/marketing.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

An effective landing page focuses visitors on one action, and the examples show varied ways to reduce friction or build interest: free tools and lead magnets, app QR codes, customer reviews, interactive demos and quizzes, personalized estimates, clear CTAs, and readable design. The article recommends matching tactics to the offer and audience; for example, multiple CTAs may suit a broad product range, while direct product access can work for easy-to-use software. It also stresses testing CTA options and adapting pages based on performance and audience response. Zapier is presented as automating follow-up after a form or button action, routing and updating leads across connected apps, with AI used for steps requiring judgment.

### Source excerpt

Any time I publish a landing page, I'm immediately thinking about what I could do next to make it better. How can I optimize the page to pull in more visitors? What new things can I add to see if we can improve conversions? Managing a successful landing page involves constantly testing new ideas, keeping an eye on the data, and adapting to whatever seems to resonate with your target audience. It's the beauty (or the drag, depending on how you look at it!) of online marketing. To inspire your nex

## Best practices for scaling A/B testing with experiment protocols

DevFeed: [Best practices for scaling A/B testing with experiment protocols](<https://devfeed.tech/articles/best-practices-for-scaling-a-b-testing-with-experiment-protocols-101076.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/best-practices-for-scaling-ab-testing-with-experiment-protocols/>)

Author: Eddie Cai; Lukas Goetz-Weiss

Published: 2026-10-26T00:00:00Z

Content type: tutorial

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog-the-monitor-blog.md>)

Topics: [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [experiments](<https://devfeed.tech/topics/experiments.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>)

Tags: [a-b-testing](<https://devfeed.tech/tags/a-b-testing.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [protocols](<https://devfeed.tech/tags/protocols.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [teams](<https://devfeed.tech/tags/teams.md>)

### AI overview

Datadog Experiments protocols turn an approved experiment methodology into reusable defaults for metrics, randomization, exposure, duration, analysis, and notifications. Published protocols prefill new experiment drafts; copied settings remain editable, and later protocol changes affect only new experiments. Define a primary outcome and secondary guardrails before launch, set acceptable thresholds, and choose stable subject identifiers and consistent assignment inputs across services. Staged exposure can limit risk, while duration should span a full business cycle. Review diagnostics before interpreting results: traffic imbalance is a blocker, mixed assignments are excluded from analysis, and dimensional or segment warnings can reveal localized problems. Agree on the analysis plan in advance: sequential frequentist analysis supports repeated checks, fixed-sample analysis requires a preset sample or duration, and Bayesian analysis needs its own decision criteria. CUPED can reduce variance when representative pre-experiment data is informative, but it does not correct biased randomization. Use separate protocols when recurring designs or decision processes differ.

### Source excerpt

Learn how to standardize A/B testing at scale with experiment protocols. Define metrics, randomization, exposure, and analysis, then reuse them across teams.

## Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals

DevFeed: [Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals](<https://devfeed.tech/articles/behind-aggregate-scores-what-systematically-varying-arena-hard-s-llm-judge-reveals-98073.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.30512>)

Author: Louise Yorke

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Temperature settings in AI](<https://devfeed.tech/topics/temperature-settings-in-ai.md>), [AI Prompt Templates](<https://devfeed.tech/topics/ai-prompt-templates.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [arena](<https://devfeed.tech/tags/arena.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [format](<https://devfeed.tech/tags/format.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [instructions](<https://devfeed.tech/tags/instructions.md>), [llm](<https://devfeed.tech/tags/llm.md>), [models](<https://devfeed.tech/tags/models.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [repeat](<https://devfeed.tech/tags/repeat.md>), [responses](<https://devfeed.tech/tags/responses.md>), [score](<https://devfeed.tech/tags/score.md>), [setup](<https://devfeed.tech/tags/setup.md>), [stable](<https://devfeed.tech/tags/stable.md>), [temperature](<https://devfeed.tech/tags/temperature.md>), [version](<https://devfeed.tech/tags/version.md>)

### AI overview

Across 500 Arena-Hard v2.0 prompts, GPT-4.1's win/tie/loss judgments changed substantially even when aggregate scores stayed similar. Repeating the default setup shifted its score by about one point while changing outcomes on 21.2% of prompts. Changing instructions produced further sensitivity: a setup omitting reasoning and several other instructions changed outcomes on 28.2 percentage points more prompts than unchanged reruns did on average and lowered the score by 4.09 points; an accuracy-prioritizing setup changed outcomes by 6.7 points more while barely moving the aggregate score. The authors conclude that aggregate scores alone do not establish prompt-level repeatability. For evaluations using individual verdicts, they recommend repeating each setup and reporting both aggregate scores and prompt-level agreement.

### Source excerpt

arXiv:2610.30512v1 Announce Type: cross Abstract: LLM judges are used to determine which of two answers is better. Trusting these verdicts requires knowing how repeatable they are and how much they depend on the judge's instructions. We show that similar aggregate scores can conceal substantial changes in individual verdicts, both when the same setup is repeated and when the instructions change. Using 500 Arena-Hard v2.0 prompts with two models' answers held fixed, we run GPT-4.1 as the judge under all eight combinations of criteria, output format and task framing, and repeat every setup unchanged at temperature zero on a pinned model version. Rerunning the default setup moves the aggregate score by about one point but changes the win/tie/loss outcome on 21.2% of prompts. Instruction changes introduce further sensitivity: in one setup, which asks for the verdict without reasoning and also drops several other instructions, outcomes change on 28.2 percentage points more prompts than the two setups' unchanged reruns do on average, and the aggregate score falls by 4.09 points. In another, where the criteria prioritise accuracy, outcomes change on 6.7 percentage points more prompts by the same comparison, while the aggregate score barely moves. A stable aggregate score therefore does not establish that individual verdicts are repeatable. This distinction matters when those verdicts are used as preference labels or to assess individual responses. Evaluations that rely on individual verdicts should repeat each setup they use and report both aggregate scores and prompt-level agreement.

## Does Agentic RAG Actually Retrieve Better Evidence? I Tested It on 298 Questions

DevFeed: [Does Agentic RAG Actually Retrieve Better Evidence? I Tested It on 298 Questions](<https://devfeed.tech/articles/does-agentic-rag-actually-retrieve-better-evidence-i-tested-it-on-298-questions-95958.md>)

Original publisher: [Read original article](<https://towardsdatascience.com/does-agentic-rag-actually-retrieve-better-evidence-i-tested-it-on-298-questions/>)

Author: Abdullahi Dattijo

Published: 2026-10-22T15:30:00Z

Content type: news

Language: en

Sources: [Towards Data Science](<https://devfeed.tech/sources/towards-data-science.md>)

Topics: [AI search](<https://devfeed.tech/topics/ai-search.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [Agentic AI](<https://devfeed.tech/topics/what-is-agentic-ai.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-rag](<https://devfeed.tech/tags/agentic-rag.md>), [agentic-retrieval](<https://devfeed.tech/tags/agentic-retrieval.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-coding](<https://devfeed.tech/tags/ai-coding.md>), [ai-coding-assistant](<https://devfeed.tech/tags/ai-coding-assistant.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-2-0](<https://devfeed.tech/tags/apache-2-0.md>), [api](<https://devfeed.tech/tags/api.md>), [api-key](<https://devfeed.tech/tags/api-key.md>), [apis](<https://devfeed.tech/tags/apis.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [bootstrap](<https://devfeed.tech/tags/bootstrap.md>), [cache](<https://devfeed.tech/tags/cache.md>), [editor-s-picks](<https://devfeed.tech/tags/editor-s-picks.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [rag](<https://devfeed.tech/tags/rag.md>), [search](<https://devfeed.tech/tags/search.md>)

### AI overview

On 298 questions about 2,000 recent arXiv papers, an agent allowed up to three searches did not clearly improve top-five retrieval over one-shot dense search: it recovered some missed papers but lost others, and the estimated gain was too uncertain to distinguish from no effect. A reranker that rescored the first 50 results improved rank quality, though not top-five recall clearly; it used no language model calls and added about 1.5 seconds on the author's laptop. The agent stopped after one search on about 59% of questions and used all three on about 26%, adding model cost and delay. The test covered retrieval only, used long questions generated from abstracts, one collection, and one agent model and instruction set, so the results may not generalize. The author suggests measuring baseline misses, trying reranking, and inspecting agent rewrites for unsupported added terms.

### Source excerpt

I let an AI agent search again whenever it chose. It found some papers plain search had missed, lost some plain search had found, and gained too little to trust. The post Does Agentic RAG Actually Retrieve Better Evidence? I Tested It on 298 Questions appeared first on Towards Data Science.

## Why Better Search Recall Did Not Mean Better RAG Answers

DevFeed: [Why Better Search Recall Did Not Mean Better RAG Answers](<https://devfeed.tech/articles/why-better-search-recall-did-not-mean-better-rag-answers-92553.md>)

Original publisher: [Read original article](<https://hackernoon.com/why-better-search-recall-did-not-mean-better-rag-answers?source=rss>)

Author: Haoxiang Li

Published: 2026-10-21T07:56:27Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [Multi-step AI processes](<https://devfeed.tech/topics/multi-step-ai-processes.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-assistance](<https://devfeed.tech/tags/ai-assistance.md>), [ai-assisted](<https://devfeed.tech/tags/ai-assisted.md>), [api](<https://devfeed.tech/tags/api.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [bm25](<https://devfeed.tech/tags/bm25.md>), [bootstrap](<https://devfeed.tech/tags/bootstrap.md>), [case-study](<https://devfeed.tech/tags/case-study.md>), [codex](<https://devfeed.tech/tags/codex.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [continuation](<https://devfeed.tech/tags/continuation.md>), [csv](<https://devfeed.tech/tags/csv.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [information-retrieval](<https://devfeed.tech/tags/information-retrieval.md>), [interfaces](<https://devfeed.tech/tags/interfaces.md>), [rag](<https://devfeed.tech/tags/rag.md>), [rag-evaluation](<https://devfeed.tech/tags/rag-evaluation.md>), [rag-performance](<https://devfeed.tech/tags/rag-performance.md>), [research](<https://devfeed.tech/tags/research.md>), [retrieval-recall](<https://devfeed.tech/tags/retrieval-recall.md>), [search](<https://devfeed.tech/tags/search.md>), [software-development](<https://devfeed.tech/tags/software-development.md>), [sqlite-fts5](<https://devfeed.tech/tags/sqlite-fts5.md>), [tool](<https://devfeed.tech/tags/tool.md>)

### AI overview

In a historical paired test on 230 BrowseComp-Plus questions, the project's search/read interface answered 217 correctly (94.35%), versus 194 (84.35%) for standard search, even though standard search had higher search-only reference-document recall (89.71% versus 86.12%). Both used the same BM25 implementation; the project interface returned excerpts with IDs and offsets and offered pagination, within-document search, and continued reading, while standard search returned five results with document openings capped at 512 tokens. The recall metric counted document IDs, not whether decisive passages reached the model. The comparison bundles interface, returned text, controls, and prompts, so it does not isolate the cause of the accuracy difference. Questions were used during development, the interval does not cover selection effects or repeated-generation variability, and equal call caps do not establish equal token use or cost. The result does not show BM25 universally outperforming dense retrieval.

### Source excerpt

Same BM25, two tool interfaces, 230 questions: higher search-only recall did not mean more correct answers. Results, research context, and limitations.

## More pentesting, same budget, no surprises

DevFeed: [More pentesting, same budget, no surprises](<https://devfeed.tech/articles/more-pentesting-same-budget-no-surprises-90102.md>)

Original publisher: [Read original article](<https://www.aikido.dev/blog/more-pentesting-same-budget-ai>)

Author: Mathias Deloof

Published: 2026-10-19T09:00:00Z

Content type: article

Language: en

Sources: [Aikido Security's Blog](<https://devfeed.tech/sources/aikido-security-s-blog.md>)

Topics: [Security](<https://devfeed.tech/topics/security.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-pentesting](<https://devfeed.tech/tags/ai-pentesting.md>), [application-security](<https://devfeed.tech/tags/application-security.md>), [attack-paths](<https://devfeed.tech/tags/attack-paths.md>), [costs](<https://devfeed.tech/tags/costs.md>), [pentesting](<https://devfeed.tech/tags/pentesting.md>), [pricing](<https://devfeed.tech/tags/pricing.md>), [security](<https://devfeed.tech/tags/security.md>)

### AI overview

AI-assisted coding is accelerating software releases, but many organizations still test security on periodic schedules. The article argues that AI pentesting can make more frequent, risk-based testing feasible by reducing test time and cost, while upfront pricing and spending controls help teams plan their budgets. It describes continuous testing triggered by security-relevant code changes and notes that specialist-led assessments remain important for complex systems and high-assurance needs.

### Source excerpt

AI pentesting lets security teams test more often without raising the budget. Learn how upfront pricing and forecasts keep costs predictable. Category: News

## Chirp Pong

DevFeed: [Chirp Pong](<https://devfeed.tech/articles/chirp-pong-88269.md>)

Original publisher: [Read original article](<https://simonwillison.net/2026/Oct/15/chirp-pong/>)

Author: Simon Willison

Published: 2026-10-15T19:47:52Z

Content type: article

Language: en

Sources: [Simon Willison's Weblog](<https://devfeed.tech/sources/simon-willison-s-weblog.md>)

Topics: [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [Generative AI](<https://devfeed.tech/topics/generative-ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-2-283](<https://devfeed.tech/tags/ai-2-283.md>), [ai-llms](<https://devfeed.tech/tags/ai-llms.md>), [api](<https://devfeed.tech/tags/api.md>), [audio](<https://devfeed.tech/tags/audio.md>), [build](<https://devfeed.tech/tags/build.md>), [chirp](<https://devfeed.tech/tags/chirp.md>), [claude](<https://devfeed.tech/tags/claude.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [claude-code-130](<https://devfeed.tech/tags/claude-code-130.md>), [claude-opus-5](<https://devfeed.tech/tags/claude-opus-5.md>), [code](<https://devfeed.tech/tags/code.md>), [game](<https://devfeed.tech/tags/game.md>), [game-design](<https://devfeed.tech/tags/game-design.md>), [game-design-25](<https://devfeed.tech/tags/game-design-25.md>), [gameplay](<https://devfeed.tech/tags/gameplay.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [generative-ai-2-025](<https://devfeed.tech/tags/generative-ai-2-025.md>), [html](<https://devfeed.tech/tags/html.md>), [llms](<https://devfeed.tech/tags/llms.md>), [llms-1-991](<https://devfeed.tech/tags/llms-1-991.md>), [need](<https://devfeed.tech/tags/need.md>), [networking](<https://devfeed.tech/tags/networking.md>), [opus-5](<https://devfeed.tech/tags/opus-5.md>), [opus-5-5](<https://devfeed.tech/tags/opus-5-5.md>), [other](<https://devfeed.tech/tags/other.md>)

### AI overview

A brief game experiment explores whether two phones can play a two-player game using chirps sent through their microphones to share game state, with the game running in static HTML. The experiment was prompted with Claude Opus 5.5 Extra in Claude Code.

### Source excerpt

Tool: Chirp Pong Another game experiment - I wanted to see if it was feasible to have a two player game on two separate phones running in static HTML where the networking code uses chirps sent via the microphone. It seems to work! Build a two player competitive pong like game where two players use their phones close to each other and the game state is sent via audio noises which the web microphone API can interpret - but add graphical and gameplay twists to make it a lot more fun and intertwine than pong Claude Opus 5.5 Extra in Claude Code. Tags: game-design, ai, generative-ai, llms, claude-code

## Benchmarking AI decision models against traditional guardrails

DevFeed: [Benchmarking AI decision models against traditional guardrails](<https://devfeed.tech/articles/benchmarking-ai-decision-models-against-traditional-guardrails-63799.md>)

Original publisher: [Read original article](<https://developers.redhat.com/articles/2026/10/02/benchmarking-ai-decision-models-against-traditional-guardrails>)

Author: Dr. Rob Geada, Dr. Mac Misiura, Shelton Cyril

Published: 2026-10-02T13:01:51Z

Content type: article

Language: en

Sources: [Red Hat](<https://devfeed.tech/sources/red-hat.md>), [Red Hat Developer](<https://devfeed.tech/sources/red-hat-developer.md>)

Topics: [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [Factuality scoring in AI](<https://devfeed.tech/topics/factuality-scoring-in-ai.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [guardrails](<https://devfeed.tech/tags/guardrails.md>), [platform-engineering](<https://devfeed.tech/tags/platform-engineering.md>), [security](<https://devfeed.tech/tags/security.md>), [typesafe-ai](<https://devfeed.tech/tags/typesafe-ai.md>)

### AI overview

The article benchmarks decision models such as Jev and DiffusionGemma against pretrained classifiers and LLM-as-a-judge guardrails for prompt-injection and content-safety tasks. It reports that pretrained classifiers remain competitive, while decision models do not reliably outperform the other approaches in speed or accuracy. Model performance varies by task, architecture, and prompt tuning.

### Source excerpt

As enterprise generative AI applications move to production, platform engineers face a key challenge: balancing the flexibility of LLM-as-a-judge guardrails with the reliability and portability of traditional classifiers that require custom training data. The recent emergence of "decision models"--highlighted by TypeSafe AI's recent announcement of Jev and "System One" models--promises a flexible middle ground by producing fixed "decisions" given a state and a list of questions rather than generating text. The post Benchmarking AI decision models against traditional guardrails appeared first on Red Hat Developer.

## Dear Software Makers

DevFeed: [Dear Software Makers](<https://devfeed.tech/articles/dear-software-makers-62924.md>)

Original publisher: [Read original article](<https://blog.jim-nielsen.com/2026/dear-software-makers/>)

Author: Jim Nielsen

Published: 2026-09-30T19:00:00Z

Content type: opinion

Language: en

Sources: [Jim Nielsen's Blog](<https://devfeed.tech/sources/jim-nielsen-s-blog.md>)

Topics: [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>)

### AI overview

Drawing on Marques Brownlee's concerns about YouTube video variants, the author argues that software makers should favor creative work and repeated releases over optimizing every decision for metrics such as engagement and efficiency.

### Source excerpt

Marques Brownlee just released a video essay titled "Dear YouTube". I loved it. He talks about how YouTube is going to start shipping this new feature where makers can essentially publish different "variants" of their videos and see which performs best (an A/B test). Which immediately begs a lot of questions. Brownlee voices his: If I'm watching a video, do I know if it's part of one of these A/B tests? If I'm reading the comments of a video, how do I know I saw the same thing as another commenter? How do linked timestamps work? It's pretty wild when you think about it. I mean, imagine sending someone a link, "Check out this cool video!" And at that point you're basically crossing your fingers, "I hope they see the same thing I did..." Marques says he spoke to some YouTube engineers about these questions and they didn't really have answers. They were kind of just like, "We don't know -- yolo! This'll ship soon." So his video is a kind of plea to YouTube creators everywhere: Don't use this feature! There's this sort of unspoken rule of community on YouTube, that like we're all having this same experience together -- we're all watching the same video. That is what makes it a cultural phenomenon is that we all saw the same thing [...] so letting people put up two or three different videos of the same thing at the same time kind of fundamentally breaks that, which doesn't feel right. I agree, 100%. Brownlee goes on to make a lot of great points in the video that, I think, can be applicable to making software as well. A few that stood out: The skill of being a "creator" is not about maximizing views, retention, etc. It's about finding creative ways to share stories, teach concepts, explore ideas, etc. Views, retention, etc. are all downstream of that. Making (and finishing) something over and over and over is a better use of your time than making one thing and then trying all kinds of variants and obsessing over their varied performance. When you focus on optimizing, you end u

## A problem well-defined is half solved: Measuring Hypothesis Quality at Booking.com

DevFeed: [A problem well-defined is half solved: Measuring Hypothesis Quality at Booking.com](<https://devfeed.tech/articles/a-problem-well-defined-is-half-solved-measuring-hypothesis-quality-at-booking-com-61615.md>)

Original publisher: [Read original article](<https://booking.ai/a-problem-well-defined-is-half-solved-measuring-hypothesis-quality-at-booking-com-a85e845178a3?source=rss----4d265f07defc---4>)

Author: Luka Beverin

Published: 2026-09-29T09:42:29Z

Content type: article

Language: en

Sources: [Booking.com Data Science](<https://devfeed.tech/sources/booking-com-data-science.md>)

Topics: [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [AI-enhanced product development](<https://devfeed.tech/topics/ai-enhanced-product-development.md>), [Tool](<https://devfeed.tech/topics/tool.md>)

Tags: [adoption](<https://devfeed.tech/tags/adoption.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ambassadors](<https://devfeed.tech/tags/ambassadors.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [behaviour](<https://devfeed.tech/tags/behaviour.md>), [booking](<https://devfeed.tech/tags/booking.md>), [change](<https://devfeed.tech/tags/change.md>), [experimentation](<https://devfeed.tech/tags/experimentation.md>), [featured](<https://devfeed.tech/tags/featured.md>), [hypothesis-testing](<https://devfeed.tech/tags/hypothesis-testing.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [measuring](<https://devfeed.tech/tags/measuring.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [problems](<https://devfeed.tech/tags/problems.md>), [quality](<https://devfeed.tech/tags/quality.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [records](<https://devfeed.tech/tags/records.md>), [reuse](<https://devfeed.tech/tags/reuse.md>), [template](<https://devfeed.tech/tags/template.md>), [test](<https://devfeed.tech/tags/test.md>), [theory](<https://devfeed.tech/tags/theory.md>), [tooling](<https://devfeed.tech/tags/tooling.md>)

### AI overview

Booking.com's advisory Hypothesis Quality Score uses an LLM to check whether experiment hypotheses state prior evidence, the change, target audience, expected user response and measurable effects. The in-house platform provides feedback during experiment creation, while a Quality Tab and product-track ambassadors help teams improve drafts. After rollout, the share of product improvement experiments rated Excellent rose from 62% to almost 90%; expected responses and observable metrics each gained 14 and 13 percentage points. The score measures how completely reasoning is documented, not idea value, experiment design or decision quality, and the reported gains do not prove the checker alone improved decisions. Booking.com keeps it advisory, pairs it with education and human review, and avoids automatic rewrites to encourage teams to clarify their own reasoning.

### Source excerpt

Even after an experiment (or A/B test) ends, it may not be clear what action to take or what was learned. The test may be properly powered, run for the right amount of time, and produce a statistically clear result. But if the hypothesis never made the original problem, reasoning for the change, or decision criteria explicit, the result is difficult to interpret. This is why hypothesis quality matters. A well-written hypothesis does more than improve an experiment at launch; it improves the quality of the decision that follows. By recording the evidence behind an idea, the user response we expect, and the metrics that would support or challenge it, teams create a clearer standard for interpreting the outcome. The benefits compound over time. A hypothesis that records only an implementation is difficult for another team (or an AI system), to find, compare, or reuse. A hypothesis that captures the underlying theory and expected behavioural response becomes a searchable explanation of why an experiment was run. Over time, this gives Booking.com a richer map of what we have learned: which user problems recur, which mechanisms tend to work, and where our assumptions repeatedly fail. In our earlier posts on experimentation quality and scaling that quality across Booking.com, we described how shared standards, measurement, education, and tooling can make better experimentation easier to practice at scale. This post focuses on the next layer of that work: how we are helping teams make the reasoning inside their hypotheses clearer, more consistent, and more useful long after an experiment has ended. The Hypothesis Template A hypothesis is not just a description of the feature we are about to build. It is the bridge between an observation, a theory about user behaviour, and the evidence that could support or challenge that theory. When that bridge is weak, the experiment may still run correctly, even score highly on other pillars of experimentation quality, but the result bec

## Helping personal agents shop more intelligently and reliably with Link

DevFeed: [Helping personal agents shop more intelligently and reliably with Link](<https://devfeed.tech/articles/helping-personal-agents-shop-more-intelligently-and-reliably-with-link-61977.md>)

Original publisher: [Read original article](<https://stripe.com/blog/helping-personal-agents-shop-more-intelligently-and-reliably-with-link>)

Author: Dan Hill

Published: 2026-09-29T00:00:00Z

Content type: release

Language: en

Sources: [Stripe](<https://devfeed.tech/sources/stripe-blog.md>)

Topics: [AI for customer service automation](<https://devfeed.tech/topics/ai-for-customer-service-automation.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [authorization](<https://devfeed.tech/tags/authorization.md>), [checkout](<https://devfeed.tech/tags/checkout.md>), [complete](<https://devfeed.tech/tags/complete.md>), [costs](<https://devfeed.tech/tags/costs.md>), [financial](<https://devfeed.tech/tags/financial.md>), [flight](<https://devfeed.tech/tags/flight.md>), [grok-bot](<https://devfeed.tech/tags/grok-bot.md>), [habits](<https://devfeed.tech/tags/habits.md>), [instinct](<https://devfeed.tech/tags/instinct.md>), [muse](<https://devfeed.tech/tags/muse.md>), [personal](<https://devfeed.tech/tags/personal.md>), [product](<https://devfeed.tech/tags/product.md>), [recommendations](<https://devfeed.tech/tags/recommendations.md>), [shop](<https://devfeed.tech/tags/shop.md>)

### AI overview

Link introduces payment features for personal shopping agents: incremental authorization for price changes, guidance for checkout obstacles, and financial insights for personalized product recommendations. The article also describes purchase protection, though the supplied text ends before detailing it.

### Source excerpt

As agents take on more purchases, agent builders have increasingly asked us to help agents navigate checkout and earn consumer trust. Today, we're sharing three major improvements to support this broader set of needs.

## Beyond A/B Testing: Designing Experiments When the Space Is Too Large to Explore

DevFeed: [Beyond A/B Testing: Designing Experiments When the Space Is Too Large to Explore](<https://devfeed.tech/articles/beyond-a-b-testing-designing-experiments-when-the-space-is-too-large-to-explore-61298.md>)

Original publisher: [Read original article](<https://building.nu.com/beyond-a-b-testing-designing-experiments-when-the-space-is-too-large-to-explore/>)

Author: Nubank Editorial

Published: 2026-09-28T14:21:31Z

Content type: tutorial

Language: en

Sources: [Building Nubank](<https://devfeed.tech/sources/building-nubank.md>), [Nubank](<https://devfeed.tech/sources/nubank.md>)

Topics: [experiments](<https://devfeed.tech/topics/experiments.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>)

Tags: [a-b-testing](<https://devfeed.tech/tags/a-b-testing.md>), [data-analytics](<https://devfeed.tech/tags/data-analytics.md>), [data-science-machine-learning](<https://devfeed.tech/tags/data-science-machine-learning.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [statistics](<https://devfeed.tech/tags/statistics.md>)

### AI overview

This practical guide explains how optimal design theory can reduce experiment runs when a decision space has many combinations or constraints. It introduces fractional factorial designs and compares them with more general optimal designs for irregular settings.

### Source excerpt

A practical guide to D-optimal designs, Fedorov's algorithm, and how business restrictions shape what we can test and estimate The post Beyond A/B Testing: Designing Experiments When the Space Is Too Large to Explore appeared first on Building Nubank.

[Next page](<https://devfeed.tech/topics/a-b-testing.md?cursor=WyIyMDI2LTA5LTI4VDE0OjIxOjMxKzAwOjAwIiwgIjRmNWExZWRiLTk1YTgtNDFkMi1iZjhkLTE4NGE5Mzc2MTFlOSJd>)