# sequential-testing

A statistical testing method that processes observations sequentially and makes real-time decisions until a stopping criterion is met.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Robustness Evaluation of GPU Cloud Pricing Benchmarks under Incomplete Observations: A Controlled Missingness Study

DevFeed: [Robustness Evaluation of GPU Cloud Pricing Benchmarks under Incomplete Observations: A Controlled Missingness Study](<https://devfeed.tech/articles/robustness-evaluation-of-gpu-cloud-pricing-benchmarks-under-incomplete-observations-a-controlled-missingness-study-135117.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17796>)

Author: Jooya Wong

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [article](<https://devfeed.tech/tags/article.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [composition](<https://devfeed.tech/tags/composition.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [daily](<https://devfeed.tech/tags/daily.md>), [days](<https://devfeed.tech/tags/days.md>), [error](<https://devfeed.tech/tags/error.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fail](<https://devfeed.tech/tags/fail.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-cloud](<https://devfeed.tech/tags/gpu-cloud.md>), [pricing](<https://devfeed.tech/tags/pricing.md>), [robustness](<https://devfeed.tech/tags/robustness.md>), [study](<https://devfeed.tech/tags/study.md>)

### AI overview

The study stress-tests an equal-weight median benchmark for daily GPU rental asking prices under four imposed patterns of missing quotes. Across 5,800 simulations, whole-source outages lowered publication coverage to 0.77 but yielded zero conditional error on dates published; composition-changing quote loss retained 0.86 coverage but caused 7.74 log points of error. A past-only, stratum-level ablation reduced error by 4.15-6.70 log points in five composition-changing cases, using richer information. An observable-signal withholding rule improved the primary error-coverage criterion in none of ten cases. These are controlled results on observed asking prices, not transaction prices or evidence about real-world missingness mechanisms.

### Source excerpt

arXiv:2611.17796v1 Announce Type: new Abstract: Daily GPU rental-price benchmarks can fail when marketplace quotes are missing. This article stress-tests a fixed equal-weight median rule under four imposed masking mechanisms and four benchmark-level recovery methods. Across 5,800 Monte Carlo realisations in 27 scenarios on 190 held-out days, whole-source outages reduce publication coverage to 0.77 while preserving zero conditional error on published dates; composition-changing loss retains 0.86 coverage but produces 7.74 log points of error. A separate past-only stratum-level ablation reduces error by 4.15 to 6.70 log points in five composition-changing cases, using a richer information set. An observable-signal withholding rule improves the primary error-coverage criterion in 0 of 10 cases. Results concern observed asking-price measurements, not transaction prices or identified real-world missingness mechanisms.

## Bet Imaginatively, not Historically in Independent-Data Sequential Testing

DevFeed: [Bet Imaginatively, not Historically in Independent-Data Sequential Testing](<https://devfeed.tech/articles/bet-imaginatively-not-historically-in-independent-data-sequential-testing-132996.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15456>)

Author: Nathaniel Xu, Feng Liu, Danica J. Sutherland

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>)

Tags: [adversarial](<https://devfeed.tech/tags/adversarial.md>), [betting](<https://devfeed.tech/tags/betting.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [independent](<https://devfeed.tech/tags/independent.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [learning](<https://devfeed.tech/tags/learning.md>), [online](<https://devfeed.tech/tags/online.md>), [performance](<https://devfeed.tech/tags/performance.md>), [power](<https://devfeed.tech/tags/power.md>), [problems](<https://devfeed.tech/tags/problems.md>), [sequential-testing](<https://devfeed.tech/tags/sequential-testing.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

FLOE (follow the leave-one-out e-power) is a betting strategy for sequential tests on independent, identically distributed data. Instead of basing bets on historical evidence as generic online learning methods do, it estimates a test's e-power by averaging over equally likely data sequences formed by shuffling the observations. The authors say a principled regularization technique makes this drop-in method substantially improve testing performance on kernel two-sample and independence problems; the supplied abstract gives no quantitative results.

### Source excerpt

arXiv:2611.15456v1 Announce Type: cross Abstract: Sequential testing avoids some of the many controversies and drawbacks related to traditional batch testing with p-values. In particular, the testing by betting paradigm has developed rapidly in recent years; it frames the level of evidence against the null as the wealth of a player in a betting game. Naturally, how much a player should bet is a key consideration, and should reflect their confidence in what they know about the next observed data point. Much current work uses standard betting strategies developed for adversarial problems. In the stochastic setting, where the observed data is independent and stationary, these approaches may be suboptimal. We propose a simple betting strategy, follow the leave-one-out e-power (FLOE), that exploits the i.i.d. nature of a data stream to estimate the e-power of our current test on the underlying distribution. We average not over the historical evidence collected so far, as generic online learning-based methods do, but instead over the equally-likely possible data sequences we might have seen based on shuffling the data. With an additional principled regularization technique, this simple drop-in method yields substantial improvements in testing performance on a variety of kernel two-sample and independence testing problems.

## Controlling the Probability of Unknown Unknowns in Pre-Deployment Testing of AI Systems

DevFeed: [Controlling the Probability of Unknown Unknowns in Pre-Deployment Testing of AI Systems](<https://devfeed.tech/articles/controlling-the-probability-of-unknown-unknowns-in-pre-deployment-testing-of-ai-systems-132177.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15835>)

Author: Rachel Li, Chris Hays, Aaditya Ramdas, Manish Raghavan

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>), [ai security](<https://devfeed.tech/topics/ai-security.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-systems](<https://devfeed.tech/tags/ai-systems.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [failure](<https://devfeed.tech/tags/failure.md>), [math-st](<https://devfeed.tech/tags/math-st.md>), [probability](<https://devfeed.tech/tags/probability.md>), [stat-th](<https://devfeed.tech/tags/stat-th.md>), [systems](<https://devfeed.tech/tags/systems.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

The method gives developers an anytime-valid upper confidence sequence for the probability that testing will reveal an unseen AI failure mode. They can repeatedly test and stop when the bound falls below a chosen threshold while retaining the statistical guarantee, making the test-versus-deploy tradeoff explicit. On SWE-Chat coding-agent traces, it produced tighter upper bounds than a multiple-testing-corrected Good-Turing baseline and outperformed a fixed-horizon Good-Turing bound, which lacks anytime validity. The authors also extend the method to account for tests drawn from a different task distribution and imperfect automated failure labels.

### Source excerpt

arXiv:2611.15835v1 Announce Type: new Abstract: In pre-deployment testing of AI systems such as AI agents, a developer can often deal with known failure modes (e.g., leaking API keys, deleting important databases, or installing a malicious package) by implementing fixes or making contingency plans for when the same issue might occur again. What is more challenging are the unknown unknowns: the unseen failure modes that haven't surfaced during testing. The decision to stop testing and deploy therefore faces an inherent tradeoff: continue testing to observe more behaviors, at the cost of time and resources, or stop testing and deploy, at the risk of failure modes remaining undiscovered. How then does the developer decide when they have tested enough? To answer this, we develop an algorithm that certifies when the probability of observing an unseen failure mode is sufficiently small. The algorithm constructs an anytime-valid upper confidence sequence on this quantity, so the developer can repeatedly test a system, inspect the bound, and decide to stop when it falls below a chosen threshold without invalidating the statistical guarantee. Our problem can be interpreted as a sequential variant of the missing-mass estimation problem which gave rise to the classic Good-Turing estimator (Good, 1953). On SWE-Chat, a dataset of real-world coding agent traces (Baumann et al., 2026), we implement our procedure and show that it achieves tighter upper bounds than a simple multiple-testing corrected Good-Turing baseline, and even dominates a fixed-horizon (non-multiple-testing-corrected) Good-Turing upper confidence bound, which is not anytime valid. Additionally, to better match realistic evaluation pipelines, we extend our method to handle sampling from a different task distribution and imperfect automated failure labeling.

## Time Series Change Point Detection with Gaussian Processes

DevFeed: [Time Series Change Point Detection with Gaussian Processes](<https://devfeed.tech/articles/time-series-change-point-detection-with-gaussian-processes-128685.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13295>)

Author: Olga Isupova, Danil Kuzin, Fredrik Gustafsson, Lyudmila Mihaylova

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Time Series](<https://devfeed.tech/topics/time-series.md>), [Temporal data](<https://devfeed.tech/topics/temporal-data.md>), [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>)

Tags: [change](<https://devfeed.tech/tags/change.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [detection](<https://devfeed.tech/tags/detection.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [framework](<https://devfeed.tech/tags/framework.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [hypothesis](<https://devfeed.tech/tags/hypothesis.md>), [hypothesis-testing](<https://devfeed.tech/tags/hypothesis-testing.md>), [math-st](<https://devfeed.tech/tags/math-st.md>), [multiple](<https://devfeed.tech/tags/multiple.md>), [offline](<https://devfeed.tech/tags/offline.md>), [online](<https://devfeed.tech/tags/online.md>), [performance](<https://devfeed.tech/tags/performance.md>), [series](<https://devfeed.tech/tags/series.md>), [specification](<https://devfeed.tech/tags/specification.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [stat-th](<https://devfeed.tech/tags/stat-th.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [theoretical](<https://devfeed.tech/tags/theoretical.md>), [time](<https://devfeed.tech/tags/time.md>), [time-series](<https://devfeed.tech/tags/time-series.md>)

### AI overview

The paper proposes a Gaussian process framework that detects time series changes as alterations in the prior's hyperparameters, which are learned from data. Framed as statistical hypothesis testing, it is demonstrated for offline and online detection of multiple change points. The abstract says the method does not require detailed specifications of the time series function before and after a change and reports theoretical guarantees and evaluations on synthetic and real data.

### Source excerpt

arXiv:2611.13295v1 Announce Type: cross Abstract: This paper proposes a novel general Gaussian process framework for change point detection in time series. A change is defined as an alteration of the hyperparameters of the Gaussian process prior. We formulate the problem in the context of statistical hypothesis testing and demonstrate its performance over both offline and online multiple change points detection. The approach provides theoretical guaranties of its performance. An important advantage of the developed framework is that it does not need a detailed specification of the time series function before and after a change. The hyperparameter values are learned from the data. Comprehensive evaluation of the proposed framework is provided both on synthetic and real data and its efficiency is demonstrated.

## pESTO: Iterative p-greedy kernel regression with risk-driven Early STOpping

DevFeed: [pESTO: Iterative p-greedy kernel regression with risk-driven Early STOpping](<https://devfeed.tech/articles/pesto-iterative-p-greedy-kernel-regression-with-risk-driven-early-stopping-123278.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.08555>)

Author: L. Bruni Bruno, P. Massa, E. Perracchione, M. Trombini

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>)

Tags: [approximation](<https://devfeed.tech/tags/approximation.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [error](<https://devfeed.tech/tags/error.md>), [estimator](<https://devfeed.tech/tags/estimator.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [greedy](<https://devfeed.tech/tags/greedy.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [noise](<https://devfeed.tech/tags/noise.md>), [power](<https://devfeed.tech/tags/power.md>), [regression](<https://devfeed.tech/tags/regression.md>), [risk](<https://devfeed.tech/tags/risk.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

pESTO adapts p-greedy kernel regression to noisy data by accounting separately for approximation error, governed by the power function, and noise amplification, represented by a Lebesgue-type term. The resulting bounds motivate using Stein's unbiased risk estimator (SURE) as a computable proxy for mean squared error and as a stopping criterion for how many nodes to select. The abstract reports numerical experiments indicating that this balances deterministic accuracy with statistical noise contributions.

### Source excerpt

arXiv:2611.08555v1 Announce Type: cross Abstract: Kernel-based greedy algorithms select suitable approximation nodes by minimizing point-wise error indicators, such as the power function; in this context, they are usually referred to as p-greedy approaches. In the presence of noisy data, deterministic criteria may fail, as the discrepancy between the true solution and its estimate is influenced by both approximation error and noise amplification, and the balance between the two is unknown. This study explicitly separates these two contributions, which are governed by the power function and a Lebesgue-type term, respectively. These bounds motivate the use of the Stein's unbiased risk estimator (SURE) as a proxy of the mean squared error, which in turn provides a principled and computable stopping criterion for the number of nodes to select in the p-greedy procedure. Numerical experiments demonstrate that this approach effectively balances deterministic accuracy and statistical noise contributions.

## Probably Safe Sim-to-Real Transfer with Lifetime Validity

DevFeed: [Probably Safe Sim-to-Real Transfer with Lifetime Validity](<https://devfeed.tech/articles/probably-safe-sim-to-real-transfer-with-lifetime-validity-123397.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09344>)

Author: Rohan Tangri, Jan-Peter Calliess

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [betting](<https://devfeed.tech/tags/betting.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [failure](<https://devfeed.tech/tags/failure.md>), [lifetime](<https://devfeed.tech/tags/lifetime.md>), [markov](<https://devfeed.tech/tags/markov.md>), [policies](<https://devfeed.tech/tags/policies.md>), [production](<https://devfeed.tech/tags/production.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [safety](<https://devfeed.tech/tags/safety.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [sim-to-real](<https://devfeed.tech/tags/sim-to-real.md>), [simulation](<https://devfeed.tech/tags/simulation.md>), [test](<https://devfeed.tech/tags/test.md>), [theoretical](<https://devfeed.tech/tags/theoretical.md>)

### AI overview

Dejavu certifies each simulation-trained reinforcement learning policy before deployment in a constrained Markov decision process, using only production trajectories gathered by a safe baseline policy. Its anytime-valid test combines per-decision importance sampling with a betting confidence sequence. The safety guarantee does not depend on simulation quality, though poor simulation quality can reduce production sample efficiency. Summable failure probabilities across tests provide lifetime validity over an unbounded sequence of policy updates. The abstract reports empirical support from three sim-to-real benchmarks.

### Source excerpt

arXiv:2611.09344v1 Announce Type: new Abstract: We present Dejavu, a method that makes sim-to-real transfer of reinforcement learning policies in constrained Markov decision processes probably safe over an unbounded sequence of policy updates. An anytime valid test using per-decision importance sampling and a betting confidence sequence certifies each candidate trained in simulation before deployment using only trajectories collected by a safe baseline policy operating in production. Simulation quality does not affect the safety guarantee, only the production sample efficiency of the method. Finally, a summable per-test failure probability yields the lifetime valid property. Empirical results across three sim-to-real benchmarks support the theoretical work.

## Sequential auditing of connected forecast errors under partial observations

DevFeed: [Sequential auditing of connected forecast errors under partial observations](<https://devfeed.tech/articles/sequential-auditing-of-connected-forecast-errors-under-partial-observations-116864.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.03928>)

Author: Yuanbo Zhou, Junjie Xiong

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Time Series](<https://devfeed.tech/topics/time-series.md>), [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>)

Tags: [acquisition](<https://devfeed.tech/tags/acquisition.md>), [auditing](<https://devfeed.tech/tags/auditing.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [errors](<https://devfeed.tech/tags/errors.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [forecast](<https://devfeed.tech/tags/forecast.md>), [graph](<https://devfeed.tech/tags/graph.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [masking](<https://devfeed.tech/tags/masking.md>), [metr](<https://devfeed.tech/tags/metr.md>), [queries](<https://devfeed.tech/tags/queries.md>), [random](<https://devfeed.tech/tags/random.md>), [time-series](<https://devfeed.tech/tags/time-series.md>), [traversal](<https://devfeed.tech/tags/traversal.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>)

### AI overview

The paper studies auditing whether a connected group of locations violates forecast intervals on the same side when some outcomes are hidden. An auditor selectively queries missing labels and stops once the event is determined or the query budget is exhausted. Across three public graph time series with 30% of labels hidden, MOMS uses 19.6%-38.5% fewer queries on average than screened random acquisition. Under localized masking on METR-LA, shortest witness finds more positive cases within ten queries, with results depending on node ordering. Direct component traversal reproduces all 9,696 screened random traces without enumerating connected subsets. The reported results also show that using fewer queries to reach a decision does not necessarily improve early event discovery or reduce local computation.

### Source excerpt

arXiv:2611.03928v1 Announce Type: new Abstract: Graph forecasting systems often rely on prediction intervals to quantify uncertainty across connected locations, but the true outcomes needed for evaluation may be only partially observed. We study how an auditor can selectively retrieve missing outcomes to determine whether there exists a connected group whose outcomes violate the same side of their respective forecast intervals. The audit stops when the observed labels determine the event or the query budget is exhausted. On three public graph time series, MOMS reduces mean queries by 19.6% to 38.5% relative to screened random acquisition when 30% of labels are hidden. Under localized masking on METR-LA, shortest witness finds more positive cases within ten queries, although the difference depends on node ordering. Direct component traversal reproduces all 9,696 screened random traces without enumerating connected subsets. The experiments show that reducing queries to a decision does not necessarily improve early event discovery or reduce local computation.

## AdapBench: Adaptive Early-Stopping and Prioritization for Efficient Microbenchmarking in Cloud-based CI/CD

DevFeed: [AdapBench: Adaptive Early-Stopping and Prioritization for Efficient Microbenchmarking in Cloud-based CI/CD](<https://devfeed.tech/articles/adapbench-adaptive-early-stopping-and-prioritization-for-efficient-microbenchmarking-in-cloud-based-ci-cd-113502.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.01255>)

Author: Sebastian Koch, Jonas Heisterberg, Nils Japke, Trever Schirmer, David Bermbach

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-coding](<https://devfeed.tech/tags/agentic-coding.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [change](<https://devfeed.tech/tags/change.md>), [ci-cd](<https://devfeed.tech/tags/ci-cd.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [execution](<https://devfeed.tech/tags/execution.md>), [false-positive-rate](<https://devfeed.tech/tags/false-positive-rate.md>), [go](<https://devfeed.tech/tags/go.md>), [go-projects](<https://devfeed.tech/tags/go-projects.md>), [java](<https://devfeed.tech/tags/java.md>), [microbenchmarking](<https://devfeed.tech/tags/microbenchmarking.md>), [online](<https://devfeed.tech/tags/online.md>), [overhead](<https://devfeed.tech/tags/overhead.md>), [performance](<https://devfeed.tech/tags/performance.md>), [prioritization](<https://devfeed.tech/tags/prioritization.md>), [runtime](<https://devfeed.tech/tags/runtime.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>)

### AI overview

AdapBench reduces microbenchmark suite runtime by prioritizing informative benchmarks and stopping each one once enough evidence supports accepting or rejecting a performance change. In evaluations on eight Java projects, it saved 42% of aggregate execution time while matching an optimistic uniform run's 90% change detection and false-positive rate; scheduling overhead was below 0.001% of runtime. On four Go projects, it detected every performance change with zero false positives while saving 85% of a full run.

### Source excerpt

arXiv:2611.01255v1 Announce Type: new Abstract: Cloud-based software systems frequently release new versions, a fact further accelerated through agentic coding. Each new release, however, brings the risk of performance regressions which can irritate end users, increase infrastructure costs, and increase carbon footprints. Therefore, developers strive to catch such regressions in testing stages, usually through so-called continuous benchmarking. The standard way of doing this is using microbenchmark suites which, however, take notoriously long to execute. Optimizing the execution of microbenchmark suites is, hence, an active field of research. In this paper, we propose AdapBench, an online scheduling system for reducing the execution duration of microbenchmark suites. AdapBench is the first approach which dynamically makes at-runtime decisions for prioritizing the most informative benchmarks and early-stopping each one once enough evidence has accumulated to accept or reject a performance change hypothesis. We evaluate our prototype on eight popular Java projects, commonly used by microbenchmark research, and four additional Go projects. At matched 90% change detections and false-positive rate compared to an optimistic uniform run, AdapBench saves 42% of the aggregate execution duration on the Java projects, with negligible scheduling overhead (below 0.001% of the total runtime). On the Go projects, AdapBench finds every performance change with zero false positives while saving 85% of the full run.

## ThinkJev: One Thought, Every Budget for Calibrated Typed Decisions

DevFeed: [ThinkJev: One Thought, Every Budget for Calibrated Typed Decisions](<https://devfeed.tech/articles/thinkjev-one-thought-every-budget-for-calibrated-typed-decisions-107959.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.37732>)

Author: Xiaolu Ding, Kai Yang, Dongdong Liu

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [arithmetic](<https://devfeed.tech/tags/arithmetic.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [choice](<https://devfeed.tech/tags/choice.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [error](<https://devfeed.tech/tags/error.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [has](<https://devfeed.tech/tags/has.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [thinking](<https://devfeed.tech/tags/thinking.md>), [tokens](<https://devfeed.tech/tags/tokens.md>)

### AI overview

ThinkJev combines direct and reasoning-based typed decisions from one hybrid-thinking model, reading probabilities over the same answer options in both modes. Escalation reuses the cached prompt, and a single sampled reasoning trace can be evaluated at different token budgets. Without training, a statistical test on labeled development data limits the fast system's accepted answers to a target error rate: at a 5% target, Qwen3.5-27B handled 24.5% of 8,359 test decisions directly with 3.9% realized error. For remaining decisions, stopping reasoning when the answer becomes confident achieved 81.6% accuracy using 72% of the thought tokens required by always thinking (82.7% accuracy). Confidence-based selection of which decisions to think through saved less. In an exploratory JevBench comparison, Qwen3.5-27B thought on 29.4% of public items and its accuracy was not distinguishable from the two listed thinking APIs.

### Source excerpt

arXiv:2610.37732v1 Announce Type: new Abstract: Many uses of language models end in a typed decision: a choice among a closed set of options, whose probabilities code can act on. Reading those probabilities from one forward pass is fast but often wrong on decisions that take several steps, such as arithmetic, counting, and date comparison. Letting the model think first fixes most of these errors, but costs seconds per decision instead of milliseconds. To pay for thinking only where it helps, ThinkJev builds a fast and a slow system from one open hybrid-thinking model that can answer directly or after a thought. System 1 reads the model's probabilities for the option letters when it answers directly; System 2 reads the same letters after the model has thought on the same cached prompt, so escalating repeats no work and both systems give probabilities over the same options. Because the letters can be read after any number of thought tokens, one sampled thought also gives the answer under every thinking budget. Without training, ThinkJev lets System 1 answer alone only where a statistical test on labeled dev data guarantees that its accepted answers stay under a target error rate. At a 5% target, Qwen3.5-27B answers 24.5% of 8359 test decisions this way with a realized error of 3.9% (Qwen3.5-4B and 9B: 6.5% and 0.0%). For the rest, stopping each thought as soon as its answer is confident reaches 81.6% accuracy, against 82.7% for always thinking, with 72% of the thought tokens. Choosing when to think from System 1's confidence alone saves less, because the decisions System 1 is least sure of are often not the ones thinking fixes. On the public items of JevBench, an independent benchmark of typed-decision services, Qwen3.5-27B thinks on 29.4% and is not distinguishable in accuracy from the two thinking APIs listed (exploratory). Code and data generators are available at https://github.com/ding-xiaolu/ThinkJev.

## Bet on Features: Anytime-Valid and Feature-Aware Auditing of Conditional Quantile Forecasters

DevFeed: [Bet on Features: Anytime-Valid and Feature-Aware Auditing of Conditional Quantile Forecasters](<https://devfeed.tech/articles/bet-on-features-anytime-valid-and-feature-aware-auditing-of-conditional-quantile-forecasters-108519.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.11653>)

Author: Ivane Antonov, Sohom Mukherjee, Richard Pibernik, Yo Joong Choe

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>)

Tags: [auditing](<https://devfeed.tech/tags/auditing.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [detection](<https://devfeed.tech/tags/detection.md>), [features](<https://devfeed.tech/tags/features.md>), [framework](<https://devfeed.tech/tags/framework.md>), [quantile](<https://devfeed.tech/tags/quantile.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

The paper proposes a distribution-free, game-theoretic framework for continuously auditing black-box conditional quantile forecasts when observations may be dependent and nonstationary. It defines calibration relative to the contextual features available to an auditor, showing that coarse information can hide miscalibration. For contextual bets linear in those features, it gives finite-time detection guarantees without assuming i.i.d. data; the resulting evidence processes are intended to make miscalibration interpretable and feature-aware. Experiments on simulated and real data report that Chronos-2 is highly miscalibrated with respect to multiple relevant features.

### Source excerpt

arXiv:2607.11653v2 Announce Type: replace Abstract: Black-box conditional quantile forecasts are widely used for sequential decisions under asymmetric costs. Once deployed, such forecasters must be monitored continuously as data streams drift and regimes change; this invalidates standard, fixed-horizon backtests for calibration. Further, existing backtests do not take into account that the notion of calibration is, in fact, information-dependent: forecasts can look calibrated to an auditor with coarse information while being miscalibrated to an auditor with richer information. We develop a distribution-free and game-theoretic testing framework for continuously auditing such forecasters with dependent, potentially nonstationary observations. We first formalize notions of conditional quantile calibration relative to the contextual features available to the auditor, showing how coarser information can limit detection. We then identify detectable alternative classes and, for contextual bets linear in the features, derive finite-time detection guarantees without an i.i.d. assumption. The resulting evidence processes provide interpretable, "feature-aware" evidence for miscalibration. We empirically validate these methods on simulated and real data, finding that a popular time series forecaster (Chronos-2) is highly miscalibrated w.r.t. multiple relevant features.

## Detecting AI Training in Datacentre Power

DevFeed: [Detecting AI Training in Datacentre Power](<https://devfeed.tech/articles/detecting-ai-training-in-datacentre-power-96565.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.30182>)

Author: Tom Kimpson, Mauricio Baker, Emlyn Graham

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Advanced AI workflows](<https://devfeed.tech/topics/advanced-ai-workflows.md>), [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [adversarial](<https://devfeed.tech/tags/adversarial.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-governance](<https://devfeed.tech/tags/ai-governance.md>), [ai-training](<https://devfeed.tech/tags/ai-training.md>), [alarm](<https://devfeed.tech/tags/alarm.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [but](<https://devfeed.tech/tags/but.md>), [cadence](<https://devfeed.tech/tags/cadence.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [compute](<https://devfeed.tech/tags/compute.md>), [computing](<https://devfeed.tech/tags/computing.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [develop](<https://devfeed.tech/tags/develop.md>), [faster](<https://devfeed.tech/tags/faster.md>), [form](<https://devfeed.tech/tags/form.md>), [governance](<https://devfeed.tech/tags/governance.md>), [power](<https://devfeed.tech/tags/power.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

The method detects synchronous model training from facility-level power by tracking the wandering cadence of training iterations with a Viterbi algorithm. In simulations, with a meter sampling at 2 Hz or faster and against a declared inference workload, it detected at least 96% of runs at a 1% false-alarm rate when operators did not try to hide training. Of eight hiding strategies considered, only extreme variation in work per iteration evaded the tracker. Its strongest version hid training in up to 73% of five-minute observations, yet at least 87% of runs were caught over a day at a 0.1% false-alarm rate per run. It preserved throughput but, in small-scale training, required 93-267% more compute to reach the same loss. These are simulation results, and the abstract presents passive power metering as a promising component of compute verification.

### Source excerpt

arXiv:2610.30182v1 Announce Type: new Abstract: AI governance agreements require methods to detect large model training runs on a monitored computing cluster, without trusting the operator. We develop a method to detect synchronous model training in facility-level power by Viterbi tracking of the wandering iteration cadence. On simulated datacentre power, when the operator makes no attempt to hide, the Viterbi tracker detects at least $96\%$ of training runs against a declared inference workload, at a $1\%$ false-alarm rate and within $0.04$ of the Neyman--Pearson optimum, given a meter that samples at $2$ Hz or faster. Of eight adversarial strategies an operator might use to hide, only extreme variation of the work per iteration evades the tracker. Its strongest form hides training in up to $73\%$ of five-minute observations, but over a day at least $87\%$ of such runs are still caught, at a $0.1\%$ false-alarm rate per run. This adversarial strategy costs no throughput, but in small-scale training the operator needs $93$--$267\%$ more compute to reach the same loss. A modest passive meter is sufficient to make hidden training both detectable and expensive, which makes power metering a promising part of compute verification that does not depend on trusting the operator.

## Diffusion-Based Hypothesis Testing and Change-Point Detection

DevFeed: [Diffusion-Based Hypothesis Testing and Change-Point Detection](<https://devfeed.tech/articles/diffusion-based-hypothesis-testing-and-change-point-detection-96790.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2506.16089>)

Author: Sean Moushegian, Taposh Banerjee, Vahid Tarokh

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>)

Tags: [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [detection](<https://devfeed.tech/tags/detection.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [hypothesis-testing](<https://devfeed.tech/tags/hypothesis-testing.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [performance](<https://devfeed.tech/tags/performance.md>), [simulations](<https://devfeed.tech/tags/simulations.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [statistics](<https://devfeed.tech/tags/statistics.md>)

### AI overview

The paper extends score-based hypothesis testing and change-point detection to diffusion-based methods by transforming score functions with a matrix-valued weight. It theoretically analyzes their performance, identifies scenarios where optimal performance is achievable, and proposes numerical optimization of the weight matrix. Simulations illustrate advantages over score-based approaches; the abstract does not specify the settings or quantify those gains.

### Source excerpt

arXiv:2506.16089v2 Announce Type: replace-cross Abstract: Score-based methods have recently seen increasing popularity in modeling and generation. Methods have been constructed to perform hypothesis testing and change-point detection with score functions, but these methods are in general not as powerful as their likelihood-based peers. Recent works consider generalizing the score-based Fisher divergence into a diffusion-divergence by transforming score functions via multiplication with a matrix-valued function or a weight matrix. In this paper, we extend the score-based hypothesis test and change-point detection stopping rule into their diffusion-based analogs. Additionally, we theoretically quantify the performance of these diffusion-based algorithms and study scenarios where optimal performance is achievable. We propose a method of numerically optimizing the weight matrix and present numerical simulations to illustrate the advantages of diffusion-based algorithms.

## Calibration-Free Early Stopping for Chain-of-Thought Reasoning via E-processes

DevFeed: [Calibration-Free Early Stopping for Chain-of-Thought Reasoning via E-processes](<https://devfeed.tech/articles/calibration-free-early-stopping-for-chain-of-thought-reasoning-via-e-processes-93444.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.25212>)

Author: Dong-Jae Lee, Gyojin Han, Junmo Kim

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [chain-of-thought](<https://devfeed.tech/tags/chain-of-thought.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [framework](<https://devfeed.tech/tags/framework.md>), [inequality](<https://devfeed.tech/tags/inequality.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [signal](<https://devfeed.tech/tags/signal.md>), [stop](<https://devfeed.tech/tags/stop.md>), [tokens](<https://devfeed.tech/tags/tokens.md>)

### AI overview

The framework stops chain-of-thought generation by accumulating sequential evidence, avoiding held-out threshold calibration. It runs two e-processes on a shared per-chunk signal: a commit stream tracks the signal's position relative to its slow trend, while an abstain stream tracks its signed velocity; they are anti-correlated by construction. Under signal-level nulls on per-chunk increments, Ville's inequality gives an anytime-valid signal-level false-alarm bound at user-chosen significance levels. The abstract reports that across various model-dataset combinations, the method matches or exceeds calibration-based stopping on the token-accuracy Pareto front with zero calibration data, and transfers the same significance level to settings where calibration is infeasible.

### Source excerpt

arXiv:2610.25212v1 Announce Type: cross Abstract: Reasoning large language models (LLMs) benefit from test-time scaling, motivating adaptive policies that spend tokens when they help and stop early otherwise. Existing rules halt when a confidence or stability signal crosses a threshold, but such thresholds rarely transfer across models and datasets, and validation-calibrated rules lose their guarantee under shifts in test-time horizon or signal dynamics. We reframe adaptive stopping as sequential evidence accumulation along the reasoning trajectory rather than threshold calibration on held-out data. Our framework runs two e-processes on a shared per-chunk signal: a commit stream tracks the signal's position relative to its slow trend, and an abstain stream tracks its signed velocity, making the two anti-correlated by construction. Under signal-level nulls on the per-chunk increments, Ville's inequality yields an anytime-valid signal-level false-alarm bound at user-specified significance levels, with no held-out calibration. Across various model--dataset combinations, the framework matches or exceeds calibration-based stopping on the token--accuracy Pareto front using zero calibration data, and the same significance level transfers unchanged across cells where calibration becomes infeasible.

## Why Spotify Is Not Using Bayesian A/B Testing

DevFeed: [Why Spotify Is Not Using Bayesian A/B Testing](<https://devfeed.tech/articles/why-spotify-is-not-using-bayesian-a-b-testing-156.md>)

Original publisher: [Read original article](<https://engineering.atspotify.com/2026/9/why-spotify-is-not-using-bayesian-a-b-testing/>)

Author: Spotify Engineering

Published: 2026-09-08T13:18:44Z

Content type: article

Language: en

Sources: [Spotify](<https://devfeed.tech/sources/spotify-engineering-blog.md>), [Spotify Engineering](<https://devfeed.tech/sources/spotify-engineering.md>)

Topics: [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>), [Developer Tools](<https://devfeed.tech/topics/developer-tools.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>), [Spotify](<https://devfeed.tech/topics/spotify.md>)

Tags: [a-b-testing](<https://devfeed.tech/tags/a-b-testing.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [bayesian-inference](<https://devfeed.tech/tags/bayesian-inference.md>), [data](<https://devfeed.tech/tags/data.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [spotify](<https://devfeed.tech/tags/spotify.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

Spotify says it has not added Bayesian inference because supporting two modes would add planning, monitoring, interpretation, and configuration costs without enough benefit for its experimentation goals. The post argues Bayesian A/B testing is a family of setups whose guarantees depend on the prior, likelihood, and stopping rule. Flat-prior posterior thresholds can reproduce frequentist peeking behavior; Bayes-factor stopping can control false positives under peeking, and a well-calibrated empirical Bayes prior can shrink winner's-curse estimates and control false discovery rates across metrics. Those benefits require representative historical data and careful calibration; pooling unlike programs or metrics, winner-biased archives, or drift can undermine results. The authors describe many Bayesian and frequentist procedures as equivalent or closely related, while noting decision rules can differ for some cost functions. They recommend choosing required guarantees based on program goals; Spotify judges prior maintenance and the costs of a second framework outweigh its benefits today.

### Source excerpt

Clearing the confusion about what Bayesian A/B testing is. The post Why Spotify Is Not Using Bayesian A/B Testing appeared first on Spotify Engineering.

## Speed with Rigor: testing smarter with group sequential design

DevFeed: [Speed with Rigor: testing smarter with group sequential design](<https://devfeed.tech/articles/speed-with-rigor-testing-smarter-with-group-sequential-design-20391.md>)

Original publisher: [Read original article](<https://tech.olx.com/speed-with-rigor-testing-smarter-with-group-sequential-design-af962363ce90?source=rss----761b019b483f---4>)

Author: Gabriela Lewenfus

Published: 2025-08-05T17:25:29Z

Content type: article

Language: en

Sources: [OLX](<https://devfeed.tech/sources/olx.md>)

Topics: [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>), [Software Testing](<https://devfeed.tech/topics/software-testing.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>)

Tags: [a-b-testing](<https://devfeed.tech/tags/a-b-testing.md>), [ab-testing](<https://devfeed.tech/tags/ab-testing.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [costs](<https://devfeed.tech/tags/costs.md>), [data-driven](<https://devfeed.tech/tags/data-driven.md>), [decision-making](<https://devfeed.tech/tags/decision-making.md>), [design](<https://devfeed.tech/tags/design.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [experimentation](<https://devfeed.tech/tags/experimentation.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [false-positive](<https://devfeed.tech/tags/false-positive.md>), [false-positive-rate](<https://devfeed.tech/tags/false-positive-rate.md>), [false-positives](<https://devfeed.tech/tags/false-positives.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [olx](<https://devfeed.tech/tags/olx.md>), [power](<https://devfeed.tech/tags/power.md>), [sample](<https://devfeed.tech/tags/sample.md>), [sequential-testing](<https://devfeed.tech/tags/sequential-testing.md>), [speed](<https://devfeed.tech/tags/speed.md>), [statistics](<https://devfeed.tech/tags/statistics.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

Group sequential testing (GST) lets experimenters check A/B test results at predefined interim stages and stop early while controlling the overall false-positive rate. Efficacy boundaries require especially strong evidence at early checks; futility boundaries can stop tests unlikely to show an effect, but overly aggressive futility rules risk ending tests prematurely. Compared with a fixed-horizon design, GST may reduce average sample use when boundaries are crossed, while its maximum sample size is higher to account for repeated analyses. The text gives an example where a 20-stage design's maximum can be nearly 25% larger than the fixed-horizon sample size. More frequent checks can increase early stopping opportunities but also affect that maximum. Accurate traffic estimates and planned analysis timing matter: unexpected traffic increases may compromise Type I error control, while drops can reduce sensitivity. The article describes OLX's move to GST but provides no specific implementation results in this excerpt.

### Source excerpt

generated by chatgptIntroduction In the world of A/B testing and experimentation, it's tempting to check results frequently and stop tests earlier if they seem significant (a practice known as peeking). However, this practice dramatically inflates the false positive rate, leading to misleading conclusions. Imagine you're running an A/B test, and after just 10 days, the results look like a clear win. You might be tempted to end the experiment early and move forward with the release of the new feature. After all, why wait another two weeks as originally planned? But here's the catch: the more often you check the results, the higher the chance you'll spot a "win" that isn't real. Frequent peeking increases the risk of false positives -- in other words, drawing the wrong conclusion just by chance. To prevent peeking while ensuring statistical power, a common practice is to set a minimum sample size that must be reached before ending the experiment. However, this fixed horizon approach can extend the experiment's duration and delay decision-making. Group sequential testing (GST) provides a statistical framework to monitor results at predefined intervals while maintaining control over error rates. At OLX, we migrated from fixed horizon experiments to GST to achieve faster insights, lower costs, and greater efficiency, enabling our teams to make smarter, data-driven decisions with speed and confidence. In this post, I'll explain how GST works, how it can accelerate experiments while still preventing the pitfalls of peeking, and how we can use it even to stop ineffective tests early. Finally, I'll discuss real-world trade-offs and implementation challenges. Peeking is a trap Before we dive into GST, let's first understand what "peeking" is and why it's so concerning. Peeking occurs when the experimenter repeatedly checks an experiment's results and finishes it once a statistically significant effect appears. This inflates the false positive rate, leading to unreliable conclu

## 22 ways PostHog makes it easier to build great products

DevFeed: [22 ways PostHog makes it easier to build great products](<https://devfeed.tech/articles/22-ways-posthog-makes-it-easier-to-build-great-products-80672.md>)

Original publisher: [Read original article](<https://posthog.com/blog/using-posthog>)

Author: Andy Vandervell

Published: 2022-07-20T00:00:00Z

Content type: article

Language: en

Sources: [PostHog](<https://devfeed.tech/sources/posthog.md>)

Topics: [product analytics](<https://devfeed.tech/topics/product-analytics.md>), [Business, Industry, Usability, User Experience, User Research, Web Strategy](<https://devfeed.tech/topics/business-industry-usability-user-experience-user-research-web-strategy.md>), [Software Process](<https://devfeed.tech/topics/software-process.md>), [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>), [Tool](<https://devfeed.tech/topics/tool.md>)

Tags: [22](<https://devfeed.tech/tags/22.md>), [acquisition](<https://devfeed.tech/tags/acquisition.md>), [activation](<https://devfeed.tech/tags/activation.md>), [ads](<https://devfeed.tech/tags/ads.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-observability](<https://devfeed.tech/tags/ai-observability.md>), [amplitude](<https://devfeed.tech/tags/amplitude.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [app-store](<https://devfeed.tech/tags/app-store.md>), [apps](<https://devfeed.tech/tags/apps.md>), [build](<https://devfeed.tech/tags/build.md>), [easier](<https://devfeed.tech/tags/easier.md>), [feature](<https://devfeed.tech/tags/feature.md>), [great](<https://devfeed.tech/tags/great.md>), [platform](<https://devfeed.tech/tags/platform.md>), [product-analytics](<https://devfeed.tech/tags/product-analytics.md>), [products](<https://devfeed.tech/tags/products.md>)

### AI overview

PostHog combines product analytics with session recordings, feature flags, experiments, error tracking, and data tools so teams can investigate product behavior in one platform. The article describes using AARRR dashboards, cohorts, custom metrics, and event sequences to track engagement, conversion, churn risk, and process time; surveys and UTM data to gather feedback and assess acquisition; and funnel correlation plus session recordings to explore what converting or dropping-off users do. Experiments use feature flags and Bayesian analysis, while flags can also act as emergency kill switches. The article says PostHog captures data automatically for retroactive analysis and supports community-built apps, scheduled dashboard reports, and Zapier workflows. It presents PostHog as a way to replace some separate analytics and data services, while noting integrations and self-hosted ClickHouse warehousing as options.

### Source excerpt

PostHog is a growing platform. We used to call ourselves a product analytics platform, but product analytics is just one feature among many these...