# benchmark overfitting machine learning

A machine-learning evaluation problem in which adaptively reusing a held-out benchmark can cause overfitting to that benchmark.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## The Price of an Explanation: What Demanding Verifiable Artifacts Costs, and What It Buys Back

DevFeed: [The Price of an Explanation: What Demanding Verifiable Artifacts Costs, and What It Buys Back](<https://devfeed.tech/articles/the-price-of-an-explanation-what-demanding-verifiable-artifacts-costs-and-what-it-buys-back-145115.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25896>)

Author: Edgar A. Du\\'e\\~nez-Guzm\\'an, Eduardo Due\\~nez, Edgar Said Hernandez Sanchez, Suzanne Sadedin

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [few-shot](<https://devfeed.tech/topics/few-shot.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [agi](<https://devfeed.tech/tags/agi.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [budget](<https://devfeed.tech/tags/budget.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ne](<https://devfeed.tech/tags/cs-ne.md>), [decoder](<https://devfeed.tech/tags/decoder.md>), [error-correction](<https://devfeed.tech/tags/error-correction.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [llm](<https://devfeed.tech/tags/llm.md>), [mitigation](<https://devfeed.tech/tags/mitigation.md>), [program](<https://devfeed.tech/tags/program.md>), [quantum](<https://devfeed.tech/tags/quantum.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [score](<https://devfeed.tech/tags/score.md>), [search](<https://devfeed.tech/tags/search.md>), [synthesis](<https://devfeed.tech/tags/synthesis.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [test](<https://devfeed.tech/tags/test.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

On ARC-AGI-2, requiring an LLM to produce a verifiable program reduced performance from 93/120 tasks (77%) to 68/120 (57%). The authors report that most answer-only wins remained out of reach for the program-producing configurations they tested, even when a program matched every demonstration, pointing to generalization as the key bottleneck. A short evolutionary search over two sampled programs improved on best-of-two sampled code by 9.2 ± 1.4 tasks without additional model cost. Checking programs against training examples also enabled early stopping, reducing budget use by up to 38% for best-of-n sampling and 49% for the evolved arm; answer-only runs could not stop this way. The same methodology was applied to quantum error-correction decoder synthesis, producing interpretable decoders evaluated on held-out shots against a certified panel of published decoders.

### Source excerpt

arXiv:2611.25896v1 Announce Type: cross Abstract: Generative models produce many candidate solutions but provide no intrinsic signal by which to rank them. One mitigation is to demand a \emph{verifiable artifact}: a program that can be executed against the available evidence. A common concern is that requiring executable verification incurs costs in both solution quality and computational budget. We measure these two costs on the $120$-task ARC-AGI-2 evaluation set. Demanding a verifiable artifact from an LLM reduced the score from $93$ to $68$ of $120$ tasks ($77\%$ to $57\%$). Of those $25$ tasks, $21$ were reached by no program-producing configuration we ran, at any budget, by sampling or by search; yet for $27$ of the $28$ tasks that only the answer-only arm solved, some program arm held a program that reproduced every demonstration and still failed the test. The bottleneck is obtaining an explanation that \emph{generalizes}, not finding one. A short evolutionary search over two already-sampled programs, at no additional model cost, beats best-of-$2$ sampled code by $9.2 \pm 1.4$ tasks, and over a single sampled program beats $1$-shot sampling by $9.9 \pm 1.3$. Moreover, because a program can be checked against the training examples, any program-producing protocol can terminate early once a task is explained: under the same stopping rule, best-of-$n$ sampling declines up to $38\%$ of its budget and the evolved arm up to $49\%$, whereas an answer-only baseline cannot terminate early. Applying the same methodology to decoder synthesis for quantum error correction yields interpretable decoders evaluated against a certified panel of published decoders on held-out exam shots.

## Moving Generality from Model to Modeler: LLM Agents for PHM on a Deployment-Style Benchmark

DevFeed: [Moving Generality from Model to Modeler: LLM Agents for PHM on a Deployment-Style Benchmark](<https://devfeed.tech/articles/moving-generality-from-model-to-modeler-llm-agents-for-phm-on-a-deployment-style-benchmark-145028.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25252>)

Author: Zhenling Chen, Zhiguo Zeng, Anne Barros

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [generalization in machine learning](<https://devfeed.tech/topics/generalization-in-machine-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [agents](<https://devfeed.tech/tags/agents.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [automl](<https://devfeed.tech/tags/automl.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [design](<https://devfeed.tech/tags/design.md>), [domain](<https://devfeed.tech/tags/domain.md>), [equipment](<https://devfeed.tech/tags/equipment.md>), [error](<https://devfeed.tech/tags/error.md>), [health](<https://devfeed.tech/tags/health.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [model](<https://devfeed.tech/tags/model.md>)

### AI overview

On NextUnit-Bench, frontier LLM coding agents outperform AutoML and most faithfully implemented published methods at building predictors for unseen equipment. The benchmark covers 24 public datasets across 58 unit- or condition-holdout folds, with two hours to produce an executable predictor. The reported advantage comes mainly from learning each machine's data: a short plain-text plan captures much of the gap between frontier agents, while the strongest agent performs at a weaker agent's level when given its plan. Pre-data choice of plan author and domain toolkits appear to matter little. A locally deployable open-weight model executes the strongest agent's designs only half as often and scores no better than without them. Reported published-method accuracies depend on each paper's conventions; capping test engines' remaining life can halve reported error.

### Source excerpt

arXiv:2611.25252v1 Announce Type: new Abstract: Coding agents built on large language models (LLMs) now help develop many models published as general solutions. Should generality instead reside in the agent that builds a model for each new task? We examine this question in prognostics and health management (PHM), where every model must work on unseen equipment. Our deployment-style benchmark, NextUnit-Bench, covers 24 public datasets split into 58 folds that hold out units or operating conditions; each agent has two hours to deliver an executable predictor, then run on the hidden units. First, frontier agents outperform AutoML, which reaches less than half the best agent's score, and faithful implementations of published methods on four datasets, of which only 6 of 30 beat the strongest agent's own design; published accuracies, where reproducible, rest on the papers' own conventions, and a single convention, capping the remaining life of test engines, can halve the reported error. Second, the agents' advantage is a design made on each machine's data; once made, a plain-text plan of a few hundred words carries most of the gap between two frontier agents, and given a weaker agent's plan, the strongest agent falls to that agent's level, often to identical predictions. Before touching the data, however, it matters little whose plan an agent carries out, and domain toolkits appear to help neither: what sets the strongest apart is how much it learns from the data. Third, such designs reach only implementers that can execute them: given the strongest agent's designs, a locally deployable open-weight model delivers half as often, even when they are lightened, and scores no higher than without them. Generality, we argue, is not disappearing but moving, from the model to the modeler.

## Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts

DevFeed: [Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts](<https://devfeed.tech/articles/bad-genius-counterfactual-guided-harness-evolution-beyond-task-specific-shortcuts-142560.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.18366>)

Author: Guojun Zhu, Xunheng Huang, Peng Yin, Jiahui Xie, Sanguo Zhang, Doudou Zhou

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [Multi-step AI processes](<https://devfeed.tech/topics/multi-step-ai-processes.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-harness](<https://devfeed.tech/tags/agent-harness.md>), [archive](<https://devfeed.tech/tags/archive.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [formatting](<https://devfeed.tech/tags/formatting.md>), [foundation-model](<https://devfeed.tech/tags/foundation-model.md>), [harness](<https://devfeed.tech/tags/harness.md>), [order](<https://devfeed.tech/tags/order.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [search](<https://devfeed.tech/tags/search.md>), [semantics](<https://devfeed.tech/tags/semantics.md>), [shortcuts](<https://devfeed.tech/tags/shortcuts.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [swe-bench](<https://devfeed.tech/tags/swe-bench.md>), [swe-bench-pro](<https://devfeed.tech/tags/swe-bench-pro.md>)

### AI overview

CHASE tests whether agent-harness improvements survive protocol changes that preserve task semantics. A Proposer edits a harness around a fixed foundation model; a Challenger searches for changes that reduce its gain over the initial harness. Executable checks verify task preservation, and confirmed held-out gain losses are archived to constrain later updates. The authors state conditions under which a finite archive bounds losses on an ideal benchmark without shortcuts. On Syn-Ledger with known shortcuts removed, CHASE reached 38.30% accuracy versus 18.27% for HarnessEvolve. On 21 SWE-Bench Pro V2 HARD-51 certification tasks, changing issue formatting and tool-output field order cut HarnessEvolve's gain from 8.57 to 3.81 percentage points; CHASE retained 10.48 of its 11.43-point gain.

### Source excerpt

arXiv:2609.18366v4 Announce Type: replace Abstract: An optimized agent harness can improve on held-out tasks by relying on formatting conventions that those tasks share with the rest of the benchmark. We introduce Counterfactual Harness Search and Evolution (CHASE) to test whether harness improvements survive changes to the evaluation protocol. A Proposer edits the harness around a fixed foundation model. A Challenger then searches for a protocol change that preserves the tasks but reduces the updated harness's gain over the initial harness. Executable checks test whether the transformation preserves task semantics. If the gain loss is confirmed on held-out tasks, CHASE adds the transformation to an archive that constrains subsequent harness updates. We give conditions under which this finite archive bounds the gain loss on an ideal benchmark with shortcuts removed. We evaluate CHASE on Syn-Ledger, OfficeQA, FinanceBench, and SWE-Bench Pro V2. On Syn-Ledger, removing the known shortcuts yields 38.30% accuracy for CHASE and 18.27% for HarnessEvolve. On the 21 certification tasks from SWE-Bench Pro V2 HARD-51, changing the issue formatting and tool-output field order reduces HarnessEvolve's gain over the initial harness from 8.57 to 3.81 percentage points. The final CHASE harness retains 10.48 points of its 11.43-point gain under the same change.

## Beyond Cross-Modal Reconstruction: Participant-Level Evaluation of sEMG-Kinematic Coupling in Post-Stroke Gait

DevFeed: [Beyond Cross-Modal Reconstruction: Participant-Level Evaluation of sEMG-Kinematic Coupling in Post-Stroke Gait](<https://devfeed.tech/articles/beyond-cross-modal-reconstruction-participant-level-evaluation-of-semg-kinematic-coupling-in-post-stroke-gait-143455.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23856>)

Author: Rafsan Jany, Md Rafid Islam, Mohammad Fahim Islam, Hafiz Abdur Rahman

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>)

Tags: [coupling](<https://devfeed.tech/tags/coupling.md>), [cross-validation](<https://devfeed.tech/tags/cross-validation.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [kinematics](<https://devfeed.tech/tags/kinematics.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [mobility](<https://devfeed.tech/tags/mobility.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [stroke](<https://devfeed.tech/tags/stroke.md>)

### AI overview

In this study, accurate sEMG-to-kinematics reconstruction did not establish participant-specific neuromuscular coupling: substituting another participant's source increased RMSE only 1.68% and 1.67% in the two directions, versus 6.90% and 11.38% when the source modality was removed. A cycle-position-age-sex template even reduced RMSE slightly. For POMA prediction, adding learned asymmetry to walking speed, age, and sex worsened MAE from 3.693 to 4.229 (absolute asymmetry) or 4.427 (signed); every asymmetry representation underperformed the covariate baseline across 20 repeated evaluations. No localized learned feature survived covariate adjustment and false-discovery-rate correction. The authors conclude that the models mainly captured shared gait-phase patterns and that participant-level controls are needed before reconstruction accuracy is interpreted as evidence of individualized physiology.

### Source excerpt

arXiv:2611.23856v1 Announce Type: new Abstract: Accurate cross-modal reconstruction is often treated as evidence of physiological coupling, although it may simply reflect temporal patterns shared across participants. We tested whether frozen bidirectional models captured subject-specific neuromuscular coupling between electromyographic activity and gait kinematics in 82 held-out able-bodied and 43 post-stroke participants. Complete source-modality removal and cross-participant source substitution, along with a cycle-position-age-sex template baseline, were applied. Learned bilateral asymmetry was also compared with transparent waveform and derivative asymmetry for predicting Performance-Oriented Mobility Assessment (POMA) scores in the case of repeated nested cross-validation. Complete source removal increased reconstruction RMSE by 6.90% for electromyography-to-kinematics and 11.38% in the reverse direction. In contrast, cross-participant substitution increased RMSE by only 1.68% and 1.67%, respectively, while the template reduced RMSE by 2.89% and 0.55%. For POMA prediction, walking speed, age, and sex achieved an MAE of 3.693, compared with 4.229 and 4.427 after adding absolute and signed learned asymmetry, respectively. In all of the 20 repeated evaluations, all asymmetry representations performed worse than the covariate baseline. No localized learned feature remained significant after covariate adjustment and false-discovery-rate correction. The model mainly relied on shared gait-phase patterns, with no clear subject-specific coupling or added clinical value. Participant-level controls are therefore needed before reconstruction accuracy is treated as evidence of individualized physiology.

## What Interpolators Store and What They Express: An Information-Theoretic Framework for Benign Overfitting and Label-Free Model Selection

DevFeed: [What Interpolators Store and What They Express: An Information-Theoretic Framework for Benign Overfitting and Label-Free Model Selection](<https://devfeed.tech/articles/what-interpolators-store-and-what-they-express-an-information-theoretic-framework-for-benign-overfitting-and-label-free-model-selection-143421.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23630>)

Author: Nikolaos Nikolaou

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [machine learning overfitting](<https://devfeed.tech/topics/machine-learning-overfitting.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>)

Tags: [bayes](<https://devfeed.tech/tags/bayes.md>), [bias](<https://devfeed.tech/tags/bias.md>), [bits](<https://devfeed.tech/tags/bits.md>), [boosting](<https://devfeed.tech/tags/boosting.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [compression](<https://devfeed.tech/tags/compression.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cross-validation](<https://devfeed.tech/tags/cross-validation.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [express](<https://devfeed.tech/tags/express.md>), [expression](<https://devfeed.tech/tags/expression.md>), [framework](<https://devfeed.tech/tags/framework.md>), [information](<https://devfeed.tech/tags/information.md>), [model](<https://devfeed.tech/tags/model.md>), [model-selection](<https://devfeed.tech/tags/model-selection.md>), [noise](<https://devfeed.tech/tags/noise.md>), [overfitting](<https://devfeed.tech/tags/overfitting.md>), [store](<https://devfeed.tech/tags/store.md>)

### AI overview

The framework distinguishes label information stored in an interpolating model from sample information expressed in predictions at fresh inputs. It says every interpolator stores the same amount of label noise, making storage-based measures such as CMI unable to distinguish them; expressed information V, together with modal bias, bounds excess risk and characterizes benign overfitting as vanishing expression. The authors report that, across 15 noisy datasets, estimating V from two disjoint halves had 60% less selection regret than holdout validation at equal cost, matched five-fold cross-validation at one-quarter its cost, and beat CMI-based selection at matched cost. The abstract also says the work characterizes failure cases and remedies.

### Source excerpt

arXiv:2611.23630v1 Announce Type: new Abstract: Overparameterized learners can fit noisy training labels perfectly and still predict well, which neither classical complexity measures nor information-theoretic generalization bounds explain. We develop an information-theoretic framework that separates two things an interpolating learner does with its sample: storage, the information the trained model retains about its training labels, and expression, the information about the sample that reaches its predictions at fresh inputs. We first characterize the ideal model: for a loss l, the least complex l-sufficient representation is the Bayes act, whose entropy is the minimal sufficient complexity; this generalizes, and corrects, an earlier "lossless maximal compression" view of boosting. We then show that every interpolator stores exactly nH(Y|X) bits of label noise, so storage, and bounds built on it (including CMI bounds), cannot tell interpolators apart. Expression can: excess risk is bounded above and below by the expressed information V, the entropy over training samples of the prediction at an input, plus a modal-bias term, exactly so for binary labels. Benign overfitting is therefore vanishing expression, a picture that covers nearest-neighbour rules, averaging, minimum-norm least squares and double descent, and that links V to classical bias-variance decompositions and to model disagreement. Finally, V gives a model-selection criterion that needs no held-out labels. Among overparameterized learners on 15 noisy datasets, V estimated from two disjoint halves has 60% less regret than holdout validation at the same cost and matches 5-fold cross-validation at a quarter of its cost; at matched cost it also beats selection by conditional mutual information. We characterize when it fails and give remedies. Code: https://github.com/nnikolaou/GLMC

## Is Randomness Necessary for Adaptive Data Analysis?

DevFeed: [Is Randomness Necessary for Adaptive Data Analysis?](<https://devfeed.tech/articles/is-randomness-necessary-for-adaptive-data-analysis-143105.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.07085>)

Author: Edith Cohen, Haim Kaplan, Yishay Mansour, Shay Sapir, Uri Stemmer, Eliad Tsfadia

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-ds](<https://devfeed.tech/tags/cs-ds.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [randomness](<https://devfeed.tech/tags/randomness.md>)

### AI overview

Adaptive data analysis asks how many adaptively chosen statistical queries a mechanism can answer about an unknown distribution using a dataset of n i.i.d. samples. The abstract says efficient randomized mechanisms support about n² queries, while computationally unbounded analysts can force any deterministic mechanism to fail after just Õ(n) queries. It concludes that randomness is strictly necessary in this information-theoretic setting. The abstract also notes that randomness is not necessary when analysts are computationally bounded.

### Source excerpt

arXiv:2607.07085v3 Announce Type: replace Abstract: The Adaptive Data Analysis (ADA) problem formalizes the challenge of preventing false discovery and overfitting when a dataset is repeatedly reused. Formally, our input is a dataset containing $n$ i.i.d. samples from an unknown distribution $\mathcal{P}$ over a domain $\mathcal{X}$, and our goal is to answer a sequence of $k$ adaptively chosen statistical queries with respect to $\mathcal{P}$. The main question is how many queries we can support (i.e., how large $k$ can be), primarily as a function of the number of samples $n$. This question has been intensively studied and is relatively well-understood for randomized mechanisms: there are computationally efficient mechanisms that support $k \approx n^2$ queries, and no computationally efficient mechanism can answer $k \gg n^2$ queries. In this paper, we address a fundamental question: is randomness necessary for ADA? Despite a decade of work on ADA, this question remains open. A folklore observation dating back to the initial works on ADA is that randomness is {\em not} necessary when the analyst is computationally bounded. Yet, the necessity of randomness against computationally unbounded analysts has remained elusive. Our main contribution resolves this gap in the information-theoretic setting. Perhaps surprisingly, we show that randomness is {\em strictly necessary} to answer a non-trivial number of adaptive queries: when the analyst is unbounded, any deterministic mechanism can be forced to fail after just $k = \tilde{O}(n)$ queries.

## Where Root Cause Analysis Fails: A Retrieval-Reranking Decomposition

DevFeed: [Where Root Cause Analysis Fails: A Retrieval-Reranking Decomposition](<https://devfeed.tech/articles/where-root-cause-analysis-fails-a-retrieval-reranking-decomposition-141609.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.36686>)

Author: Hada Melino Muhammad, Luan Pham, Laure Barri\\\`ere, Sachin Shetty, Leonardo Pulga, Flora D. Salim

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [observability](<https://devfeed.tech/topics/observability.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>)

Tags: [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [reranking](<https://devfeed.tech/tags/reranking.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [root-cause-analysis](<https://devfeed.tech/tags/root-cause-analysis.md>)

### AI overview

The paper argues that top@k accuracy hides whether root cause analysis fails to retrieve the true cause or retrieves it but ranks it too low. On industrial CPS benchmarks, statistical scoring mis-ranks the cause 79-100% of the time, while graph-based methods do not clearly outperform the best statistical baseline. On microservice benchmarks, retrieval reaches 98-100%, where injected faults sharply raise the faulty service's own metric. A two-stage multi-signal retriever and LLM reranker, requiring neither causal graphs nor labeled data, matches or exceeds baseline top@1 accuracy across six benchmark suites by up to 12 points. When the cause is guaranteed among candidates, adding a short system description improves the reranker's lead over the best baseline to 7-18 points across benchmarks.

### Source excerpt

arXiv:2609.36686v2 Announce Type: replace Abstract: Identifying the root cause of an anomaly among hundreds of sensors is critical for preventing safety incidents and costly downtime in complex monitored systems. Existing studies evaluate root cause analysis (RCA) methods using top@k accuracy. We show that this metric has a fundamental blind spot: it conflates two failure modes, retrieval failure, where the true cause is never considered, and reranking failure, where it is considered but ranked too low. In this work, we introduce a retrieval-reranking decomposition and audit four well-known benchmarks to expose this blind spot. Our experiments show that, on industrial CPS benchmarks, the statistical baseline scoring every sensor mis-ranks the true cause 79-100% of the time, and graph-based methods never clearly beat the best statistical baseline, whether their causal graphs are learned on short fault windows, on retrieved candidate pools guaranteed to contain the cause, or on multi-day normal-operation data. Meanwhile, on microservice benchmarks, where each injected fault drives the faulty service's own metric far above its baseline, retrieval is nearly solved (98-100%). Guided by the decomposition, we build a two-stage pipeline combining a multi-signal retriever with an LLM reranker that, as one fixed configuration, matches or exceeds the best baseline's top@1 accuracy on all six benchmark suites (by up to +12 points), with no causal graph or labeled data required. When all methods rank the same retrieved candidates with the true cause guaranteed present, adding a short system-description document lets the reranker lead the best baseline by +7 to +18 points on every benchmark. Code is available at https://github.com/cruiseresearchgroup/DecompRCA.

## Testing World Models on Actions They Have Never Seen: Held-Out Plans, Forecast Accuracy and Price Rules in Retail

DevFeed: [Testing World Models on Actions They Have Never Seen: Held-Out Plans, Forecast Accuracy and Price Rules in Retail](<https://devfeed.tech/articles/testing-world-models-on-actions-they-have-never-seen-held-out-plans-forecast-accuracy-and-price-rules-in-retail-140045.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22537>)

Author: Karthik Iyer

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [World models](<https://devfeed.tech/topics/world-models.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>)

Tags: [actions](<https://devfeed.tech/tags/actions.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [forecast](<https://devfeed.tech/tags/forecast.md>), [models](<https://devfeed.tech/tags/models.md>), [retail](<https://devfeed.tech/tags/retail.md>), [rules](<https://devfeed.tech/tags/rules.md>), [test](<https://devfeed.tech/tags/test.md>), [world-model](<https://devfeed.tech/tags/world-model.md>)

### AI overview

The study tests whether retail world models can predict outcomes for price and promotion plans withheld from training. Across 2,853 fitted models, ordinary forecast accuracy was a moderate guide across model types (rank correlation 0.32-0.66), but could reverse between versions of one world model for high-demand weeks (down to −0.69). Action-focused training did not improve held-out-plan error. In generated worlds, recorded regular prices and three price rules helped; with the same rules, log-linear models did best when their curve could represent the truth, while rule-equipped world models led in 36 of 40 worlds when it could not. On three real retail datasets, rules improved results for unusually deep pure price cuts, though one improvement was within chance, and did not help on lasting price moves. Random test plans modestly helped all models. A generated dosing-world test showed a similar forecast-accuracy mismatch; including accumulated drug level reduced error, but the model captured only 0.12-0.32 of the true gain.

### Source excerpt

arXiv:2611.22537v1 Announce Type: cross Abstract: A world model is trusted to answer "what if": how its world responds to an action it has never seen. We give a test of that trust for any action-conditioned world model: actions held out of training by design and scored against a known answer; whether forecast accuracy predicts accuracy on those actions; and the same structure given to simpler models. We run it in retail, where the action is a price and promotion plan, across 2,853 fitted models. First, forecast accuracy is a fair guide between kinds of model (rank correlation with error on new plans 0.32 to 0.66) and a poor guide between versions of one world model: where merchants plan into high-demand weeks the link reverses, down to -0.69. Second, training the world model to learn the effect of actions (JEPA pretraining, a latent consistency check, adaptation at the plan date) leaves its error on new plans unchanged. Third, in generated worlds the recorded regular price and three price rules improve it. Given the same rules, a log-linear model is best when its curve can match the truth; in a world whose curve lies outside the rules, the world model with the rules is ahead of the log-linear model with the rules in 36 of 40 worlds, and does as well without its recurrent cell. On real data (Breakfast at the Frat, The Complete Journey and M5) the rules lower the world model's error on pure price cuts deeper than any it saw (0.63 to 0.56 on the Frat; 0.45 to 0.44, within chance, on The Complete Journey), but not on lasting price moves. Fourth, a small share of random test plans helps every model a little. In a generated dosing world, a stress test in a second domain, the first finding repeats, and the built-up drug level lowers the world model's error (2.86 to 1.91) though it still predicts only 0.12 to 0.32 of the true gain. The test carries to any domain where a world model is asked about actions outside its history.

## Sampling Luck Masquerades as Allocation Gain: Auditing Test-Time Budget Allocation for Neural Combinatorial Optimization

DevFeed: [Sampling Luck Masquerades as Allocation Gain: Auditing Test-Time Budget Allocation for Neural Combinatorial Optimization](<https://devfeed.tech/articles/sampling-luck-masquerades-as-allocation-gain-auditing-test-time-budget-allocation-for-neural-combinatorial-optimization-139805.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.13087>)

Author: Jinhyung Bae

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Combinatorial optimization](<https://devfeed.tech/topics/combinatorial-optimization.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [Operations research and optimization](<https://devfeed.tech/topics/operations-research-and-optimization.md>)

Tags: [auditing](<https://devfeed.tech/tags/auditing.md>), [bias](<https://devfeed.tech/tags/bias.md>), [budget](<https://devfeed.tech/tags/budget.md>), [combinatorial-optimization](<https://devfeed.tech/tags/combinatorial-optimization.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [headroom](<https://devfeed.tech/tags/headroom.md>), [math-oc](<https://devfeed.tech/tags/math-oc.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [neural](<https://devfeed.tech/tags/neural.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [out-of-sample](<https://devfeed.tech/tags/out-of-sample.md>), [phantom](<https://devfeed.tech/tags/phantom.md>), [reporting](<https://devfeed.tech/tags/reporting.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [statistics](<https://devfeed.tech/tags/statistics.md>), [tsp](<https://devfeed.tech/tags/tsp.md>)

### AI overview

On uniform TSP-100 workloads, apparent 2.2-2.6% gains from non-uniform test-time sampling allocation disappear out of sample: across POMO, AM, and SymNCO, measured gains are 0.457%, 0.015%, and −0.512%. The authors report that this in-sample bias persists across the tested sample and instance counts. Under a shifted workload mixing uniform and clustered instances, held-out-statistics-guided allocation improves best-of-k by 11.5% for AM and 12.0% for SymNCO at equal evaluation budget, though signal-acquisition cost is excluded. POMO, used as a negative control, shows −0.3%. A distribution-label baseline trails by 4.2 points; an exploratory policy that charges a 20-sample probe to the budget retains 3.4% and 4.6% gains for AM and SymNCO. The abstract also reports a correction procedure and reporting checklist.

### Source excerpt

arXiv:2608.13087v2 Announce Type: replace-cross Abstract: Neural combinatorial optimization (NCO) solvers report the best of many sampled solutions per instance, and the sample count is, by convention, identical for every instance. Whether a non-uniform allocation of a fixed total budget would buy anything has not been measured. We measure it, and we audit the measurement itself. First, on in-distribution workloads the allocation headroom is not detectable. Across three pretrained solvers (POMO, AM, SymNCO) on uniform TSP-100, an oracle allocation computed and evaluated on the same stored samples reports a 2.2-2.6% gain with intervals excluding zero; measured out of sample the same gain is indistinguishable from zero (0.457, 0.015, -0.512 percent). Following the customary in-sample procedure, all three solvers would have supported a published 2%-level gain that does not exist. We calibrate this bias against an instance-wise null in which the true gain is zero by construction; over the ranges we test it does not shrink with more samples or more instances. Second, the same correction that removes the phantom gains preserves a real one. Under distribution shift (a workload mixing uniform and clustered instances), a pre-registered confirmatory experiment finds that allocation guided by held-out sample statistics improves best-of-k by 11.5% (AM, primary endpoint; 95% CI [7.4, 19.7]) and 12.0% (SymNCO, replication) at equal evaluation budget, with the signal-acquisition cost not charged; a pre-registered negative control (POMO, an order of magnitude more robust to shift) shows -0.3% [-0.7, 0.24]. The gain exceeds a frozen distribution-label baseline by 4.2 points [1.9, 7.7]. An exploratory policy charging a 20-sample probe against the same budget retains 3.4% (AM) and 4.6% (SymNCO). We give a correction procedure and a reporting checklist, and release all data, code, and the pre-registration record.

## Refinement vs. Replanning: Inference Compute Allocation in Robot Policies

DevFeed: [Refinement vs. Replanning: Inference Compute Allocation in Robot Policies](<https://devfeed.tech/articles/refinement-vs-replanning-inference-compute-allocation-in-robot-policies-137380.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19034>)

Author: Alba Maria Tellez Fernandez

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [robot sense of touch](<https://devfeed.tech/topics/robot-sense-of-touch.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [smolvla](<https://devfeed.tech/topics/smolvla.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>)

Tags: [compute](<https://devfeed.tech/tags/compute.md>), [compute-allocation](<https://devfeed.tech/tags/compute-allocation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [inference](<https://devfeed.tech/tags/inference.md>), [robot](<https://devfeed.tech/tags/robot.md>)

### AI overview

The case study finds that robot-policy inference compute should be evaluated across both action refinement and replanning frequency. With a frozen SmolVLA checkpoint in synchronous LIBERO simulation and chunk length fixed at K=50, reducing denoising from d=5 to d=2 at h=2 raised success from 122/160 to 139/160 and used 5.9% less arithmetic per control step. At d=2, increasing execution horizon from h=2 to h=5 cut arithmetic per attempt by 60.7%, while successes were similar (139/160 vs. 138/160); the study does not establish equivalence. Arithmetic and host-time latency also reflect different cost profiles. Results come from one checkpoint and setting, so they do not identify a universal deployment configuration.

### Source excerpt

arXiv:2611.19034v1 Announce Type: cross Abstract: A frozen chunked robot policy can spend deployment-time inference compute in two coupled ways: more work within each policy call to refine a predicted action chunk, or more frequent policy calls to replan from fresh observations. We study this allocation without updating model weights. Our evaluation procedure reports success with both nominal compute rate and complete attempt cost, selects comparisons on discovery tasks, and tests prespecified comparisons on held-out tasks. Comparisons added after inspecting the held-out results only summarize observed patterns. We demonstrate the procedure with one frozen SmolVLA checkpoint in synchronous LIBERO simulation, using denoising count as the refinement control and fixing chunk length at K=50. Arithmetic and latency characterize cost differently: at d=10, 82.1% of counted arithmetic is independent of denoising, whereas a descriptive affine fit to fixed-input host time assigns 17.7% to its intercept. At h=2, d=2 succeeds on 139/160 attempts versus 122/160 for d=5 (+10.6 percentage points; task-conditional 95% interval: +4.4 to +16.9), while using 5.9% less arithmetic per control step. Executing h=5 rather than h=2 at d=2 yields 138/160 versus 139/160 successes and 60.7% lower arithmetic per attempt; equivalence is not established. The case study shows why deployment adaptation must evaluate generation effort and feedback frequency jointly; it does not define a universal setting.

## DataTrace: Benchmarking Data Agents for Evidence-Grounded Root-Cause Localization

DevFeed: [DataTrace: Benchmarking Data Agents for Evidence-Grounded Root-Cause Localization](<https://devfeed.tech/articles/datatrace-benchmarking-data-agents-for-evidence-grounded-root-cause-localization-138290.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18926>)

Author: Tung Sum Thomas Kwok, Xiaotian Lin, Yuyu Luo

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [data-agents](<https://devfeed.tech/tags/data-agents.md>), [databases](<https://devfeed.tech/tags/databases.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fault-injection](<https://devfeed.tech/tags/fault-injection.md>), [graphs](<https://devfeed.tech/tags/graphs.md>), [grounded](<https://devfeed.tech/tags/grounded.md>), [localization](<https://devfeed.tech/tags/localization.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

DataTrace evaluates whether database agents' root-cause diagnoses are supported by evidence they inspected. Across 83 runnable environments, controlled fault injection supplies root causes and evidence paths; the benchmark scores diagnosis correctness separately from evidence-path coverage. Its CoSE protocol uses an external verifier to check reproducible faults, submitted answers, and inspected evidence. Seven proprietary models averaged 48% root-cause accuracy, with a mean per-model conditional unsupported rate of 18%; the gap varied by fault mechanism, dependency distance, and evidence modality. In-loop training on a matched held-out batch increased grounded correct diagnoses from 83.7% to 94.6%, while accuracy rose from 29.2% to 36.7%. The abstract argues that evidence coverage can expose unsupported correct answers and help improve agents.

### Source excerpt

arXiv:2611.18926v1 Announce Type: new Abstract: Database agents diagnose pipeline failures, but a correct root cause is trustworthy only when it is supported by evidence the agent inspected. Existing benchmarks usually grade the final query or answer but fail to distinguish an evidence-based diagnosis from an unsupported correct answer. We introduce DataTrace, a benchmark for evidence-grounded root-cause localization over typed graphs of database artifacts. DataTrace uses controlled fault injection to derive root causes and evidence paths within 83 runnable environments without human annotations, and scores root-cause correctness and coverage of the supporting path separately. We also introduce CoSE, a generator-solver protocol in which an external verifier admits reproducible faults and checks both the submitted answer and the inspected evidence. Our evaluation shows that seven proprietary models achieve average root-cause accuracy 48%, with a mean per-model conditional unsupported rate of 18%. Fine-grained analysis shows that this gap varies sharply with fault mechanism, dependency distance, and evidence modality. After in-loop training on a matched held-out batch, the share of grounded correct diagnoses rises from 83.7% to 94.6%, while root-cause accuracy rises from 29.2% to 36.7%. These results show that while final-answer accuracy can hide unsupported diagnoses, evidence coverage provides an actionable signal to improve database agents. Code and data are available at https://hkustdial.github.io/datatrace/.

## Predicting, Fast and Slow: SSD Already Knows Which Guess Is Wrong

DevFeed: [Predicting, Fast and Slow: SSD Already Knows Which Guess Is Wrong](<https://devfeed.tech/articles/predicting-fast-and-slow-ssd-already-knows-which-guess-is-wrong-138874.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19457>)

Author: Philipp Bogdan

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>)

Tags: [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-os](<https://devfeed.tech/tags/cs-os.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [operating-systems](<https://devfeed.tech/tags/operating-systems.md>), [ssd](<https://devfeed.tech/tags/ssd.md>)

### AI overview

Heimdall flags 52% of SSD reads as slow when 30% are actually slow, while nine uncertainty estimators identify its wrong guesses no better than chance. The abstract reports that a 4,096-entry table tracking recent read outcomes, requiring no training and taking 21 ns per decision, ranks wrong guesses at median AUROC 0.68 across 57 held-out traces; the nine estimators score 0.46-0.52. It also predicts slow reads better than Heimdall. On one trace, reads of 4-8 KiB with less than 4 KiB outstanding while the drive responded were slow only 2% of the time, yet Heimdall labeled 96% of them slow.

### Source excerpt

arXiv:2611.19457v1 Announce Type: cross Abstract: Heimdall, a learned controller, calls slow on 52% of an SSD's reads when 30% are slow. Uncertainty estimators, the remedy prescribed for its wrong guesses, cannot find them: nine of them do no better than a coin. The drive can. On one trace, 2% of the reads of 4 to 8 KiB that found less than 4 KiB outstanding while the drive was answering came back slow, and Heimdall called 96% of them slow. A 4,096-entry table of how the drive's recent reads came back, untrained and 21 ns a decision, ranks the wrong guesses at a median AUROC of 0.68 on 57 held-out traces, where the nine sit at 0.46 to 0.52, and it predicts slow reads better than Heimdall itself.

## Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models

DevFeed: [Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models](<https://devfeed.tech/articles/benchleaking-is-accelerating-benchmark-contamination-has-worsened-sharply-in-the-past-month-from-post-training-down-to-pre-training-base-models-137683.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21315>)

Author: Yiwei Qin, Muhang Xie, Xuefeng Li, Mohan Jiang, Dayuan Fu, Tiantian Mi, Qipeng Guo, Pengfei Liu

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [post-training](<https://devfeed.tech/topics/post-training.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [audits](<https://devfeed.tech/tags/audits.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [capability](<https://devfeed.tech/tags/capability.md>), [continuation](<https://devfeed.tech/tags/continuation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [data](<https://devfeed.tech/tags/data.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [deepseek-v4](<https://devfeed.tech/tags/deepseek-v4.md>), [deepseek-v4-flash](<https://devfeed.tech/tags/deepseek-v4-flash.md>), [few-shot](<https://devfeed.tech/tags/few-shot.md>), [humaneval](<https://devfeed.tech/tags/humaneval.md>), [leaderboard](<https://devfeed.tech/tags/leaderboard.md>), [leak](<https://devfeed.tech/tags/leak.md>), [mimo](<https://devfeed.tech/tags/mimo.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [open](<https://devfeed.tech/tags/open.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

The authors report that benchmark test examples are appearing in model outputs, including in pre-training base models, where scores may reflect recall rather than capability. In few-shot continuation tests across 20 benchmarks and four public base models, ERNIE-5.5-Flash-Base leaked 96% of HumanEval problems with reference implementations and 85% of OpenBookQA questions with answers; DeepSeek-V4-Flash-Base leaked 9% and none, respectively. Their Anti-Hack score recomputes benchmark results with leaked examples marked wrong: averaged across the suite, this reduced scores by 16.6 points for ERNIE and 1.5 for DeepSeek, with MiMo-V2.5 at 5.0 and Hy3-preview at 1.3. The abstract also cites post-training audits in which an agent stated hidden test requirements in 23% of attempts and retained them when tasks contradicted them. The authors call for decontamination reports and contamination-adjusted scores.

### Source excerpt

arXiv:2611.21315v1 Announce Type: new Abstract: Benchmark contamination is not new, but recent model releases show it in a sharper form. We use the term benchleaking for the case in which a benchmark's own test examples enter training data, so that a score reflects recall rather than capability. This differs from benchmaxxing, which optimizes toward a benchmark without seeing its test set. We document two escalations over the past month. First, in post-training, the traces have become explicit: recent audits show an agent stating hidden test requirements it was never given in 23% of its attempts, where its predecessor did so in none, and continuing to follow them after the task is changed to contradict them (Schaeffer and R\"auker, 2026; Lo and Lo, 2026). Second, the problem has reached pre-training. Using few-shot continuation on 20 benchmarks across four public base models, we find that some base models write out other test examples---complete questions, options, and answers---that appear nowhere in their input. For the recently released ERNIE-5.5-Flash-Base, 96% of HumanEval problems leak with their reference implementations and 85% of OpenBookQA questions leak with their answers, against 9% and none for DeepSeek-V4-Flash-Base. To quantify what a reported score owes to such examples, we define an Anti-Hack score: the benchmark score recomputed with every leaked example marked wrong and the number of examples unchanged. Averaged over the suite, it costs DeepSeek-V4-Flash-Base 1.5 points and Hy3-preview-Base 1.3, against 5.0 for MiMo-V2.5-Base and 16.6 for ERNIE-5.5-Flash-Base. Open base models are the foundation of much downstream research, so benchleaking now affects more than leaderboard rankings. We call for decontamination reports and contamination-adjusted scores to become standard practice.

## Self-Improvement Memory Enables Continual Recursive Self-Improvement

DevFeed: [Self-Improvement Memory Enables Continual Recursive Self-Improvement](<https://devfeed.tech/articles/self-improvement-memory-enables-continual-recursive-self-improvement-137352.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18856>)

Author: Xin Yu, Fei Xia, Jiaqi Pan, Youbao Tang, Chengzhi Liu, Adams Yu, Sicong Jiang, Yanfei Chen, Lingzhou Xue, Ting Yu

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [coding](<https://devfeed.tech/tags/coding.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [continuation](<https://devfeed.tech/tags/continuation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [domains](<https://devfeed.tech/tags/domains.md>), [edits](<https://devfeed.tech/tags/edits.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [extend](<https://devfeed.tech/tags/extend.md>), [harness](<https://devfeed.tech/tags/harness.md>), [history](<https://devfeed.tech/tags/history.md>), [memory](<https://devfeed.tech/tags/memory.md>), [models](<https://devfeed.tech/tags/models.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [recursive-self-improvement](<https://devfeed.tech/tags/recursive-self-improvement.md>), [self-improvement](<https://devfeed.tech/tags/self-improvement.md>)

### AI overview

Self-Improvement Memory (SIM) helps harness optimizers continue optimization across compatible benchmarks or policy models by grouping edits around recurring harness problems and recording each edit's rationale, predicted and observed outcomes, and evaluation context. As optimization continues, new attempts extend that history. Across four domains, SIM achieved higher final scores than harness-only and root-cause-only continuation in cross-dataset and pairwise cross-model comparisons. In coding, paired continuations across ten independent source runs improved final scores by an average of 2.36 points over harness-only continuation (p=0.003).

### Source excerpt

arXiv:2611.18856v1 Announce Type: cross Abstract: Harness optimization is often studied with a fixed policy model and benchmark. When either changes, the optimizer must decide which parts of an evolved harness to reuse or revise, but the harness alone may not explain why earlier edits were made or how they performed. In this work, we study Continual Recursive Self-Improvement (CRSI), a setting for continuing existing harness-optimization methods across compatible benchmarks or policy models. We propose Self-Improvement Memory (SIM) for the harness optimizer, rather than the task-solving policy. SIM groups edits by recurring harness problem and records their rationale, predicted and observed outcomes, and evaluation context. New attempts extend this history as optimization continues. In our experiments across four domains, SIM achieves higher final scores than harness-only and root-cause-only continuation in cross-dataset and pairwise cross-model comparisons. Across ten independent source runs in coding, paired continuations with SIM improve final scores by an average of 2.36 points over harness-only continuation ($p=0.003$).

## SkillBoost: A Constrained Exploration-Exploitation Process for Mitigating Skill Overfitting

DevFeed: [SkillBoost: A Constrained Exploration-Exploitation Process for Mitigating Skill Overfitting](<https://devfeed.tech/articles/skillboost-a-constrained-exploration-exploitation-process-for-mitigating-skill-overfitting-137268.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.26643>)

Author: Hongqiang Lin, Chao Liu, Xiaofan Bai, Xuan Jin, Yuhong Li, Nenggan Zheng, Xipeng Cao

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Agent Skills](<https://devfeed.tech/topics/agent-skills.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [acceptance](<https://devfeed.tech/tags/acceptance.md>), [agents](<https://devfeed.tech/tags/agents.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data-driven](<https://devfeed.tech/tags/data-driven.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [exploitation](<https://devfeed.tech/tags/exploitation.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [failures](<https://devfeed.tech/tags/failures.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generate](<https://devfeed.tech/tags/generate.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [model](<https://devfeed.tech/tags/model.md>), [optimize](<https://devfeed.tech/tags/optimize.md>), [optimized](<https://devfeed.tech/tags/optimized.md>), [overfitting](<https://devfeed.tech/tags/overfitting.md>), [skill](<https://devfeed.tech/tags/skill.md>)

### AI overview

SkillBoost treats an LLM agent's skills as trainable states and addresses overfitting when optimizing them from limited interaction trajectories. It combines structured exploitation, which locates observed failures in editable skill components; prior-guided exploration, which generates diverse repair candidates; and verified acceptance, which adopts a candidate only if it improves performance within a regression bound. Across 23 model-benchmark configurations, the authors report state-of-the-art performance, less overfitting, and better results than human-crafted and LLM-generated skills. Transfer experiments indicate optimized skills can be reused by other agents on similar tasks.

### Source excerpt

arXiv:2607.26643v3 Announce Type: replace Abstract: Enabling large language model (LLM) agents to accumulate and reuse experience from past interactions remains a central challenge in real-world applications. A promising solution is to treat skills as trainable states and optimize them in the same way as model parameters in neural network training. However, data-driven skill optimization is prone to overfitting to the limited trajectories collected from real environments. Overexploiting these trajectories overfits the current batch, while unconstrained exploration causes regression on previously solved cases. This tension motivates a constrained search view of skill self-evolution, governed by an exploration--exploitation trade-off. We propose SkillBoost, a three-stage framework that mitigates both risks: structured exploitation localizes observed failures to editable skill components, prior-guided exploration draws on prior knowledge in the LLM to generate diverse repair candidates, and verified acceptance commits a candidate only when it improves performance within a regression bound. Experiments across 23 model--benchmark configurations show that SkillBoost achieves state-of-the-art performance while mitigating overfitting, outperforming both human-crafted and LLM-generated skills. Transfer experiments further show that optimized skills can be reused by other agents on similar tasks. The code is available at https://github.com/HQ-Lin/SkillBoost

## Verifiable Random Sampling of Private Data Without a Trusted Single Party

DevFeed: [Verifiable Random Sampling of Private Data Without a Trusted Single Party](<https://devfeed.tech/articles/verifiable-random-sampling-of-private-data-without-a-trusted-single-party-137666.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21222>)

Author: Muhammed Karamuk, Miko{\\l}aj Glinka, Ruben Wolff

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [Cryptography](<https://devfeed.tech/topics/cryptography.md>), [audit](<https://devfeed.tech/topics/audit.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-benchmarks](<https://devfeed.tech/tags/ai-benchmarks.md>), [audit](<https://devfeed.tech/tags/audit.md>), [audits](<https://devfeed.tech/tags/audits.md>), [beacon](<https://devfeed.tech/tags/beacon.md>), [cryptography](<https://devfeed.tech/tags/cryptography.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [data](<https://devfeed.tech/tags/data.md>), [data-privacy](<https://devfeed.tech/tags/data-privacy.md>), [decentralized](<https://devfeed.tech/tags/decentralized.md>), [private](<https://devfeed.tech/tags/private.md>), [random](<https://devfeed.tech/tags/random.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [security](<https://devfeed.tech/tags/security.md>)

### AI overview

NoCherry is designed to prevent cherry-picking when auditing a private dataset: the holder commits to hashes of all items and registers the commitment with independent servers, then a decentralized randomness beacon value from two rounds later determines the sample. Servers issue signed receipts so others can verify that the disclosed sample matches the selection. The stated unbiasedness depends on the beacon threshold holding and at least one required server being honest; a pre-agreed name checked with the servers is intended to prevent repeated or abandoned draws. The protocol aims to let auditors verify representative sampling without requiring the holder to reveal the full dataset, for uses such as private benchmarks, quality claims, security audits, fairness review, vulnerability handling, and compliance sampling.

### Source excerpt

arXiv:2611.21222v1 Announce Type: cross Abstract: Problem: When a party supports claims about a private dataset by disclosing a sample, auditors cannot tell whether it was drawn at random or "cherry-picked" to hide flaws in the rest of the data. Solution: The data holder commits to the hashes of every item in its dataset and registers the commitment with independent servers. Selection then waits on a decentralized randomness beacon: a network that jointly emits a fresh random value every few seconds, outside the control of the data holder and the auditors. The value emitted two beacon rounds after the commitment picks the sample. Each server returns a signed receipt, so anyone can later check that the revealed sample is exactly the one that value prescribes. Selection from the committed set stays unbiased as long as the beacon's threshold holds and one required server is honest, and a pre-agreed name, checked with the servers, stops repeated or abandoned draws. We call this protocol NoCherry. Applications: Any audit that spot-checks a private corpus: private AI benchmarks and held-out test sets, dataset quality claims, code and security audits, test-fairness review, vulnerability handling, and compliance sampling. Auditors need not trust the data holder to hand over a representative sample of the committed data, since the protocol proves it, and data holders need not expose their entire dataset.

## Not All Data Leakage Is Equal: A Controlled Benchmark of Evaluation Inflation in Tabular Classification

DevFeed: [Not All Data Leakage Is Equal: A Controlled Benchmark of Evaluation Inflation in Tabular Classification](<https://devfeed.tech/articles/not-all-data-leakage-is-equal-a-controlled-benchmark-of-evaluation-inflation-in-tabular-classification-138747.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18598>)

Author: Ankit Sharma

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [Feature Engineering](<https://devfeed.tech/topics/feature-engineering.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [machine learning overfitting](<https://devfeed.tech/topics/machine-learning-overfitting.md>)

Tags: [auditing](<https://devfeed.tech/tags/auditing.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data-leakage](<https://devfeed.tech/tags/data-leakage.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [inflation](<https://devfeed.tech/tags/inflation.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [roc-auc](<https://devfeed.tech/tags/roc-auc.md>)

### AI overview

In this benchmark, leakage mechanisms differed substantially in how much they inflated held-out ROC-AUC. Across six public datasets, two synthetic stress tests, three model families, and repeated stratified splits, global target encoding raised ROC-AUC by an average 2.42 percentage points, supervised feature selection by 1.93, and duplicate contamination by 1.52; all three effects remained significant after multiplicity correction. Global standardization and median imputation had near-zero average effects under the i.i.d. settings studied. A target-encoding stress test found sharply worse leakage with sparser categories. The authors recommend preventing all dependence on held-out data, with especially aggressive auditing of label-dependent transformations and observation reuse.

### Source excerpt

arXiv:2611.18598v1 Announce Type: new Abstract: Data leakage is widely recognized as a threat to trustworthy machine-learning evaluation, yet the phrase covers mechanisms with very different statistical consequences. We present a controlled empirical benchmark that measures the inflation in held-out ROC-AUC caused by five common leakage mechanisms: global standardization, global median imputation, duplicate contamination, supervised feature selection before splitting, and target encoding before splitting. The benchmark spans six public datasets and two synthetic stress tests, three model families, and five repeated stratified train/test splits, yielding 600 paired clean-versus-leaked comparisons. To avoid pseudo-replication, inference is performed on dataset-level effects with cluster bootstrap confidence intervals and Holm-corrected one-sided Wilcoxon tests. Label-dependent or direct data-reuse mechanisms produced systematic optimism: global target encoding increased ROC-AUC by 2.42 percentage points on average (95% cluster-bootstrap CI 0.62--4.89), global supervised feature selection by 1.93 points (0.63--3.39), and duplicate contamination by 1.52 points (0.40--2.80). All three remained significant after multiplicity correction. In contrast, global scaling and global median imputation had near-zero average effects under the i.i.d. settings studied here. A controlled target-encoding stress test further shows that leakage becomes sharply worse as categories become sparse. The results support a severity-aware view of leakage: evaluation pipelines should prevent all held-out-data dependence, but label-dependent transformations and observation reuse deserve especially aggressive auditing.

## DECODE: Decomposable Environments for COntrolled Sequential-Decision Evaluation

DevFeed: [DECODE: Decomposable Environments for COntrolled Sequential-Decision Evaluation](<https://devfeed.tech/articles/decode-decomposable-environments-for-controlled-sequential-decision-evaluation-135889.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18367>)

Author: Diego Silva Prado, Bruno Caldas de Souza, Tiago Amaral Penteado de Jesus, Ricardo Guimar\\~aes Herrmann, Cristiano Breuel, Cristiano Bevitori Maffia De Oliveira

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [catboost](<https://devfeed.tech/topics/catboost.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [consumer](<https://devfeed.tech/tags/consumer.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decode](<https://devfeed.tech/tags/decode.md>), [environments](<https://devfeed.tech/tags/environments.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [generator](<https://devfeed.tech/tags/generator.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [horizon](<https://devfeed.tech/tags/horizon.md>), [information](<https://devfeed.tech/tags/information.md>), [logs](<https://devfeed.tech/tags/logs.md>), [model](<https://devfeed.tech/tags/model.md>), [optimum](<https://devfeed.tech/tags/optimum.md>), [planning](<https://devfeed.tech/tags/planning.md>), [q-fin-rm](<https://devfeed.tech/tags/q-fin-rm.md>), [rules](<https://devfeed.tech/tags/rules.md>), [score](<https://devfeed.tech/tags/score.md>), [search](<https://devfeed.tech/tags/search.md>), [sequence](<https://devfeed.tech/tags/sequence.md>), [sequential-decision](<https://devfeed.tech/tags/sequential-decision.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [state](<https://devfeed.tech/tags/state.md>)

### AI overview

DECODE tests how planning rules and available information affect sequential decision systems while holding the rest of the solver fixed. Its consumer-credit collections generator offers five partially observed environments with known dynamics, varying memory and process noise; comparisons specify state inputs, response inputs, and planning horizon. In the deterministic base environment, extending planning from one to six months raises two gradient-boosted solvers' scores by 15-17 percentage points of the exact optimum. The best graded solver still falls 15.6 points short. In one intervention sequence, three state-input changes lower its score by 6.3 points; replacing its log-fitted repayment model with the true response, evaluated using hidden borrower traits, raises it by 15.6, recovering a net nine points. These intermediate effects depend on the model and intervention order, so they do not uniquely attribute the performance gap to separate causes.

### Source excerpt

arXiv:2611.18367v1 Announce Type: new Abstract: When a sequential decision system falls short of an optimum, its score does not reveal which modifications would improve its decisions. DECODE is a benchmark for testing such modifications through controlled experiments on planning and available information. Its consumer-credit collections generator provides five partially observed environments that share known dynamics matrices and vary a memory parameter and process noise. The experiments change selected inputs or planning rules while keeping the rest of the solver fixed. Each comparison specifies the solver's state inputs, response inputs and planning horizon. On the deterministic base environment, which was selected for a large gap between six-month and one-month planning with the hidden state, extending planning from one to six months increases the scores of two gradient-boosted solvers by 15 to 17 percentage points of the exact optimum, computed with the hidden state and borrower traits. Even at this horizon, the best graded solver, whose repayment response is fitted on logs, remains 15.6 points below that optimum. In a specified intervention sequence, three changes to its state inputs and their evaluation during search lower its score by 6.3 points, and replacing its fitted repayment model with the true response, evaluated with hidden borrower traits, then raises it by 15.6, a net recovery of nine of those points. Because the intermediate effects are conditional on the repayment model and the order of the changes, they do not uniquely assign the gap to separate causes. Within these limits, DECODE makes hypotheses about solver performance testable through specified interventions under known dynamics and explicit information conditions.

## When evaluation becomes intervention in clinical AI

DevFeed: [When evaluation becomes intervention in clinical AI](<https://devfeed.tech/articles/when-evaluation-becomes-intervention-in-clinical-ai-135185.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18297>)

Author: Anthony Abou Mrad, Aviona Conti, Samuel Schmidgall, Jayanth Pratap, Ruhika Iyer, Joseph Kim, King Shi, Yann Sakref, Luis M. Ahumada, Lisa Knowlton, Andrew M. Cameron, Khyzer Aziz, Jeffrey K. Jopling

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Trustworthy AI](<https://devfeed.tech/topics/trustworthy-ai.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [human review](<https://devfeed.tech/topics/human-review.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [clinical-ai](<https://devfeed.tech/tags/clinical-ai.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>)

### AI overview

AI judges can scale rubric-based review of clinical AI outputs, but using their assessments to choose prompts or train models makes evaluation part of the intervention. This selection may favor errors the judge misses. The abstract calls for reports that specify what a judge assesses and an independent clinical test for each change guided by judge assessments.

### Source excerpt

arXiv:2611.18297v1 Announce Type: new Abstract: AI judges, models that grade another system's output against a rubric, extend clinical review to the scale at which clinical AI writes. When teams use judge assessments to select a prompt or train a model, the judge becomes part of the intervention. Selection can then favor errors the judge overlooks. We describe what a judge's report must contain and the independent clinical test each judge-driven change needs.

## GDP.xlsx: Benchmarking Frontier Models on Professional Workbooks

DevFeed: [GDP.xlsx: Benchmarking Frontier Models on Professional Workbooks](<https://devfeed.tech/articles/gdp-xlsx-benchmarking-frontier-models-on-professional-workbooks-135108.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17766>)

Author: Suhaas Garre, Sushant Mehta, Edwin Chen

Published: 2026-11-16T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Frontier AI](<https://devfeed.tech/topics/frontier-ai.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [argon](<https://devfeed.tech/tags/argon.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [can](<https://devfeed.tech/tags/can.md>), [cells](<https://devfeed.tech/tags/cells.md>), [complete](<https://devfeed.tech/tags/complete.md>), [completeness](<https://devfeed.tech/tags/completeness.md>), [container](<https://devfeed.tech/tags/container.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [csv](<https://devfeed.tech/tags/csv.md>), [domains](<https://devfeed.tech/tags/domains.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [frontier-models](<https://devfeed.tech/tags/frontier-models.md>), [models](<https://devfeed.tech/tags/models.md>), [workbooks](<https://devfeed.tech/tags/workbooks.md>)

### AI overview

GDP_xlsx measures how well spreadsheet agents complete realistic professional workbook requests, and even the top configuration passes fewer than half of attempts. The benchmark pairs 70 requests across 13 domains with complete workbooks or CSVs and 791 binary rubric criteria; agents use spreadsheet tools in a container, and an agentic judge checks the files. Strict pass@1 requires meeting every criterion, including completeness and exclusivity for set-valued answers. Across 20 configurations, Gemini 4 Argon scores highest at 38.3%, while the median is 19.3%. The evaluation harness and three tasks are public; 67 tasks are held out to reduce contamination.

### Source excerpt

arXiv:2611.17766v1 Announce Type: cross Abstract: We introduce GDP_xlsx, a benchmark of 70 realistic workplace requests written by professionals in 13 domains, each paired with an expert rubric of binary criteria and with the complete workbook or CSV file it refers to. Together the files hold 2.18 million populated cells, and the rubrics 791 criteria. Agents work in a container with spreadsheet tools, and an agentic judge that can open the file applies the rubric. Completeness and exclusivity criteria require set-valued answers to be found in full and with nothing extra. We rank models by strict pass@1, the share of attempts that meet every criterion. Across 20 model configurations, including frontier models from Google, Anthropic, and OpenAI, the top-scoring one, Gemini 4 Argon, passes only 38.3% of attempts, and the median score is 19.3%, leaving wide headroom. We release the evaluation harness and three public tasks; the other 67 are held out to protect the leaderboard from contamination.

## JFinBench: A Continuously Refreshable Japanese Financial Benchmark from the Latest Certification Exams

DevFeed: [JFinBench: A Continuously Refreshable Japanese Financial Benchmark from the Latest Certification Exams](<https://devfeed.tech/articles/jfinbench-a-continuously-refreshable-japanese-financial-benchmark-from-the-latest-certification-exams-135171.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18130>)

Author: Issa Sugiura, Taro Makino, Takashi Ishida

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [Finance](<https://devfeed.tech/topics/finance.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Frontier AI](<https://devfeed.tech/topics/frontier-ai.md>), [gpt-6-astra](<https://devfeed.tech/topics/gpt-6-astra.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [accounting](<https://devfeed.tech/tags/accounting.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [banking](<https://devfeed.tech/tags/banking.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [certification](<https://devfeed.tech/tags/certification.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [exam](<https://devfeed.tech/tags/exam.md>), [exams](<https://devfeed.tech/tags/exams.md>), [financial](<https://devfeed.tech/tags/financial.md>), [gpt-6-astra](<https://devfeed.tech/tags/gpt-6-astra.md>), [human-in-the-loop](<https://devfeed.tech/tags/human-in-the-loop.md>), [japanese](<https://devfeed.tech/tags/japanese.md>), [kimi-k3-1-pro](<https://devfeed.tech/tags/kimi-k3-1-pro.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [leak](<https://devfeed.tech/tags/leak.md>), [llms](<https://devfeed.tech/tags/llms.md>)

### AI overview

JFinBench provides a refreshable benchmark for Japanese financial knowledge, with 321 examples drawn from the 2026 editions of six certification exams in accounting, tax, financial planning, and banking. Its human-in-the-loop pipeline uses a coding agent to handle exam-specific formats and a human to verify every example and return corrections, making it easier to incorporate new editions and mitigate question leakage into training data. In tests of 12 LLMs, Claude Fable 5 and GPT-6 Astra scored above 93% overall; the best open-weight model, Kimi K3.1 Pro, was 12 percentage points behind Claude Fable 5. Terminal and web access improved every model and narrowed the gap to about four points, while rankings stayed largely similar. The authors say they release the dataset, evaluation code, and construction pipeline.

### Source excerpt

arXiv:2611.18130v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly applied to finance, creating a need for benchmarks that evaluate their capabilities across financial systems with different regulations and practices. However, existing financial benchmarks provide limited coverage of country-specific financial systems, and they are difficult to keep up to date as released questions leak into training data. We introduce JFinBench, a Japanese financial benchmark comprising 321 examples from the 2026 editions of six regularly administered certification exams covering accounting, tax, financial planning, and banking. To construct it, we develop a human-in-the-loop agentic pipeline in which a coding agent writes exam-specific conversion code and a review page, while a human verifies every example and feeds corrections back to the agent. This pipeline adapts to heterogeneous exam formats and reduces the effort of incorporating new exam editions, enabling periodic refreshes that mitigate data contamination. Evaluating 12 recent LLMs, we find that Claude Fable 5 and GPT-6 Astra exceed 93% overall, while even the best open-weight model, Kimi K3.1 Pro, trails Claude Fable 5 by 12 percentage points. When models can additionally use a terminal and web access, they retrieve the knowledge they lack and every model improves. The gap narrows but about 4 points remain, and the ranking of models stays largely the same, suggesting that knowledge held in the model itself remains valuable even with tools. We release the dataset, the evaluation code, and the construction pipeline.

## Stress-Testing Sandbagging Detection Under Adaptive Attack

DevFeed: [Stress-Testing Sandbagging Detection Under Adaptive Attack](<https://devfeed.tech/articles/stress-testing-sandbagging-detection-under-adaptive-attack-132937.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.14950>)

Author: Bhoj Rani Soopal, Aarohi Agrawal, Josh Karthikeyan, Edward Lue Chee Lip

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [ai security](<https://devfeed.tech/topics/ai-security.md>), [Adversarial attacks](<https://devfeed.tech/topics/adversarial-attacks.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [attack](<https://devfeed.tech/tags/attack.md>), [audit](<https://devfeed.tech/tags/audit.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [detection](<https://devfeed.tech/tags/detection.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [math](<https://devfeed.tech/tags/math.md>), [model](<https://devfeed.tech/tags/model.md>), [qwen](<https://devfeed.tech/tags/qwen.md>)

### AI overview

The abstract reports that a sandbagging attack adapted to a text-based audit can sharply reduce detection while preserving its effect on model accuracy. On Qwen 2.5-3B, two retraining rounds lowered audit AUC from 0.978 to 0.560, while the prompt marker still reduced accuracy by 40-49 points; the audit missed all retrained attacks, with AUCs of 0.451-0.576. The attack succeeded in 7 of 8 retrained seeds, including a reported Llama run among the missed cases. Activation probes flagged 99% of benign fine-tune examples and matched probes transferred to just 1 of 5 held-out runs. Eight additional trigger attacks activated under their own prompts but not the detectors' prompt. These results show limits of the tested detectors against adaptive attacks; the excerpt does not establish how broadly they generalize.

### Source excerpt

arXiv:2611.14950v1 Announce Type: cross Abstract: Developers, auditors and researchers read evaluation scores as evidence of what a model can do. A model that sandbags, scoring low on purpose while keeping the capability, turns a low score into false reassurance. Sandbagging detectors are usually tested once against a fixed attack, but a real attacker sees what gets flagged and adapts. We build sandbagging models by fine-tuning small open-weight models to fail math word problems (GSM8K) they can otherwise solve, but only when an audit marker is in the prompt. On Qwen 2.5-3B, we adapt the attack over two retraining rounds against a text-based audit. Two rounds of retraining cut the text audit's AUC from 0.978 to 0.560 while the marker still removes 40-49 points of accuracy. The attack worked in 7 of 8 retrained seeds, and the audit missed every retraining (AUC 0.451-0.576), including one Llama run. Activation probes also flag 99% of benign fine-tunes' examples, and matched probes transfer to only 1 of 5 held-out training runs. Eight further trigger attacks are all active under their own prompt but not under the detectors' prompt. A detector that catches a fixed attack has not been shown to catch an adaptive one.

## A Free Knob: A Sensitivity Audit for Calibration in Threshold-based Evaluation

DevFeed: [A Free Knob: A Sensitivity Audit for Calibration in Threshold-based Evaluation](<https://devfeed.tech/articles/a-free-knob-a-sensitivity-audit-for-calibration-in-threshold-based-evaluation-132830.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.33457>)

Author: Md Tanveer Hossain Munim, Bijoy Ahmed Saiem, Al-Amin Sany, Tanzima Hashem

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [audit](<https://devfeed.tech/topics/audit.md>)

Tags: [audit](<https://devfeed.tech/tags/audit.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bias](<https://devfeed.tech/tags/bias.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [csi](<https://devfeed.tech/tags/csi.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [imagery](<https://devfeed.tech/tags/imagery.md>), [map](<https://devfeed.tech/tags/map.md>), [model](<https://devfeed.tech/tags/model.md>), [order](<https://devfeed.tech/tags/order.md>), [r](<https://devfeed.tech/tags/r.md>), [regression](<https://devfeed.tech/tags/regression.md>), [report](<https://devfeed.tech/tags/report.md>), [reverse](<https://devfeed.tech/tags/reverse.md>), [score](<https://devfeed.tech/tags/score.md>), [share](<https://devfeed.tech/tags/share.md>)

### AI overview

At fixed rare-event thresholds, max-pooled CSI mixes spatial event placement with amplitude calibration. The proposed FreeKnob Audit fits a monotone calibration map on held-out data for each system; because the map preserves pixel order and needs no retraining, score changes under the audit quantify a calibration-sensitive component rather than improved spatial ranking. In the reported results, calibration changed two SEVIR checkpoints' CSI difference from −29.5% to +5.3%; among 450 system contrasts, 64 reversed sign, and pooled frequency-bias separation tracked score movement (r = 0.896 for maps fitted after pooling). At CasCast's published extreme-event point, calibration removed 88%-106% of the cascade's gain over its backbone across three controls. The authors recommend reporting pooled frequency bias and a symmetric held-out audit alongside rare-event scores; the effect requires both a fixed threshold and poor calibration at that threshold.

### Source excerpt

arXiv:2609.33457v2 Announce Type: replace Abstract: Many benchmarks for rare events score a prediction in three steps: pool the prediction and the target over spatial blocks, threshold each pooled field, and score the resulting contingency table. We show that at a fixed rare threshold the max-pooled Critical Success Index (CSI) cannot separate two different things: how well a system places events, and how well its amplitudes are calibrated. Sharp observations push many blocks above the threshold, while the attenuated predictions of squared-error regression leave those same blocks below it. A monotone calibration map is fitted on held-out data and applied separately to each system. The map needs no retraining and cannot change the order of any two pixels, so whatever part of a difference it reproduces cannot be evidence of better spatial ranking. We report how much of a reported difference the map removes. On SEVIR, two released checkpoints of one architecture differ by -29.5% in extreme-threshold CSI before the control and by +5.3% after it. Across 450 pairwise contrasts among 6 systems, how far apart two systems sit in pooled frequency bias predicts how far their CSI difference moves under the control (r = 0.896 when the map is fitted after pooling), and 64 contrasts reverse sign. At CasCast's published extreme-event operating point, calibration alone removes 88%-106% of the cascade's gain over its backbone across three controls. Calibration also reveals advantages hidden by a better-calibrated baseline. On geostationary infrared imagery, a larger share of the score can be explained by calibration as events get rarer. The problem therefore needs two things at the same time: a threshold that is fixed in advance, and a model whose output is poorly calibrated at that threshold. We recommend reporting pooled frequency bias and a symmetric held-out FreeKnob Audit alongside rare-event pool-and-threshold scores.

## Does a Self-Learning Migration Agent Learn? An Audited Evaluation of Evidence-Graded Experience Memory for Oracle-to-PostgreSQL Migration

DevFeed: [Does a Self-Learning Migration Agent Learn? An Audited Evaluation of Evidence-Graded Experience Memory for Oracle-to-PostgreSQL Migration](<https://devfeed.tech/articles/does-a-self-learning-migration-agent-learn-an-audited-evaluation-of-evidence-graded-experience-memory-for-oracle-to-postgresql-migration-131729.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16143>)

Author: Oleg Grynets, Vasyl Lyashkevych, Valeriia Oliferuk, Vladyslav Babenko, Danylo Kovalchuk, Dmytro Kostetskyi

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>)

Tags: [22](<https://devfeed.tech/tags/22.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [added](<https://devfeed.tech/tags/added.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [all](<https://devfeed.tech/tags/all.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [audit](<https://devfeed.tech/tags/audit.md>), [base](<https://devfeed.tech/tags/base.md>), [conformity](<https://devfeed.tech/tags/conformity.md>), [copy](<https://devfeed.tech/tags/copy.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cs-lo](<https://devfeed.tech/tags/cs-lo.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [experience](<https://devfeed.tech/tags/experience.md>), [learn](<https://devfeed.tech/tags/learn.md>), [learning](<https://devfeed.tech/tags/learning.md>), [memory](<https://devfeed.tech/tags/memory.md>), [migration](<https://devfeed.tech/tags/migration.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>)

### AI overview

Persistent memory did not show a reliable cross-task learning benefit in the audited Oracle-to-PostgreSQL migration evaluation. On documentation-derived pairs, its apparent reference-conformity gain was small and fell to 0.002 on leak-free units; a nearest-target copy baseline outperformed every LLM system. Grouping the evaluation by reference reduced copying, and a rerun found no statistically distinguishable memory or policy effect. In a separate test on real PL/SQL, 10-22% of PostgreSQL-accepted units were behaviorally incorrect, while a human-gated fix cycle raised demonstrably correct results from 98/216 to 167/224. Retrieval margin also declined 54% as the knowledge base grew from 12 to 1,200 entries. The abstract presents these findings as reasons to audit leakage, copying, behavioral correctness, and memory stability when evaluating agent learning.

### Source excerpt

arXiv:2611.16143v1 Announce Type: cross Abstract: Do large language model (LLM) migration agents learn from accumulated experience, or do apparent gains reflect retrieval and reference reuse? We evaluate an evidence-graded agent for Oracle-to-PostgreSQL migration across four studies. On 1,833 documentation-derived source-reference pairs, persistent memory improved reference conformity (RC) over a no-memory baseline by 0.015 [0.004, 0.025], while policy adaptation added no gain (-0.004 [-0.010, 0.001]). An audit found that 264 of 367 held-out units (72%) shared a PostgreSQL reference with training, and the memory effect on the 103 leak-free units fell to 0.002 [-0.008, 0.011]. A model-free nearest-target copy baseline scored RC 0.222, outperforming all LLM systems (at most 0.163). A reference-grouped split reduced copying to 0.119; a rerun on 374 units in 118 independent target groups found no statistically distinguishable memory effect (0.005 [-0.003, 0.017]) and no policy effect. A separate Oracle-grounded evaluation on 216-224 real PL/SQL units showed that 10-22% of PostgreSQL-accepted units were behaviorally incorrect; a human-gated fix cycle increased demonstrably correct units from 98/216 to 167/224. A write-back study showed a 54% decline in retrieval margin as the knowledge base grew from 12 to 1,200 entries. We derive six evaluation rules that distinguish cross-task learning from reference leakage, copying, metric artifacts, and unstable feedback.

[Next page](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md?cursor=WyIyMDI2LTExLTEzVDA1OjAwOjAwKzAwOjAwIiwgImUyZmMwZmE1LWUzM2EtNDliYi1iOWQyLWFjNzg4ZTBhYWYzOCJd>)