# reproducibility

Computational reproducibility is obtaining consistent results using the same data, computational steps, methods, code, and analysis conditions.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Measurable Does Not Imply Transportable: A Prospective Audit of Model Explanations in Multi-Year Field Trials

DevFeed: [Measurable Does Not Imply Transportable: A Prospective Audit of Model Explanations in Multi-Year Field Trials](<https://devfeed.tech/articles/measurable-does-not-imply-transportable-a-prospective-audit-of-model-explanations-in-multi-year-field-trials-145091.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25692>)

Author: Rayhaneh Shabani Nia, Ali Karkehabadi

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [machine learning overfitting](<https://devfeed.tech/topics/machine-learning-overfitting.md>)

Tags: [2020](<https://devfeed.tech/tags/2020.md>), [2023](<https://devfeed.tech/tags/2023.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [audit](<https://devfeed.tech/tags/audit.md>), [bootstrap](<https://devfeed.tech/tags/bootstrap.md>), [concepts](<https://devfeed.tech/tags/concepts.md>), [context](<https://devfeed.tech/tags/context.md>), [cosine](<https://devfeed.tech/tags/cosine.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [environment](<https://devfeed.tech/tags/environment.md>), [failure](<https://devfeed.tech/tags/failure.md>), [feature](<https://devfeed.tech/tags/feature.md>), [feature-importance](<https://devfeed.tech/tags/feature-importance.md>), [field](<https://devfeed.tech/tags/field.md>), [fine](<https://devfeed.tech/tags/fine.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [future](<https://devfeed.tech/tags/future.md>), [half](<https://devfeed.tech/tags/half.md>), [model](<https://devfeed.tech/tags/model.md>), [transportable](<https://devfeed.tech/tags/transportable.md>)

### AI overview

In multi-year Genomes-to-Fields maize trials, reliable measurement and reproducible explanations within a year did not guarantee that explanations would carry over to future years. Coarse environment-level explanations transported well (median cosine 0.83; locked 2024 value 0.811 within the prespecified interval), while genotype-conditioned grain-yield explanations varied widely across years (median 0.30; range 0.006-0.822). GxE explanations were reproducible within a fixed context (0.77-0.95) but aligned weakly across 2020-2023 transitions (0.01-0.36). The year with the highest split-half reliability had the lowest explanation transport, and non-temporal validation overstated interaction-level transport by 0.22. A second phenotype transported strongly at both resolutions. The abstract concludes that reliability and within-context reproducibility alone do not establish that an explanation supports a reproducible scientific claim.

### Source excerpt

arXiv:2611.25692v1 Announce Type: new Abstract: Model explanations are increasingly used as scientific evidence, yet it remains unclear what they license scientists to conclude beyond the data on which they were constructed. When a fine-grained explanation fails to replicate, the failure may be attributed to noisy measurement of its target, raising a fundamental question: does reliable measurement imply a reproducible explanation? We test this directly. We formalize prospective explanation transportability: the extent to which an explanation learned from past data is recovered in a genuinely future context. This property is distinct from feature importance and measurement reliability. Using multi-year Genomes-to-Fields maize trials, where each new year is a real future context rather than a random resample, we evaluate 16 agronomically interpretable environmental concepts across five forward splits and one prospectively locked test year. Coarse environment-level explanations transport reliably (median cosine 0.83); their locked 2024 value, 0.811, fell within a prespecified 80% bootstrap percentile interval. For grain yield, genotype-conditioned explanations have a historical median cosine of 0.30 and range from 0.006 to 0.822 across years. Reliability does not order these outcomes: the year with the highest split-half reliability had the lowest explanation transport. Within-context resampling showed that GxE explanations were reproducible within a fixed context (0.77-0.95) while exhibiting lower prospective alignment across the 2020-2023 transitions (0.01-0.36). A second phenotype transports strongly at both resolutions, and non-temporal validation inflates apparent interaction-level transport by +0.22. These results show that target reliability and within-context explanation reproducibility do not by themselves establish when a model explanation can support a reproducible scientific claim.

## The Price of an Explanation: What Demanding Verifiable Artifacts Costs, and What It Buys Back

DevFeed: [The Price of an Explanation: What Demanding Verifiable Artifacts Costs, and What It Buys Back](<https://devfeed.tech/articles/the-price-of-an-explanation-what-demanding-verifiable-artifacts-costs-and-what-it-buys-back-145115.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25896>)

Author: Edgar A. Du\\'e\\~nez-Guzm\\'an, Eduardo Due\\~nez, Edgar Said Hernandez Sanchez, Suzanne Sadedin

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [few-shot](<https://devfeed.tech/topics/few-shot.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [agi](<https://devfeed.tech/tags/agi.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [budget](<https://devfeed.tech/tags/budget.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ne](<https://devfeed.tech/tags/cs-ne.md>), [decoder](<https://devfeed.tech/tags/decoder.md>), [error-correction](<https://devfeed.tech/tags/error-correction.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [llm](<https://devfeed.tech/tags/llm.md>), [mitigation](<https://devfeed.tech/tags/mitigation.md>), [program](<https://devfeed.tech/tags/program.md>), [quantum](<https://devfeed.tech/tags/quantum.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [score](<https://devfeed.tech/tags/score.md>), [search](<https://devfeed.tech/tags/search.md>), [synthesis](<https://devfeed.tech/tags/synthesis.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [test](<https://devfeed.tech/tags/test.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

On ARC-AGI-2, requiring an LLM to produce a verifiable program reduced performance from 93/120 tasks (77%) to 68/120 (57%). The authors report that most answer-only wins remained out of reach for the program-producing configurations they tested, even when a program matched every demonstration, pointing to generalization as the key bottleneck. A short evolutionary search over two sampled programs improved on best-of-two sampled code by 9.2 ± 1.4 tasks without additional model cost. Checking programs against training examples also enabled early stopping, reducing budget use by up to 38% for best-of-n sampling and 49% for the evolved arm; answer-only runs could not stop this way. The same methodology was applied to quantum error-correction decoder synthesis, producing interpretable decoders evaluated on held-out shots against a certified panel of published decoders.

### Source excerpt

arXiv:2611.25896v1 Announce Type: cross Abstract: Generative models produce many candidate solutions but provide no intrinsic signal by which to rank them. One mitigation is to demand a \emph{verifiable artifact}: a program that can be executed against the available evidence. A common concern is that requiring executable verification incurs costs in both solution quality and computational budget. We measure these two costs on the $120$-task ARC-AGI-2 evaluation set. Demanding a verifiable artifact from an LLM reduced the score from $93$ to $68$ of $120$ tasks ($77\%$ to $57\%$). Of those $25$ tasks, $21$ were reached by no program-producing configuration we ran, at any budget, by sampling or by search; yet for $27$ of the $28$ tasks that only the answer-only arm solved, some program arm held a program that reproduced every demonstration and still failed the test. The bottleneck is obtaining an explanation that \emph{generalizes}, not finding one. A short evolutionary search over two already-sampled programs, at no additional model cost, beats best-of-$2$ sampled code by $9.2 \pm 1.4$ tasks, and over a single sampled program beats $1$-shot sampling by $9.9 \pm 1.3$. Moreover, because a program can be checked against the training examples, any program-producing protocol can terminate early once a task is explained: under the same stopping rule, best-of-$n$ sampling declines up to $38\%$ of its budget and the evolved arm up to $49\%$, whereas an answer-only baseline cannot terminate early. Applying the same methodology to decoder synthesis for quantum error correction yields interpretable decoders evaluated against a certified panel of published decoders on held-out exam shots.

## An AI-Powered Multiverse Analysis of Empirical Finance Research: Evidence from Reused Natural Experiments

DevFeed: [An AI-Powered Multiverse Analysis of Empirical Finance Research: Evidence from Reused Natural Experiments](<https://devfeed.tech/articles/an-ai-powered-multiverse-analysis-of-empirical-finance-research-evidence-from-reused-natural-experiments-145477.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25184>)

Author: Ruijiang Gao, Steven Chong Xiao

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Finance](<https://devfeed.tech/topics/finance.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-coding](<https://devfeed.tech/tags/agentic-coding.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [authors](<https://devfeed.tech/tags/authors.md>), [bias](<https://devfeed.tech/tags/bias.md>), [code](<https://devfeed.tech/tags/code.md>), [coding](<https://devfeed.tech/tags/coding.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [data](<https://devfeed.tech/tags/data.md>), [difference-in-differences](<https://devfeed.tech/tags/difference-in-differences.md>), [effect](<https://devfeed.tech/tags/effect.md>), [effects](<https://devfeed.tech/tags/effects.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [finance](<https://devfeed.tech/tags/finance.md>), [journals](<https://devfeed.tech/tags/journals.md>), [research](<https://devfeed.tech/tags/research.md>)

### AI overview

AI agents analyzed 324 finance papers reusing 13 natural experiments, rebuilding data and reproducing published estimates before testing 62,409 defensible specifications. They reproduced 42% of signed main estimates under the papers' original specifications; across specification grids, an average of 25% retained the published sign with p < 0.10. Regression-discontinuity claims were more robust than difference-in-differences claims. Within experiments, robustness was higher for first-order effects, larger published t-statistics, papers with author data or code available, top-journal papers, and claims with fewer earlier papers on the same experiment. Experiments reused by more papers were not less robust on average. The abstract presents this as evidence that agentic coding can make large-scale multiverse analysis feasible.

### Source excerpt

arXiv:2611.25184v1 Announce Type: new Abstract: Empirical finance research often reuses natural experiments to study other outcomes, raising concerns about multiple testing and publication bias. We use AI agents to run a multiverse analysis of 324 papers that reuse 13 natural experiments. The agents rebuild the data, reproduce each paper's main estimate and re-estimate it across grids totaling 62,409 estimated defensible specifications. Based on the reproductions and the specification grids, we develop measures of empirical robustness. We reproduce 42% of the main estimates of claims with a signed effect under the papers' own specifications. Across all defensible specifications, on average 25% keep the published sign and have $p < 0.10$. Across experiments, claims based on regression discontinuity are more robust than difference-in-differences claims. Within experiments, claims about first-order effects, claims with larger published $t$-statistics, claims whose authors' data or code we find, claims in top journals, and claims with fewer earlier papers on the same experiment are more robust. Claims on experiments that more papers reuse are not less robust on average. Our findings quantify how much published claims depend on analysis choices, thereby showing that large-scale multiverse analysis becomes feasible with agentic coding.

## When AI Analysts Disagree: Agentic Multiverse Analysis and the Robustness of Empirical Research

DevFeed: [When AI Analysts Disagree: Agentic Multiverse Analysis and the Robustness of Empirical Research](<https://devfeed.tech/articles/when-ai-analysts-disagree-agentic-multiverse-analysis-and-the-robustness-of-empirical-research-145480.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25198>)

Author: Ruijiang Gao, Steven Chong Xiao

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-assisted](<https://devfeed.tech/tags/ai-assisted.md>), [ai-coding](<https://devfeed.tech/tags/ai-coding.md>), [ai-coding-agents](<https://devfeed.tech/tags/ai-coding-agents.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analysts](<https://devfeed.tech/tags/analysts.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [autonomous](<https://devfeed.tech/tags/autonomous.md>), [based](<https://devfeed.tech/tags/based.md>), [can](<https://devfeed.tech/tags/can.md>), [case](<https://devfeed.tech/tags/case.md>), [case-studies](<https://devfeed.tech/tags/case-studies.md>), [coding](<https://devfeed.tech/tags/coding.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [research](<https://devfeed.tech/tags/research.md>), [robustness](<https://devfeed.tech/tags/robustness.md>)

### AI overview

The agentic multiverse analysis (AMA) framework tests how empirical findings change across defensible analytical choices. In a pilot, 150 AI coding agents made similar methodological choices to human analysts, though across a narrower range; AI peer review barely changed their disagreement, while exposure to top-rated reports sharply reduced it, largely through imitation. AMA has agents map possible choices, assess them with a validity rubric, run the validated specifications, and cross-examine results under separated duties. Across ten management studies and 1,755 estimates, it reproduced the original estimates; some conclusions held across all validated specifications, while others depended on choices not discussed in the papers. The authors propose AMA measures as robustness proxies and report that analyses took hours of agent time and tens of dollars per paper.

### Source excerpt

arXiv:2611.25198v1 Announce Type: new Abstract: Every empirical analysis on observational data rests on dozens of defensible analytical choices ("forks") that its readers cannot see. We develop an agentic multiverse analysis (AMA) framework that makes a conclusion's sensitivity to those choices visible at scale, and we position it as a decision-support instrument for human experts. A pilot study first deploys 150 autonomous AI coding agents on one dataset and one set of research questions and finds that they disagree through the same defensible methodological choices human analysts make, though over a narrower range. AI peer review leaves their dispersion almost unchanged. Exposure to top-rated exemplar reports then collapses it, largely through imitation rather than analysis. Built on those lessons, AMA has agents enumerate a published paper's possible fork space, score every fork against a validity rubric, execute the validated specification grid, and cross-examine one another's work under strict separation of duties. Applied to ten empirical studies from a leading management journal's replication packages (1,755 specification estimates), the framework reproduces every paper's original estimates, and shows that some conclusions hold across every specification the assessment validates, while others are sensitive to individual analytical choices that the published texts do not discuss. The case studies suggest our AMA-based measures can serve as robustness proxies for empirical findings. AI makes multiverse analysis practical, where it once required resources few research teams can afford: our exercises take hours of agent time and tens of dollars per paper. We discuss the implications for replication policy and for AI-assisted peer review in information systems.

## Evidence from a Multi-Regime Causal Discovery Benchmark: UCMT Algorithms, Comparative Performance, and Controlled Ablation Studies

DevFeed: [Evidence from a Multi-Regime Causal Discovery Benchmark: UCMT Algorithms, Comparative Performance, and Controlled Ablation Studies](<https://devfeed.tech/articles/evidence-from-a-multi-regime-causal-discovery-benchmark-ucmt-algorithms-comparative-performance-and-controlled-ablation-studies-145172.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26383>)

Author: Mahyar Eslami

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [causality](<https://devfeed.tech/topics/causality.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [algorithms](<https://devfeed.tech/tags/algorithms.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [f1](<https://devfeed.tech/tags/f1.md>), [multiple-testing](<https://devfeed.tech/tags/multiple-testing.md>), [orientation](<https://devfeed.tech/tags/orientation.md>), [performance](<https://devfeed.tech/tags/performance.md>), [permutation](<https://devfeed.tech/tags/permutation.md>)

### AI overview

In this internal, pre-registered benchmark, UCMT v2 Global beats NOTEARS and DAGMA on directed F1 in selected comparisons but trails leading permutation-search methods in other settings. UCMT v4 improves over BOSS on CSuite at the paired-run level (+0.279; Holm p < 0.001), though the dataset-level adjusted result is not significant (p = 0.188). In a separate fixed-skeleton study on standardized iSCM data, v2 Global exceeds an adapted R²-SortnRegress orientation baseline (0.537 vs 0.468; dataset-level Holm p = 0.031), with absolute orientation accuracy of 0.563 and no established broad advantage over RESIT, ANM, or DirectLiNGAM. Ablations indicate sparsity and BIC-based refinement affect some outcomes; arbitration and automatic regime selection show no confirmed general improvement. The evidence supports empirical investigation of the UCMT family, not overall superiority or independent validation.

### Source excerpt

arXiv:2611.26383v1 Announce Type: new Abstract: We present an internal, pre-registered evaluation of the UCMT causal discovery algorithm family alongside established alternatives across heterogeneous benchmark regimes, complemented by a separately pre-registered, controlled study of UCMT v2 Global components. The contribution is both empirical and methodological: a family-stratified comparison of three actively developed algorithm configurations and targeted evidence about which internal stages affect their outcomes. In the main benchmark, UCMT v2 Global has higher directed F1 than NOTEARS and DAGMA in selected real, semi-synthetic and Gaussian-network comparisons at the paired-run level, while trailing leading permutation-search methods in other settings. The BOSS-based UCMT v4 produces a CSuite-specific directed-F1 improvement over BOSS (mean paired difference +0.279; pair-level Holm p < 0.001), but the dataset-level adjusted result is not significant (p = 0.188). In a separate fixed-skeleton ablation, UCMT v2 Global exceeds an adapted R^2-SortnRegress orientation baseline on standardized iSCM data (directed F1 0.537 vs 0.468, dataset-level Holm p = 0.031); absolute orientation accuracy is 0.563 and there is no established broad advantage over RESIT, ANM or DirectLiNGAM. Controlled ablations show that sparsity and BIC-based refinement materially affect some outcomes, whereas the examined arbitration and automatic regime-selection mechanisms do not yield confirmed general improvements. We report performance shortcomings, multiple-testing correction, runtime costs, evaluation limitations and reproducibility checks. The results support UCMT as an implemented and empirically investigated algorithm family, not a claim of overall superiority, independent validation or demonstrated commercial impact.

## AI helps researchers investigate how thousands of medical studies share code

DevFeed: [AI helps researchers investigate how thousands of medical studies share code](<https://devfeed.tech/articles/ai-helps-researchers-investigate-how-thousands-of-medical-studies-share-code-145239.md>)

Original publisher: [Read original article](<https://news.mit.edu/2026/ai-helps-researchers-investigate-how-medical-studies-share-code-1119>)

Author: Bwiza Gihana Deborah | Institute for Medical Engineering and Science

Published: 2026-11-19T22:15:00Z

Content type: news

Language: en

Sources: [MIT AI News](<https://devfeed.tech/sources/mit-ai-news.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [reusable code](<https://devfeed.tech/topics/reusable-code.md>), [Documentation](<https://devfeed.tech/topics/documentation.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [ai](<https://devfeed.tech/tags/ai.md>), [all](<https://devfeed.tech/tags/all.md>), [allowed](<https://devfeed.tech/tags/allowed.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [code](<https://devfeed.tech/tags/code.md>), [code-sharing](<https://devfeed.tech/tags/code-sharing.md>), [dana-farber-cancer-institute](<https://devfeed.tech/tags/dana-farber-cancer-institute.md>), [data](<https://devfeed.tech/tags/data.md>), [documenting-code](<https://devfeed.tech/tags/documenting-code.md>), [helps](<https://devfeed.tech/tags/helps.md>), [information](<https://devfeed.tech/tags/information.md>), [institute-for-medical-engineering-and-science-imes](<https://devfeed.tech/tags/institute-for-medical-engineering-and-science-imes.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large](<https://devfeed.tech/tags/large.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [medical](<https://devfeed.tech/tags/medical.md>), [medical-study-data](<https://devfeed.tech/tags/medical-study-data.md>), [medicine](<https://devfeed.tech/tags/medicine.md>), [meta-analysis](<https://devfeed.tech/tags/meta-analysis.md>), [mit](<https://devfeed.tech/tags/mit.md>), [mit-imes](<https://devfeed.tech/tags/mit-imes.md>), [models](<https://devfeed.tech/tags/models.md>), [repositories](<https://devfeed.tech/tags/repositories.md>), [research](<https://devfeed.tech/tags/research.md>), [reuse](<https://devfeed.tech/tags/reuse.md>), [school-of-engineering](<https://devfeed.tech/tags/school-of-engineering.md>), [science-communications](<https://devfeed.tech/tags/science-communications.md>), [share](<https://devfeed.tech/tags/share.md>), [team](<https://devfeed.tech/tags/team.md>), [thomas-sounack](<https://devfeed.tech/tags/thomas-sounack.md>), [tom-pollard](<https://devfeed.tech/tags/tom-pollard.md>), [tripod-code](<https://devfeed.tech/tags/tripod-code.md>), [used](<https://devfeed.tech/tags/used.md>)

### AI overview

An MIT and Dana-Farber-led team used large language models to identify code-sharing statements in nearly 4,000 clinical prediction studies and assess hundreds of linked repositories. Only about one in eight studies reported sharing code, and public repositories varied widely in documentation and information needed to understand and rerun analyses. The automated review also checked repository features such as software requirements and reuse terms; human reviewers found agreement stronger for concrete items, like a license file, than for judging documentation quality. The authors therefore emphasize overall patterns rather than exact counts. Sharing practices varied by journal and country, and written journal policies did not always coincide with high sharing rates. Some researchers used synthetic datasets to support reuse when patient records could not be made public. The findings are informing TRIPOD-Code, guidance intended to clarify essential elements of shared analytical code for researchers and journals.

### Source excerpt

An MIT and Dana-Farber Institute-led team used large language models to examine nearly 4,000 studies and hundreds of code repositories, revealing gaps in the information needed to scrutinize and reuse published research.

## How to Generate Synthetic Healthcare Data Using Synthea: A Hands On Tutorial

DevFeed: [How to Generate Synthetic Healthcare Data Using Synthea: A Hands On Tutorial](<https://devfeed.tech/articles/how-to-generate-synthetic-healthcare-data-using-synthea-a-hands-on-tutorial-144721.md>)

Original publisher: [Read original article](<https://www.freecodecamp.org/news/how-to-generate-synthetic-healthcare-data-using-synthea-a-hands-on-tutorial/>)

Author: Lakshmi Mahabaleshwara

Published: 2026-11-19T22:00:42Z

Content type: tutorial

Language: en

Sources: [Free Code Camp](<https://devfeed.tech/sources/freecodecamp-programming-tutorials-python-javascript-git-more.md>)

Topics: [Tutorial](<https://devfeed.tech/topics/tutorial.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Structured-data](<https://devfeed.tech/topics/structured-data.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-pipeline](<https://devfeed.tech/tags/analytics-pipeline.md>), [applications](<https://devfeed.tech/tags/applications.md>), [csv](<https://devfeed.tech/tags/csv.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [dataengineering](<https://devfeed.tech/tags/dataengineering.md>), [generate](<https://devfeed.tech/tags/generate.md>), [healthcare](<https://devfeed.tech/tags/healthcare.md>), [healthcareai](<https://devfeed.tech/tags/healthcareai.md>), [healthcaredata](<https://devfeed.tech/tags/healthcaredata.md>), [healthtech](<https://devfeed.tech/tags/healthtech.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [synthetichealthcaredata](<https://devfeed.tech/tags/synthetichealthcaredata.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>), [using](<https://devfeed.tech/tags/using.md>)

### AI overview

Synthea simulates longitudinal patient records, making it useful for prototyping and testing healthcare data workflows without using real patient histories. The tutorial shows how to build it with Java 17 or newer, generate about 100 patients, enable CSV export explicitly, and explore linked patient, encounter, condition, observation, medication, and procedure files with Python and Pandas. A fixed seed helps reproduce runs; version, configuration, and reference date also affect results. Default population overflow can produce more records than requested (115 in the described run); disabling it targets the requested count but may change how closely survivor demographics match population targets. The tutorial demonstrates basic relationship and missing-value checks and reconstructing a patient's timeline. Synthetic data is not necessarily representative, clinically accurate, unbiased, or suitable evidence for real-world model performance, so validate concepts, data quality, and representativeness before use.

### Source excerpt

Healthcare generates huge amounts of data every day, from patient visits and lab results to diagnoses, medications, and medical imaging. For developers and data engineers building healthcare applicati

## Equal Steps, Unequal Curvature: Where Muon's Update Goes Wrong

DevFeed: [Equal Steps, Unequal Curvature: Where Muon's Update Goes Wrong](<https://devfeed.tech/articles/equal-steps-unequal-curvature-where-muon-s-update-goes-wrong-143542.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24644>)

Author: Biswa Sengupta, Jinhua Wang, L\\'eo Maxime Brunswic

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [gradient\_accumulation](<https://devfeed.tech/topics/gradient-accumulation.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [arxiv](<https://devfeed.tech/tags/arxiv.md>), [band](<https://devfeed.tech/tags/band.md>), [block](<https://devfeed.tech/tags/block.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoder](<https://devfeed.tech/tags/decoder.md>), [diagnostic](<https://devfeed.tech/tags/diagnostic.md>), [hessian](<https://devfeed.tech/tags/hessian.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [input](<https://devfeed.tech/tags/input.md>), [lean](<https://devfeed.tech/tags/lean.md>), [leverage](<https://devfeed.tech/tags/leverage.md>), [matching](<https://devfeed.tech/tags/matching.md>), [matmul](<https://devfeed.tech/tags/matmul.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [momentum](<https://devfeed.tech/tags/momentum.md>), [muon](<https://devfeed.tech/tags/muon.md>), [neuron](<https://devfeed.tech/tags/neuron.md>), [order](<https://devfeed.tech/tags/order.md>), [over](<https://devfeed.tech/tags/over.md>), [parameter](<https://devfeed.tech/tags/parameter.md>), [update](<https://devfeed.tech/tags/update.md>)

### AI overview

In a 124M-parameter decoder trained on FineWeb-Edu, Muon's equal step across momentum singular directions over-steps the high-curvature top of the spectrum. Held-out gradients and Hessian-vector products show that shifting step length away from the top helps mid-training and early cooldown, but not at cooldown's end; the authors formalize when this curvature diagnostic supports intervention at Muon's update norm. With per-step norm matching, tuned baselines, and paired seeds, separately orthogonalizing Q, K, and V reduces loss by 0.0250 nats and explains the gain from block-wise variants. A cooldown spectral tilt of σ⁻¹ᐟ⁴ further reduces loss by 0.0120 nats on that baseline, adds with NorMuon for a 0.0158-nat reduction, and gives a 0.0188-nat reduction at 355M parameters. Energy-matched Schatten-1 decay adds 0.0052 nats. Exact polar factors, input whitening, and top-heavy decay do not help. The abstract reports these results but does not establish broader applicability beyond the tested settings.

### Source excerpt

arXiv:2611.24644v1 Announce Type: new Abstract: Muon updates each hidden weight matrix along the polar factor of its momentum, the steepest-descent direction under the spectral norm. That choice fixes two allocations without consulting the loss: every singular direction of the momentum receives the same step, and in a non-square matrix each neuron's step is set by its leverage score. Whether they are good is a question about curvature, which steepest-descent arguments do not answer; we measure it. Using held-out gradients and Hessian-vector products, we decompose Muon's step by singular-value band and by neuron group and compute each group's second-order optimal multiplier. In a 124M-parameter decoder trained on FineWeb-Edu, the multipliers rise monotonically from the top singular directions to the tail: Muon over-steps the high-curvature top of its spectrum. We prove in Lean when such a diagnostic licenses an intervention at Muon's own update norm. On the measured curvature, the criterion predicts, and held-out losses confirm, that moving step length off the top helps mid-training and early in the cooldown, but not at its end. We then test interventions under per-step norm matching, tuned baselines and paired seeds. Orthogonalizing Q, K and V separately rather than as one fused matrix lowers the loss by 0.0250 nats and accounts for all of the gain of block-wise variants. A matmul-only, rotation-equivariant spectral tilt $\sigma^{-1/4}$ applied in the cooldown lowers the loss of this stronger baseline by a further 0.0120 nats, additively with NorMuon (0.0158 nats together, three seeds; the baseline needs about 4.3% more tokens to match), and by 0.0188 nats at 355M parameters (three seeds). Energy-matched Schatten-1 weight decay adds a further 0.0052 nats. Geometrically principled changes that do not act on the measured miscalibration, namely exact polar factors, steepest-descent input whitening and top-heavy decay, do not help.

## What Moves the Ceiling on Automated Audit: from General Code to Scientific Code, Results and Data

DevFeed: [What Moves the Ceiling on Automated Audit: from General Code to Scientific Code, Results and Data](<https://devfeed.tech/articles/what-moves-the-ceiling-on-automated-audit-from-general-code-to-scientific-code-results-and-data-142127.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23250>)

Author: Zhaohe Dong

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [audit](<https://devfeed.tech/topics/audit.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [all](<https://devfeed.tech/tags/all.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [audit](<https://devfeed.tech/tags/audit.md>), [automated](<https://devfeed.tech/tags/automated.md>), [behaviour](<https://devfeed.tech/tags/behaviour.md>), [but](<https://devfeed.tech/tags/but.md>), [can](<https://devfeed.tech/tags/can.md>), [code](<https://devfeed.tech/tags/code.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [data](<https://devfeed.tech/tags/data.md>), [decision](<https://devfeed.tech/tags/decision.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [error](<https://devfeed.tech/tags/error.md>), [figure](<https://devfeed.tech/tags/figure.md>), [flag](<https://devfeed.tech/tags/flag.md>), [general](<https://devfeed.tech/tags/general.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [independent](<https://devfeed.tech/tags/independent.md>), [results](<https://devfeed.tech/tags/results.md>)

### AI overview

The study finds that automated audits have different ceilings across general code, scientific code, reports and data. For general-purpose code, eight auditor readings achieved 30.0% recall while flagging 16.0% of correct code, with no plateau; results also shifted with the reading model and sampling, a rulebook sentence, and the decision rule. Many misses were later judged to concern behavior the specification left undefined. In three preregistered science studies, source-tracing checks caught all 49 inconsistent reports and none of 25 consistent fabrications; the model auditor flagged 23 fabrications, 21 with a sound finding. On data, a documentation-derived validator and model auditor caught different faulty items, together flagging 112 of 140 while flagging 2.9% of clean data. Reframing scientific-code auditing around one step reduced flags on correct code from 68.7% to 36.0%. Deterministic checks avoided clean-item flags but could not detect consistent errors; the model auditor caught some such errors but also made mistakes.

### Source excerpt

arXiv:2611.23250v1 Announce Type: new Abstract: Models increasingly check what other models write. We study one shipped arrangement, in which an auditor from a different vendor reads the work and deterministic checks verify what code can verify, and measure its limits against ground truth no model supplies. On general-purpose code with hidden tests, eight readings of the auditor reach 30.0% recall [20.0, 40.7] at 16.0% [10.1, 22.3] flags on correct code, with diminishing returns and no plateau. The reading model (confounded with its sampling), one rulebook sentence and the decision rule each move that figure, and much of what is missed is, on a post hoc rating, behaviour the specification never determined. In three preregistered science studies, checking that reported numbers trace to their sources flagged all 49 inconsistent reports and none of 25 consistent fabrications, of which the model auditor flagged 23 (21 with a sound finding, post hoc); on data, a documentation-derived validator and the auditor flagged overlapping but different faulty items, together 112 of 140 at a 2.9% [0.7, 5.7] flag rate on clean data; and on scientific code, reframing the task around one step cut flags on correct code from 68.7% to 36.0%. Where both tiers ran, the deterministic one flagged no clean item and could not see consistent error; the model saw some and erred itself. Studies making model calls were preregistered, the two unregistered analyses are named, and every quoted report passed independent cross-vendor review.

## What Survival Benchmarks Don't Tell You: Impact of Model Selection and Dataset Regimes

DevFeed: [What Survival Benchmarks Don't Tell You: Impact of Model Selection and Dataset Regimes](<https://devfeed.tech/articles/what-survival-benchmarks-don-t-tell-you-impact-of-model-selection-and-dataset-regimes-143568.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24837>)

Author: Sandra Dening, Emre Kavak, Christian Wachinger, George H. Chen

Published: 2026-11-19T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [bayesian](<https://devfeed.tech/tags/bayesian.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [brier-score](<https://devfeed.tech/tags/brier-score.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [model](<https://devfeed.tech/tags/model.md>), [model-selection](<https://devfeed.tech/tags/model-selection.md>), [performance](<https://devfeed.tech/tags/performance.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [scale](<https://devfeed.tech/tags/scale.md>), [spaces](<https://devfeed.tech/tags/spaces.md>), [survival](<https://devfeed.tech/tags/survival.md>), [utility](<https://devfeed.tech/tags/utility.md>), [variable](<https://devfeed.tech/tags/variable.md>)

### AI overview

A benchmark of 10 classical, tree-based, and deep learning survival methods across 49 datasets finds that evaluation choices and dataset regimes shape performance conclusions. For complex deep learning and tree-based models, tuning for Integrated Brier Score (IBS) improves probabilistic accuracy and calibration over tuning for C-index, with little loss in ranking. Deep learning benefits from larger samples but performs worse in high-dimensional feature spaces. Heavy censoring can artificially inflate absolute calibration metrics and produce overly conservative tests. The authors argue for reporting evaluation setups transparently instead of relying on global single-number summaries.

### Source excerpt

arXiv:2611.24837v1 Announce Type: new Abstract: Despite the proliferation of survival analysis methods, current benchmarks often operate under narrow conditions: they heavily rely on the concordance index (C-index) for evaluation and model selection, and frequently either exclude deep learning (DL) methods or underrepresent challenging dataset regimes. To address these limitations, we conduct a large-scale, neutral benchmark comparing 10 classical, tree-based, and DL methods across 49 diverse datasets. Crucially, we treat the model selection criterion (optimizing for C-index, Integrated Brier Score (IBS), or Mean Absolute Error (MAE-PO)) as an independent experimental variable. Using Bayesian mixed-effects modeling, we demonstrate that evaluation design heavily dictates conclusions. For complex DL and tree-based architectures, selecting hyperparameters to optimize for IBS instead of C-index yields substantial improvements in probabilistic accuracy and calibration at a negligible cost to discriminative ranking. Furthermore, method performance (C-index, IBS, MAE-PO, and X-CAL) is fundamentally governed by dataset characteristics. While DL models scale exceptionally well with larger sample sizes, they suffer notable performance drops in high-dimensional feature spaces. Additionally, we expose a structural vulnerability where heavy censoring artificially inflates absolute calibration metrics, creating overly conservative tests. Ultimately, our findings challenge the utility of global single-number performance summaries and advocate for survival analysis results that transparently report the evaluation setup. Our code is available at: https://github.com/sdening/surv-bench-study

## WorkWorlds: Evaluating Agents in a Company Built Before the Task

DevFeed: [WorkWorlds: Evaluating Agents in a Company Built Before the Task](<https://devfeed.tech/articles/workworlds-evaluating-agents-in-a-company-built-before-the-task-142563.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.23806>)

Author: Yining Hua, Levi Lian

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [environment](<https://devfeed.tech/tags/environment.md>), [environments](<https://devfeed.tech/tags/environments.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [materials](<https://devfeed.tech/tags/materials.md>), [pass](<https://devfeed.tech/tags/pass.md>), [pharmaceutical](<https://devfeed.tech/tags/pharmaceutical.md>), [report](<https://devfeed.tech/tags/report.md>), [research](<https://devfeed.tech/tags/research.md>), [score](<https://devfeed.tech/tags/score.md>), [switching](<https://devfeed.tech/tags/switching.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [test](<https://devfeed.tech/tags/test.md>), [work](<https://devfeed.tech/tags/work.md>), [workplace](<https://devfeed.tech/tags/workplace.md>)

### AI overview

WorkWorlds tests how workplace environment construction affects knowledge-work agent scores. Using historical data from PharmaCo, two experienced experts each created task-specific environments for eight real-event tasks. Across 192 evaluations, giving agents all materials actually visible to the employee, rather than a task-specific environment, reduced evidence access from 90.4% to 72.8% and criterion pass rates from 76.7% to 68.0%. When agents found the required evidence, pass rates were nearly identical across conditions. Changing which expert built the environment shifted average pass rates by 6.6 percentage points, and the three agents benefited by different amounts. The authors recommend reporting how benchmark materials were selected and, where possible, measuring score changes across environment builders.

### Source excerpt

arXiv:2609.23806v3 Announce Type: replace Abstract: Many knowledge work benchmarks are built around single tasks, where the context and files for each task are selected concurrently or after task definition. Research has argued that this construction method does not reflect real workplace work, yet the impact has never been measured under controlled conditions. To compare the two constructions under controlled conditions, we introduce WorkWorlds, an infrastructure that modularizes company components into revision, date, and employee seat to be separately adjusted. Using historical data donated by a real company, we reconstructed a pharmaceutical company, PharmaCo. Two experts with over ten years of industry experience each wrote eight tasks based on real events that occurred at the company, and each independently constructed a task-specific environment for every task. Across 192 controlled evaluations, when the agent was switched from the task-specific environment to all the materials truly visible to the employee, evidence access dropped from 90.4 percent to 72.8 percent, and the criterion pass rate dropped from 76.7 percent to 68.0 percent. However, in cases where the agent found the required evidence, the pass rates under both conditions were almost identical. Furthermore, switching between the environments built by the two experts for the same task resulted in an average pass rate difference of 6.6 percentage points. The three agents also gained by different amounts, which changed the gaps between them. We argue that benchmarks should report how environment materials were selected and, if possible, they should test agent score changes after changing the environment builder.

## Same Model, Different People: Numerical Execution Contracts and the Reproducibility of Fairness Audits

DevFeed: [Same Model, Different People: Numerical Execution Contracts and the Reproducibility of Fairness Audits](<https://devfeed.tech/articles/same-model-different-people-numerical-execution-contracts-and-the-reproducibility-of-fairness-audits-143426.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23657>)

Author: Erzheng Zhang, Agnes Chu

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [ML model versioning](<https://devfeed.tech/topics/ml-model-versioning.md>)

Tags: [audits](<https://devfeed.tech/tags/audits.md>), [contracts](<https://devfeed.tech/tags/contracts.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [execution](<https://devfeed.tech/tags/execution.md>), [fairness](<https://devfeed.tech/tags/fairness.md>), [model](<https://devfeed.tech/tags/model.md>), [reproducibility](<https://devfeed.tech/tags/reproducibility.md>)

### AI overview

Fairness audit results can depend on the numerical execution configuration, even when the model, records, threshold and audit code are fixed. With byte-identical weights, changing arithmetic altered some near-threshold decisions, but no prespecified unweighted verdict; reduced-precision deployment configurations changed more decisions and some diagnostic verdicts, but no four-fifths verdict. In one example, vLLM FP8 serving of a 24B model selected 54% of people versus 45% for strict FP32 and passed an employment-status audit by sex that FP32 failed. At FP32's selection count, FP8 failed demographic parity, and the equalized-odds change had no paired interval excluding zero; the result does not establish that FP8 is fairer. Every changed pass/fail verdict had a sampling interval containing its criterion. The proposed stability protocol declares execution contracts, groups, weights and tests before scoring, then reports verdict variation separately from sampling uncertainty.

### Source excerpt

arXiv:2611.23657v1 Announce Type: cross Abstract: Fairness audits treat a trained model as a fixed decision function, but the arithmetic that evaluates it is part of that function. This study holds the model, records, threshold and audit code fixed and varies the numerical execution contract in two tiers. Tier 1 changes only how products and sums are computed, with the serialized weights byte-identical, and is the causal test of arithmetic. Tier 2 runs reduced-precision deployment configurations (BF16 compute, FP8 and 8-bit weights), which can also round weights and change engines and kernels, so its effects belong to the configuration. It audits three tabular model families and two open-weight language models (4B and 24B parameters) on four public tasks, including hospital readmission, on CPUs and GPUs from NVIDIA, AMD and Apple. Tier 1 changed decisions near the threshold, and which ones depended on the GPU and its software stack, but no verdict under the prespecified, unweighted tests. Tier 2 configurations changed more decisions and some diagnostic verdicts, but no four-fifths verdict. Serving the 24B model with vLLM FP8 selected 54% of people rather than 45% and passed an employment-status audit by sex that its strict FP32 reference failed: its demographic-parity and equalized-odds gaps fell from 0.0567 and 0.0515 to 0.0417 and 0.0394, against a diagnostic tolerance of 0.05. At the reference's selection count FP8 failed demographic parity, and no reading gave its equalized-odds change a paired interval excluding zero. The declared audit depends on the configuration; FP8 is not shown to be fairer. At the frozen thresholds, every pass-fail verdict a configuration changed had a sampling interval containing its criterion. It proposes an audit stability protocol in which the auditor declares contracts, groups, weights and tests before scoring and records each verdict's range over them apart from its sampling uncertainty.

## Training for sustainable scientific software in the AI era: from code production to scientific stewardship

DevFeed: [Training for sustainable scientific software in the AI era: from code production to scientific stewardship](<https://devfeed.tech/articles/training-for-sustainable-scientific-software-in-the-ai-era-from-code-production-to-scientific-stewardship-142177.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23330>)

Author: Juliette Alimena

Published: 2026-11-19T05:00:00Z

Content type: opinion

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [Sustainable Computing](<https://devfeed.tech/topics/sustainable-computing.md>)

Tags: [agentic-artificial-intelligence](<https://devfeed.tech/tags/agentic-artificial-intelligence.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-assisted](<https://devfeed.tech/tags/ai-assisted.md>), [ai-assisted-workflows](<https://devfeed.tech/tags/ai-assisted-workflows.md>), [ai-systems](<https://devfeed.tech/tags/ai-systems.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [code](<https://devfeed.tech/tags/code.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [hep-ex](<https://devfeed.tech/tags/hep-ex.md>), [physics](<https://devfeed.tech/tags/physics.md>), [privileged-access](<https://devfeed.tech/tags/privileged-access.md>), [productivity](<https://devfeed.tech/tags/productivity.md>), [provenance](<https://devfeed.tech/tags/provenance.md>), [reproducibility](<https://devfeed.tech/tags/reproducibility.md>), [resource](<https://devfeed.tech/tags/resource.md>), [resources](<https://devfeed.tech/tags/resources.md>), [software](<https://devfeed.tech/tags/software.md>), [training](<https://devfeed.tech/tags/training.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

### AI overview

For high-energy physics, AI-assisted software training should prioritize scientific stewardship alongside code production. Because AI-generated or modified workflows can weaken researchers' command of the choices and evidence behind a result, training should combine dedicated AI instruction with established software and analysis training. Researchers should practice testing, tracking provenance, reproducing and maintaining workflows, judging resource use, and validating results. Competence should be assessed by the evidence supporting scientific results, not code generation alone. The abstract frames sustainability as keeping workflows trustworthy and maintainable, using resources responsibly, and avoiding dependence on privileged access to particular tools or infrastructure.

### Source excerpt

arXiv:2611.23330v1 Announce Type: new Abstract: Generative and agentic artificial intelligence (AI) systems are changing how researchers interact with scientific software. Their effects on productivity vary across tasks and users, but they can weaken the connection between producing a successful computational artifact and retaining scientific command over the consequential choices and evidence on which it depends. For high-energy physics, this shifts part of the training challenge from code production toward scientific stewardship: researchers must be able to validate, explain, reproduce, maintain, and justify software and workflows partly generated or modified by AI. We argue that AI-aware software training should therefore combine dedicated instruction on AI systems with the integration of AI into established software and analysis training. Testing, provenance, reproducibility, resource judgment, and scientific validation should be exercised in AI-assisted workflows, with competence judged by the evidence supporting a result rather than by code generation alone. We frame this as a problem of sustainable scientific software: AI-assisted workflows should remain trustworthy and maintainable, use computational resources responsibly, and avoid making core competence unnecessarily dependent on privileged access to particular tools or infrastructure.

## The Obsolescence of High-Level Code: Agentic Generation from Semantic Artifacts

DevFeed: [The Obsolescence of High-Level Code: Agentic Generation from Semantic Artifacts](<https://devfeed.tech/articles/the-obsolescence-of-high-level-code-agentic-generation-from-semantic-artifacts-142126.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23249>)

Author: Di Santi Eduardo

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Spec Driven Development](<https://devfeed.tech/topics/spec-driven-development.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [Model driven development](<https://devfeed.tech/topics/model-driven-development.md>), [LLVM](<https://devfeed.tech/topics/llvm.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [application](<https://devfeed.tech/tags/application.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [case](<https://devfeed.tech/tags/case.md>), [code](<https://devfeed.tech/tags/code.md>), [coherence](<https://devfeed.tech/tags/coherence.md>), [contracts](<https://devfeed.tech/tags/contracts.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-pl](<https://devfeed.tech/tags/cs-pl.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [developers](<https://devfeed.tech/tags/developers.md>), [development](<https://devfeed.tech/tags/development.md>), [domain](<https://devfeed.tech/tags/domain.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [external](<https://devfeed.tech/tags/external.md>), [generation](<https://devfeed.tech/tags/generation.md>), [semantic](<https://devfeed.tech/tags/semantic.md>)

### AI overview

The paper argues that software teams could maintain semantic specifications as the primary authority and regenerate executable implementations as needed, instead of treating high-level source code as the main persistent artifact. Its framework separates semantic obligations--requirements, contracts, and interfaces--from tests and operational goals, which are evaluated by an external oracle. In an experiment, multiple LLVM artifacts generated independently from one frozen specification were assessed by the same oracle; the reported result is coherence across observable outcomes within the evaluated domain, not proof of absolute correctness. The abstract situates this proposal alongside earlier application generators, CASE tools, model-driven development, and GeneXus, and points to reproducibility and specification-centered evolution as implications.

### Source excerpt

arXiv:2611.23249v1 Announce Type: new Abstract: This paper argues that high-level source code need not remain the primary persistent artifact of software development when AI agents can generate, validate, and regenerate executable implementations from semantic intent. We place this claim in historical context by revisiting application generators, CASE tools, model-driven development, and platforms such as GeneXus, where developers already maintained higher-level representations from which code and other artifacts were derived. We propose a unifying view in which generation is the fundamental operation: given a semantic specification S = (R, C, I, T, O), maintained as the semantic authority, an agent produces a generated executable artifact A. Functional requirements, contracts, and interfaces are assessed as semantic obligations, while tests and operational objectives are assessed by an explicit external oracle V(A, S). We operationalize this framework through an experiment in independent generation: multiple LLVM artifacts are generated directly from one frozen semantic authority and evaluated by the same external oracle. The experiment reports regeneration coherence on an evaluated domain, not a proof of absolute correctness. It provides initial evidence that independently generated executable artifacts can produce the same observable outcomes under a common semantic authority and external oracle. We conclude by discussing reproducibility, specification-centered evolution, and the implications of treating implementation representation as a deployment choice rather than the primary maintained artifact.

## ProvCards: Artifact-Oriented Provenance for Heterogeneous Scientific Workflows

DevFeed: [ProvCards: Artifact-Oriented Provenance for Heterogeneous Scientific Workflows](<https://devfeed.tech/articles/provcards-artifact-oriented-provenance-for-heterogeneous-scientific-workflows-142291.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24894>)

Author: Tanja Auge, Marinus Holitschke, Meike Klettke

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [RDF (Resource Description Framework)](<https://devfeed.tech/topics/rdf.md>)

Tags: [cs-db](<https://devfeed.tech/tags/cs-db.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [provenance](<https://devfeed.tech/tags/provenance.md>), [querying](<https://devfeed.tech/tags/querying.md>), [reproducibility](<https://devfeed.tech/tags/reproducibility.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

### AI overview

ProvCards captures workflow provenance as portable artifacts grounded in the W3C PROV Data Model. Components can independently create, exchange, reuse, and compose cards without relying on a particular workflow system or centralized execution context. ProvDecks group related cards while preserving their identities; derived ProvGraphs support reconstruction, querying, visualization, and analysis. A graph-based implementation evaluated in a scientific data-processing workflow demonstrated component-level capture, cross-component composition, and reconstruction. The evidence is limited to the abstract and does not specify comparative performance or broader evaluation results.

### Source excerpt

arXiv:2611.24894v1 Announce Type: new Abstract: Reproducibility in scientific workflows is hindered by heterogeneous components and execution environments, as well as fragmented provenance capture. Although existing approaches offer established semantics and workflow-specific solutions, provenance remains difficult to capture, exchange, and compose across workflow boundaries. We present ProvCards, an artifact-oriented approach that treats provenance as a first-class artifact grounded in the W3C PROV Data Model. Individual workflow components can generate cards that are stored, exchanged, reused, and composed independently of any workflow system or centralized execution context. ProvDecks group related cards into portable scopes while preserving their identities. ProvGraphs provide derived views for provenance reconstruction, querying, visualization, and analysis. Together, these representations support coherent, task-specific provenance views across heterogeneous components, facilitating traceability, reuse, and reproducibility. We implement ProvCards using a graph-based infrastructure and evaluate them in a scientific data-processing workflow. The results demonstrate independent provenance capture at the component level, composition across component boundaries, and reconstruction for querying and analysis. ProvCards provide a portable, composable, and incrementally extensible foundation for provenance management in heterogeneous scientific workflows.

## Integrating AI Coding Agents Into Scientific Research

DevFeed: [Integrating AI Coding Agents Into Scientific Research](<https://devfeed.tech/articles/integrating-ai-coding-agents-into-scientific-research-142179.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23334>)

Author: Rory Barnes

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [version-control](<https://devfeed.tech/topics/version-control.md>), [trust](<https://devfeed.tech/topics/trust.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [3](<https://devfeed.tech/tags/3.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [accessible](<https://devfeed.tech/tags/accessible.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-development](<https://devfeed.tech/tags/agentic-development.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-assisted](<https://devfeed.tech/tags/ai-assisted.md>), [ai-coding](<https://devfeed.tech/tags/ai-coding.md>), [ai-coding-agents](<https://devfeed.tech/tags/ai-coding-agents.md>), [also](<https://devfeed.tech/tags/also.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [archive](<https://devfeed.tech/tags/archive.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [astro-ph-im](<https://devfeed.tech/tags/astro-ph-im.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [research](<https://devfeed.tech/tags/research.md>), [scientific-research](<https://devfeed.tech/tags/scientific-research.md>)

### AI overview

AI coding agents can speed up scientific analysis and figure generation, but errors may damage a researcher's computer, increase validation effort, and weaken confidence in results. The paper proposes the six-level PROOF scale, which classifies AI-assisted numerical experiments by provenance, reproducibility, openness, oversight, and falsifiability. Its levels progress from locally self-consistent, version-controlled, human-verified results to a curated public archive with files, change history, prompt transcripts, independent agent regeneration, and cryptographically verifiable attestations. The open-source vaibify package supports this workflow: Docker containers isolate agents, while a GUI lets researchers prompt command-line agents, monitor local and remote file changes, and work toward PROOF Level 3, described as reproducible. The authors suggest these methods may reduce verification time and help communicate trust in AI-assisted science.

### Source excerpt

arXiv:2611.23334v1 Announce Type: new Abstract: New software tools have emerged that enable AI agents to quickly develop code that analyzes data and generates figures, but their mistakes can also harm a researcher's computer, require extensive resources for validation, and cause the results to be received more skeptically. These concerns could be mitigated if scientific software is developed in a secure environment with a system for monitoring file changes and well-defined standards for handling AI agents. This paper first describes how coding agents change eleven aspects of generating, verifying, and reproducing numerical results. A six-level scale is then proposed for classifying AI-assisted numerical experiments by their provenance, reproducibility, openness, oversight, and falsifiability (PROOF). The scale begins with locally self-consistent, version-controlled, human-verified results and culminates in a publicly accessible and curated archive of the files, change history, prompt transcripts, independent agent regeneration of the analysis, and multiple cryptographically verifiable attestations of regeneration by independent researchers. These aspects of AI-assisted scientific software development have been synthesized into a new open source software package called vaibify that optimizes the agentic development of computational experiments ascending the PROOF Ladder. Docker containers sequester the agents while a GUI enables researchers to prompt command line agents, monitor the current status of local and remote files, and drive a project to PROOF Level 3 (reproducible). The methods described in this paper could help researchers reduce the time required to verify agent-written software and efficiently communicate trust in AI-assisted science.

## MANGO Arena: Same Seeds, Different Chains Evaluating LLM Agents in a Disrupted Supply Chain

DevFeed: [MANGO Arena: Same Seeds, Different Chains Evaluating LLM Agents in a Disrupted Supply Chain](<https://devfeed.tech/articles/mango-arena-same-seeds-different-chains-evaluating-llm-agents-in-a-disrupted-supply-chain-142680.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23613>)

Author: Sara Amar

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [api](<https://devfeed.tech/tags/api.md>), [arena](<https://devfeed.tech/tags/arena.md>), [batches](<https://devfeed.tech/tags/batches.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [constraint](<https://devfeed.tech/tags/constraint.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ma](<https://devfeed.tech/tags/cs-ma.md>), [deal](<https://devfeed.tech/tags/deal.md>), [demand](<https://devfeed.tech/tags/demand.md>), [disrupted](<https://devfeed.tech/tags/disrupted.md>), [disruption](<https://devfeed.tech/tags/disruption.md>), [environment](<https://devfeed.tech/tags/environment.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [factory](<https://devfeed.tech/tags/factory.md>), [failed](<https://devfeed.tech/tags/failed.md>), [failures](<https://devfeed.tech/tags/failures.md>), [fire](<https://devfeed.tech/tags/fire.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [supply-chain](<https://devfeed.tech/tags/supply-chain.md>)

### AI overview

MANGO Arena evaluates four LLM agents managing a serial supply chain under truck, cash and production constraints, while tracking bullwhip, rule adherence and collusion proposals. In a pilot with three small reasoning models, all amplified demand variability upstream: 7-25x at the Retailer and 13-179x at the Supplier, often through batch ordering. Rule violations were rare and no agent proposed a money side deal. The results were highly sensitive to evaluation reliability: API failures silently counted as no orders, and upstream bullwhip varied by up to 17x across reruns with the same seeds and no failed calls. With only three seeds, conclusions about disruption effects did not hold on rerun. The authors report these failures and propose practices for multi-agent evaluations.

### Source excerpt

arXiv:2611.23613v1 Announce Type: cross Abstract: We introduce MANGO Arena, an open-source testbed in which four LLM agents each run one company in a serial supply chain adapted from MANGO, a classroom factory-management game. The environment enforces hard rules (a truck limit, a no-credit cash constraint and production capacity), logs every blocked attempt to break them, and gives agents a public message board on which side payments are explicitly forbidden. This lets one environment measure three things at once: an economic outcome (the bullwhip effect), rule adherence under pressure, and proposals to collude. In a pilot with three small reasoning models, two scripted reference policies, two conditions (with and without a supplier fire) and three demand seeds, all LLM agents amplified demand variability as it moved upstream, by 7 to 25 times at the Retailer and 13 to 179 times at the Supplier, often more than a textbook order-up-to policy. They did so largely by ordering in batches (zero orders in up to 60% of decisions). Real rule violations were rare (at most 7 of 288 decisions per model) and no agent proposed a money side deal. The pilot's main lesson concerns evaluation method. API failures (rate limits and runaway hidden reasoning) entered the metrics silently as "no order" decisions. And when we reran games with the same seeds and no failed calls, upstream bullwhip changed by up to 17 times between runs. With three seeds, conclusions about how a disruption affects a given model did not survive a rerun. We report these failures in full and propose reporting practices for multi-agent evaluations of this kind. Code, prompts and every logged decision are public.

## A Grey Literature Review of Graduate Students' Research Practices and Skills in the LLM Era

DevFeed: [A Grey Literature Review of Graduate Students' Research Practices and Skills in the LLM Era](<https://devfeed.tech/articles/a-grey-literature-review-of-graduate-students-research-practices-and-skills-in-the-llm-era-139567.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.29519>)

Author: Enne Rebeca Silva de Freitas, Gustavo Pinto, Danilo Monteiro

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Programming](<https://devfeed.tech/topics/programming.md>), [human review](<https://devfeed.tech/topics/human-review.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [coding](<https://devfeed.tech/tags/coding.md>), [critical-thinking](<https://devfeed.tech/tags/critical-thinking.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [debugging](<https://devfeed.tech/tags/debugging.md>), [graduate-students](<https://devfeed.tech/tags/graduate-students.md>), [inspection](<https://devfeed.tech/tags/inspection.md>), [literature](<https://devfeed.tech/tags/literature.md>), [llm](<https://devfeed.tech/tags/llm.md>), [practices](<https://devfeed.tech/tags/practices.md>), [reddit](<https://devfeed.tech/tags/reddit.md>), [reproducibility](<https://devfeed.tech/tags/reproducibility.md>), [research](<https://devfeed.tech/tags/research.md>), [review](<https://devfeed.tech/tags/review.md>), [sharing](<https://devfeed.tech/tags/sharing.md>), [skills](<https://devfeed.tech/tags/skills.md>), [validating](<https://devfeed.tech/tags/validating.md>), [verify](<https://devfeed.tech/tags/verify.md>), [writing](<https://devfeed.tech/tags/writing.md>)

### AI overview

The review identifies a risk in graduate students' LLM use: accepting generated output without inspection, or delegating effortful work such as debugging, reading, and source validation, can leave research skills undeveloped. The authors analyzed 54 Reddit posts selected from 1,383 posts across five research-oriented subreddits, identifying seven categories of practices and nine of skills. They argue that programming and critical thinking help students verify LLM output, and that graduate training should support LLM use while preserving skill development. The proposed categories also add research-specific skills, including data sharing and scientific reproducibility, to AI competency frameworks. Findings are based on a grey literature review of Reddit discussions.

### Source excerpt

arXiv:2607.29519v2 Announce Type: replace Abstract: Context: Graduate students in computing now use Large Language Models (LLMs) throughout their research, from literature reviews and coding to data analysis and writing. Yet there is little shared understanding of which skills competent use of these tools requires, so graduate programs struggle to teach, mentor, or assess it, and unverified outputs can propagate into the scientific record. Objective: We investigate which LLM-based practices graduate students employ in their research and which skills they need to benefit from these practices. Method: We conducted a grey literature review of Reddit discussions. We retrieved 1,383 posts from five research-oriented subreddits, screened them against explicit inclusion and exclusion criteria, and analyzed the 54 included posts with thematic analysis, in which two authors independently coded each excerpt. Results: We identified seven categories of practices and nine categories of skills. Commenters locate the problem in accepting generated output without inspection rather than in using LLMs, and warn that delegating the effortful parts of a task, such as debugging, reading, or validating sources, delivers the output without building the skill the task was meant to develop. Traditional skills, such as programming and critical thinking, are what allow students to verify what an LLM produces. Conclusions: Our categories give graduate programs a concrete basis to teach students to work with LLMs without eroding the skills that research training should build, and they add research-specific skills, such as data sharing and scientific reproducibility, to existing AI competency frameworks.

## SOTA or Luck? The Winner's Curse in LLM Leaderboards

DevFeed: [SOTA or Luck? The Winner's Curse in LLM Leaderboards](<https://devfeed.tech/articles/sota-or-luck-the-winner-s-curse-in-llm-leaderboards-141751.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22227>)

Author: Sikun Xu, Raphael Thomadsen, Dennis J. Zhang, Heng Zhang

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [benchmark-design](<https://devfeed.tech/tags/benchmark-design.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bias](<https://devfeed.tech/tags/bias.md>), [bootstrap](<https://devfeed.tech/tags/bootstrap.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [leaderboards](<https://devfeed.tech/tags/leaderboards.md>), [llm](<https://devfeed.tech/tags/llm.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [ruler](<https://devfeed.tech/tags/ruler.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [swe-bench](<https://devfeed.tech/tags/swe-bench.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>)

### AI overview

Noisy benchmark scores can make top-ranked LLMs look better than their underlying performance: selecting winners rewards sampling luck, creating rank-induced bias across the leaderboard. The paper proposes an m-out-of-N bootstrap to adjust scores, estimate plausible rank intervals, and group models into tiers. It assumes a fixed model pool and independent, identically distributed bounded tasks, while allowing arbitrary within-task correlation among models. Across 14 benchmarks in five families, rank-one adjustments range from about 0.09 to 3.86 percentage points; tied leaders in the reported SWE-bench Verified snapshot receive a 0.90-point adjustment, and 13 of 173 models are placed in the leading tier. The authors combine adjustments and rank intervals in RULER, validate finite-sample performance through sample splitting and calibrated simulations, and recommend publishing it alongside raw rankings. Their bias reduction result requires exact population tiers separated by positive gaps.

### Source excerpt

arXiv:2611.22227v1 Announce Type: new Abstract: Public LLM leaderboards guide model evaluation and adoption, yet the reported performance of top-ranked models can be systematically overstated. This leaderboard winner's curse arises because ranking noisy benchmark scores rewards sampling luck alongside genuine ability. The same selection effect can distort scores throughout the leaderboard, a phenomenon we call the rank-induced bias. We develop an $m$-out-of-$N$ bootstrap to correct for this bias. It assumes a fixed model pool with independent, identically distributed bounded tasks while allowing arbitrary within-task correlation among models. Under exact population tiers separated by positive gaps, we prove that the bootstrap reduces $\Theta(N^{-1/2})$ ranking bias to expected residual bias $o(N^{-1/2})$. Across 14 benchmarks from 5 benchmark families, the rank-1 adjustments range from approximately 0.09 to 3.86 percentage points. In our SWE-bench Verified snapshot, the tied leaders receive an adjustment of 0.90 percentage points. Such overstatement is large enough to make the apparent winner statistically fragile. Thus, we use the same bootstrap to further quantify the intervals of plausible rank positions for each model and group models into tiers. We establish sufficient conditions for consistent recovery of population tiers. On SWE-bench Verified, this procedure places 13 of 173 models in the leading tier. We combine the score adjustment and rank intervals in the Rank Uncertainty and Leaderboard Evaluation Report (RULER). We validate its finite-sample performance using sample splitting and calibrated simulations, and advocate publishing RULER alongside raw rankings so readers can assess both reported performance and the precision of the ordering. RULER also provides guidance for benchmark design that accounts for selection effects and ranking uncertainty.

## X-LocVul: An Evidence-Propagation Audit of Cascaded C/C++ Vulnerability Detection, Localization, and Explanation

DevFeed: [X-LocVul: An Evidence-Propagation Audit of Cascaded C/C++ Vulnerability Detection, Localization, and Explanation](<https://devfeed.tech/articles/x-locvul-an-evidence-propagation-audit-of-cascaded-c-c-vulnerability-detection-localization-and-explanation-139585.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21626>)

Author: Hoang-Phuong-Linh Le, Quyet-Thang Huynh

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [audit](<https://devfeed.tech/topics/audit.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [account](<https://devfeed.tech/tags/account.md>), [after](<https://devfeed.tech/tags/after.md>), [all](<https://devfeed.tech/tags/all.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [audit](<https://devfeed.tech/tags/audit.md>), [but](<https://devfeed.tech/tags/but.md>), [c](<https://devfeed.tech/tags/c.md>), [c-c-plus-plus](<https://devfeed.tech/tags/c-c-plus-plus.md>), [c-plus-plus](<https://devfeed.tech/tags/c-plus-plus.md>), [can](<https://devfeed.tech/tags/can.md>), [cases](<https://devfeed.tech/tags/cases.md>), [cause](<https://devfeed.tech/tags/cause.md>), [class](<https://devfeed.tech/tags/class.md>), [components](<https://devfeed.tech/tags/components.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [detection](<https://devfeed.tech/tags/detection.md>), [localization](<https://devfeed.tech/tags/localization.md>), [vulnerability](<https://devfeed.tech/tags/vulnerability.md>), [vulnerability-detection](<https://devfeed.tech/tags/vulnerability-detection.md>)

### AI overview

X-LocVul audits how evidence moves through a C/C++ vulnerability pipeline combining UniXCoder, CodeT5, and Qwen2.5-Coder. Under duplication and temporal-leakage controls, true-CWE supervision was statistically indistinguishable from shuffled-CWE supervision on PR-AUC, MCC, and vulnerable-class F1. Generated statements failed exact source matching in 90.83% of cases; fuzzy projection achieved 83.79% Top-1 accuracy, 1.59 points above semantic projection, though its advantage over raw ranking was not established. In a masked two-rater study of 50 paired cases, predicted lines improved evidence support, while predicted CWEs improved no endpoint. Oracle evidence improved evidence support and CWE consistency, but not corrected root-cause quality; unsupported claims remained common. Chronological testing retained ranking signal (ROC-AUC 0.782; PR-AUC 0.110), but a transferred threshold produced zero recall and F1. Public E4 scores regenerate all 15 tests, while missing manifests, final prompts, and profiler logs limit stronger reproducibility claims.

### Source excerpt

arXiv:2611.21626v1 Announce Type: new Abstract: Developer-facing vulnerability analysis should link a function-level warning to a source location and a supported explanation. Cascaded systems can propagate weak evidence despite plausible components. We present X-LocVul, a measurement-first audit of a modular C/C++ detector-localizer-explainer pipeline combining UniXCoder, CodeT5, and Qwen2.5-Coder. We test whether evidence passed across stages is useful, merely source-resident, or misleading. Under duplication and temporal-leakage controls, UniXCoder has higher point estimates than CodeBERT, but true-CWE supervision is statistically indistinguishable from a shuffled-CWE control on PR-AUC, MCC, and vulnerable-class F1. Raw generated statements fail exact source matching in 90.83% of cases. Fuzzy projection reaches 83.79% Top-1 accuracy and exceeds semantic projection by 1.59 percentage points, although its advantage over raw ranking is not established. In a randomized, condition-label-masked two-rater evaluation of 50 paired cases, a predicted line improves evidence support, whereas a predicted CWE improves no endpoint. Oracle evidence improves evidence support (Holm-adjusted p=0.00453) and CWE consistency (p=3.49 x 10^-7), but not root-cause quality after correction (p=0.250); unsupported claims remain common. A chronological stress test retains ranking signal (ROC-AUC 0.782; PR-AUC 0.110), while the transferred threshold yields zero recall and F1. The contribution is not a state-of-the-art claim, but an auditable account of evidence construction, transfer, and consumption. It separates source residency from localization correctness, auxiliary labels from semantic supervision, and predicted evidence from useful evidence. Public E4 scores regenerate all 15 tests; missing manifests, final prompts, and profiler logs bound stronger reproducibility claims.

## Context Has a Price: Adding and Removing Context for LLM Coding Agents under Prefix Caching

DevFeed: [Context Has a Price: Adding and Removing Context for LLM Coding Agents under Prefix Caching](<https://devfeed.tech/articles/context-has-a-price-adding-and-removing-context-for-llm-coding-agents-under-prefix-caching-139987.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22238>)

Author: Zerui Yang, Miaomiao Dong, Quan Xue, Chaorui Zhang, Xuanyu Zheng, Xiyuan Wang, Haolin Ruan, Xueyan Niu, Guangcheng Song, Weixi Zhang, Wei Han, Bo Bai

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [caching](<https://devfeed.tech/tags/caching.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [context](<https://devfeed.tech/tags/context.md>), [context-management](<https://devfeed.tech/tags/context-management.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [llm](<https://devfeed.tech/tags/llm.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [prefix-caching](<https://devfeed.tech/tags/prefix-caching.md>), [prompts](<https://devfeed.tech/tags/prompts.md>), [swe-bench](<https://devfeed.tech/tags/swe-bench.md>), [token](<https://devfeed.tech/tags/token.md>), [tokens](<https://devfeed.tech/tags/tokens.md>)

### AI overview

Across more than 20,000 runs of open models on SWE-bench Verified, SWE-bench and Loc-Bench, the study finds that context changes can alter coding-agent costs in opposite ways depending on prefix-cache pricing and subsequent behavior. In the main evidence-package experiments, retrieved code cut read and search commands by 18-34% but raised cost by 33-42% when cached input cost one tenth as much as uncached input, with no measurable resolution improvement. Removing old observations saved 19-36% without a cache discount but raised cost by 38-116% under the tenth-price cache. Shorter prompts did not guarantee cheaper runs because agents sometimes took more steps. Concurrent experiments could share cached prefixes and distort comparisons. The authors therefore evaluate context-management methods by accounting separately for uncached input, cached input and output tokens, and by measuring agents' resulting behavior.

### Source excerpt

arXiv:2611.22238v1 Announce Type: new Abstract: Coding agents repeatedly send their conversation history to a language model. Adding useful code may reduce search, while removing old tool outputs may shorten later prompts. But do these changes reduce the cost of solving a task? We study this as a measurement problem in more than 20,000 runs of open models on SWE-bench Verified, SWE-bench and Loc-Bench. We account separately for uncached input, cached input and output tokens. Three findings emerge. First, in our main evidence-package experiments, retrieved code reduces read and search commands by 18--34%, but increases cost by 33--42% when cached input costs one tenth of uncached input, without a measurable improvement in resolution. Second, removing old observations can either save money or increase cost: the same approach saves 19--36% without a cache discount but increases cost by 38--116% with a tenth-price cache. Third, shorter prompts do not guarantee cheaper runs because agents may respond by taking more steps. We also find that concurrent experiments can share cached prefixes and distort cost comparisons. These results show why context-management methods should be evaluated using both token prices and the agent's subsequent behavior.

## Can an LLM dream? Testing the overfitted brain hypothesis on an agent with text memory

DevFeed: [Can an LLM dream? Testing the overfitted brain hypothesis on an agent with text memory](<https://devfeed.tech/articles/can-an-llm-dream-testing-the-overfitted-brain-hypothesis-on-an-agent-with-text-memory-139887.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21648>)

Author: Eric Zou

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [brain](<https://devfeed.tech/tags/brain.md>), [claude-opus-5-5](<https://devfeed.tech/tags/claude-opus-5-5.md>), [corruption](<https://devfeed.tech/tags/corruption.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [llm](<https://devfeed.tech/tags/llm.md>), [memory](<https://devfeed.tech/tags/memory.md>), [recall](<https://devfeed.tech/tags/recall.md>), [text](<https://devfeed.tech/tags/text.md>)

### AI overview

Across 30 runs of a ten-day conversation-memory task, nightly corruption did not improve the agent's use of earlier learning on new tasks. Detail corruption affecting 40% of turns reduced recall from 76% to 60%; when wrong details conflicted with correct ones, the agent often stored the errors as facts and answered related questions correctly 40% of the time, versus 75% without corruption. Random Moby-Dick text had no consistent recall effect at 40% corruption (paired difference 0 ± 2 points), as the model recognized and excluded it from memory. The authors suggest this pretrained model's ability to distinguish corrupted text limits the test, and propose testing a learner fine-tuned on corrupted days or an agent in an invented world.

### Source excerpt

arXiv:2611.21648v1 Announce Type: cross Abstract: Hoel (2021) proposed that dreams distort replayed experience so that the brain does not overfit to recent events. We test this hypothesis on an LLM agent (Claude Opus 5.5) that stores what it learns in a text memory and rewrites that memory each night from the day's conversation transcript. The agent goes through ten simulated days of conversation built from LongMemEval. We corrupt 5, 10, 20 or 40 percent of the transcript's turns before each nightly rewrite in one of two ways. Detail corruption changes one or two details in a turn to plausible wrong ones, or replaces the turn with one by the same speaker from an earlier day. Random-text corruption replaces half of the selected turns with a passage of Moby-Dick and inserts Moby-Dick words into the other half. A size limit on the memory, rising from 19,500 to 117,000 tokens over ten nights, set from human free-recall data, forces it to forget. We learned that corruption did not help the agent apply what it learned in earlier conversations to new tasks in any of 30 runs (ten conditions, three different conversation histories). Replacing details in 40 percent of turns lowered recall from 76 to 60 percent. Random text had no consistent effect on recall, with a paired difference of 0 $\pm$ 2 points at 40 percent, because the model recognized it as corrupted and left it out of memory. Replaced details were stored as fact even when another turn held the correct value, and the agent answered 40 percent of those questions correctly, compared with 75 percent without corruption. These results suggest that a pretrained model can separate corrupted text from real events, so a fair test needs a learner that cannot make this separation as easily, such as a model fine-tuned on the corrupted days or an agent whose days take place in an invented world.

## SLICES: A Benchmark and Evaluation Framework for Self-Supervised Learning on Sparse ICU Time Series

DevFeed: [SLICES: A Benchmark and Evaluation Framework for Self-Supervised Learning on Sparse ICU Time Series](<https://devfeed.tech/articles/slices-a-benchmark-and-evaluation-framework-for-self-supervised-learning-on-sparse-icu-time-series-138799.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18868>)

Author: Hannes Ill, Dimitrios Proios, Leo Anthony Celi, Peter J. Sch\\"uffler

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Self-supervised learning](<https://devfeed.tech/topics/self-supervised-learning.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Time Series](<https://devfeed.tech/topics/time-series.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [classification](<https://devfeed.tech/tags/classification.md>), [code](<https://devfeed.tech/tags/code.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [contrastive-learning](<https://devfeed.tech/tags/contrastive-learning.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [error](<https://devfeed.tech/tags/error.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [framework](<https://devfeed.tech/tags/framework.md>), [hospital](<https://devfeed.tech/tags/hospital.md>), [metric](<https://devfeed.tech/tags/metric.md>), [objectives](<https://devfeed.tech/tags/objectives.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [self-supervised-learning](<https://devfeed.tech/tags/self-supervised-learning.md>), [slices](<https://devfeed.tech/tags/slices.md>), [time-series](<https://devfeed.tech/tags/time-series.md>)

### AI overview

SLICES standardizes comparisons of self-supervised learning for sparse ICU time series by fixing 24-hour value and observation-mask inputs, a Transformer encoder, patient-level splits, four endpoints, two adaptation protocols, and five matched seeds across MIMIC-IV, eICU-CRD, and their pooled union. Of the tested SSL objectives, masked autoencoding (MAE) leads mean AUPRC in all nine frozen classification comparisons and seven of nine fine-tuning comparisons. Fine-tuned MAE leads hospital mortality at full labels and beats a same-architecture supervised Transformer at every label fraction; SSL fine-tuning also has the lowest length-of-stay error. Supervised baselines remain strongest for 24-hour mortality (GRU-D) and acute kidney injury (XGBoost). Rankings vary with adaptation protocol, comparator set, and probe-loss alignment. A two-seed MIMIC-IV rerun stays within two across-seed standard deviations in all 48 downstream runs. Generalization to unseen hospitals was not tested.

### Source excerpt

arXiv:2611.18868v1 Announce Type: new Abstract: Self-supervised learning (SSL) results for intensive care unit (ICU) time series are hard to compare when data processing, encoders, and adaptation vary together. We present SLICES, a benchmark and evaluation framework for comparing and extending SSL methods under a shared clinical setup. It fixes 24-hour value and observation-mask inputs, a Transformer encoder, patient-level splits, four endpoints, two adaptation protocols, and five matched seeds across MIMIC-IV, eICU-CRD, and their pooled union. Masked autoencoding (MAE), joint-embedding prediction (JEPA), and contrastive learning provide reference implementations. Among these objectives, MAE leads mean AUPRC in all nine frozen classification comparisons, also under converged probes, and in seven of nine fine-tuning comparisons. Fine-tuned MAE leads hospital mortality at full labels and scores higher than a supervised Transformer of the same architecture at every label fraction, and SSL fine-tuning gives the lowest length-of-stay error. Supervised baselines remain strong elsewhere: GRU-D leads 24-hour mortality and XGBoost leads acute kidney injury in all three regimes. Rankings depend on the adaptation protocol, the comparator set, and whether the probe loss matches the metric. A two-seed MIMIC-IV rerun with the maintained code stays within two across-seed SDs of the reference in all 48 downstream runs. The code repository supports new objectives, shared downstream evaluation, and regeneration of the reference results, so new SSL methods can be compared with these references under the same conditions. Generalization to unseen hospitals is not tested here and is the natural next extension.

## MegNIST, A Benchmark for Non-Invasive Inner Speech Decoding

DevFeed: [MegNIST, A Benchmark for Non-Invasive Inner Speech Decoding](<https://devfeed.tech/articles/megnist-a-benchmark-for-non-invasive-inner-speech-decoding-139164.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21399>)

Author: Teyun Kwon, Pratik Somaiya, Gereon Elvers, Francesco Mantegna, Anna Camera, Sven Braeutigam, Mark Woolrich, Oiwi Parker Jones

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Decoding](<https://devfeed.tech/topics/decoding.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmark-dataset](<https://devfeed.tech/tags/benchmark-dataset.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [format](<https://devfeed.tech/tags/format.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [python](<https://devfeed.tech/tags/python.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [recordings](<https://devfeed.tech/tags/recordings.md>), [speech](<https://devfeed.tech/tags/speech.md>), [tensors](<https://devfeed.tech/tags/tensors.md>)

### AI overview

MegNIST provides 12,000 MEG trials from one participant imagining the digits zero through nine across four sessions, offering a benchmark for non-invasive inner-speech decoding. Recordings have 306 channels sampled at 250 Hz and are split by experimental run into 10,000 training, 1,000 validation, and 1,000 test trials to preserve temporal independence. The data are available in BIDS and serialized-tensor formats; the description also notes example code and a PyTorch-compatible loader in pnpl. Baseline classifiers decode imagined digits significantly above chance. The evidence describes a single participant and digit task, so it does not establish performance across people or broader inner speech.

### Source excerpt

arXiv:2611.21399v1 Announce Type: new Abstract: MegNIST is a benchmark dataset for developing and evaluating machine learning methods for inner speech decoding from magnetoencephalography (MEG). The dataset comprises 12,000 trials of a single participant imagining speaking the digits zero through nine, recorded across four sessions. Each trial contains 306-channel MEG recordings sampled at 250 Hz, preprocessed using signal space separation, filtering, and downsampling. Data are provided as pre-defined training (10,000 trials), validation (1,000 trials), and test (1,000 trials) splits, with entire experimental runs assigned to each split to preserve temporal independence. The dataset is available in both BIDS format for neuroimaging analysis and as serialised tensors for machine learning workflows. Baseline classification results demonstrate that imagined digits can be decoded significantly above chance. Standardised splits, validated baselines, example code, and a PyTorch-compatible loader in the pnpl Python library enable straightforward integration with machine learning frameworks, lowering barriers for researchers to rapidly develop and benchmark inner speech decoding methods for non-invasive brain-computer interfaces.

[Next page](<https://devfeed.tech/topics/reproducibility.md?cursor=WyIyMDI2LTExLTE3VDA1OjAwOjAwKzAwOjAwIiwgImMyOGE4MGJmLTRhY2MtNDU1ZC04Y2E4LTAyZGFmYWExNmEwMSJd>)