# Ground truth / benchmark quality

The quality and reliability of ground-truth labels and benchmarks used to evaluate machine-learning systems.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## AesDistill: Distilling Generative Aesthetics for Image Color Enhancement

DevFeed: [AesDistill: Distilling Generative Aesthetics for Image Color Enhancement](<https://devfeed.tech/articles/aesdistill-distilling-generative-aesthetics-for-image-color-enhancement-145556.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25635>)

Author: Chuanzhi Xu, Chengkun Yue, Huiqi Liang, Haoyang Li, Zihan Deng, Qingwen Zeng, Pengfei Ye, Weidong Cai

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Color Grading](<https://devfeed.tech/topics/color-grading.md>), [image editing](<https://devfeed.tech/topics/image-editing.md>), [Image](<https://devfeed.tech/topics/image.md>), [ai-image-generator](<https://devfeed.tech/topics/ai-image-generator.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>)

Tags: [color](<https://devfeed.tech/tags/color.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [distilling](<https://devfeed.tech/tags/distilling.md>), [enhancement](<https://devfeed.tech/tags/enhancement.md>), [generative](<https://devfeed.tech/tags/generative.md>), [image](<https://devfeed.tech/tags/image.md>)

### AI overview

AesDistill transfers color-editing aesthetics from four open-source generative models into a compact 1.13-million-parameter student, aiming to preserve image semantics while avoiding the narrow preferences represented by expert targets. The authors generated 64,644 teacher-edited images at a cost of about 3,360 H100 GPU-hours and trained the student with confidence-weighted supervision. In evaluations using 12 aesthetic and 4 image-quality assessment models, teacher targets ranked above the FiveK and PPR10K expert-target means; across both datasets, the student outperformed 20 baseline methods and ranked behind the four teachers. These are reported results from the abstract.

### Source excerpt

arXiv:2611.25635v1 Announce Type: cross Abstract: Image color enhancement aims to improve visual appeal, but previous methods often learn to reproduce ground-truth expert targets. These targets reflect the preferences of a few experts and cover a limited range of aesthetic choices. Matching them does not ensure stronger aesthetic quality across different tastes and scenes. By contrast, large generative models learn rich visual knowledge from large image collections and often produce vivid, appealing image edits. However, direct generation can change the source image's semantic structure along with its colors. In this paper, we introduce AesDistill, a framework for distilling generative aesthetics into image color enhancement. We use 4 leading open-source large generative models to produce 64,644 edited images as teacher targets at a cost of about 3,360 H100 GPU-hours, forming the AesDistill Dataset. AesDistill also includes AesDistill Student, a 1.13M-parameter model that learns teacher color changes through confidence-weighted supervision while preserving the source image's semantic structure. We evaluate AesDistill with 12 aesthetic and 4 image-quality assessment models. Every teacher's targets rank above the FiveK and PPR10K expert target means in the joint assessment. Across both datasets, AesDistill Student outperforms all 20 baseline methods and ranks directly behind the 4 teachers. These results demonstrate that distilling aesthetics from generative models enables a compact student to outperform methods trained on expert targets.

## BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection

DevFeed: [BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection](<https://devfeed.tech/articles/buggymle-scaling-machine-learning-engineering-difficulty-with-bug-injection-145007.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25088>)

Author: John Wu, Rushi Qiang, Yuchen Zhuang, Guowang Li, Girish Narayanswamy, Max Xu, Sam Schmidgall, Mark Malhotra, Shwetak Patel, Jimeng Sun, Daniel McDuff, Bo Dai, Xin Liu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [AI Development](<https://devfeed.tech/topics/ai-development.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [ai-systems](<https://devfeed.tech/tags/ai-systems.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [based](<https://devfeed.tech/tags/based.md>), [bench](<https://devfeed.tech/tags/bench.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bug](<https://devfeed.tech/tags/bug.md>), [code](<https://devfeed.tech/tags/code.md>), [coding](<https://devfeed.tech/tags/coding.md>), [complex](<https://devfeed.tech/tags/complex.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ma](<https://devfeed.tech/tags/cs-ma.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [learning](<https://devfeed.tech/tags/learning.md>), [machine](<https://devfeed.tech/tags/machine.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

BuggyMLE creates harder repository-level machine-learning engineering tasks by injecting controlled bugs into working ML repositories. Its Code Complexity Index uses ten structural features, including repository scale, dependency depth, and patch size; across eight coding benchmarks, it predicts average solve rates with R² = 0.951. From 222 repositories, the framework produces over 6,600 variants and nearly 260,000 lines of injected code. On BuggyMLE-Bench-Lite, the highest solve rate among 14 models falls from 95% at the easiest tier to 9% at the hardest. A three-level curriculum built from 200 additional repositories fine-tuned Qwen3.5-9B to rival frontier models on that benchmark; the model also improved reported results on SWE-bench Lite (24% to 33%) and LiveCodeBench v6 pass@1 (69% to 71%).

### Source excerpt

arXiv:2611.25088v1 Announce Type: cross Abstract: Enabling AI systems to autonomously improve AI models requires agents capable of performing machine learning engineering (MLE). In practice, designing, training, and evaluating complex models demands modifying multi-file repositories to reproduce results, debug training runs, integrate datasets, and port frameworks. However, existing MLE benchmarks limit agents to single-file edits without full repository or shell access, and common repository-level software engineering benchmarks have rapidly saturated. To address this, we introduce the Code Complexity Index (CCI), a difficulty metric based on ten structural features spanning repository scale, dependency depth, and patch size. CCI strongly predicts average solve rates across eight public coding benchmarks ($R^2 = 0.951$), identifying patch size and complexity as top predictors. Guided by CCI, we develop BuggyMLE, a scalable framework where agents inject controlled mutations into working ML repositories, synthesizing over 6,600 bug-injected MLE repository variants and nearly 260K lines of injected code based on 222 source repositories. Using BuggyMLE, we make two contributions. First, we construct BuggyMLE-Bench-Lite, the first difficulty-graded, repository-level variant of MLE-Bench Lite; across 14 evaluated models, the top solve rate drops from 95% to 9% as difficulty is scaled from the easiest to the hardest tier. Second, we synthesize a training curriculum across three difficulty levels over 200 additional ML repositories, fine-tuning a Qwen3.5-9B model to rival frontier models on BuggyMLE-Bench-Lite. Our trained model transfers beyond ML, raising its SWE-bench Lite resolve rate from 24% to 33% and its LiveCodeBench v6 pass@1 from 69% to 71%.

## Fool's Gold: Defensive Deception Against Safety-Removal Attacks on Open-Weight Models

DevFeed: [Fool's Gold: Defensive Deception Against Safety-Removal Attacks on Open-Weight Models](<https://devfeed.tech/articles/fool-s-gold-defensive-deception-against-safety-removal-attacks-on-open-weight-models-145363.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.17202>)

Author: Mark Russinovich

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>)

Tags: [abliteration](<https://devfeed.tech/tags/abliteration.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [deception](<https://devfeed.tech/tags/deception.md>), [defensive](<https://devfeed.tech/tags/defensive.md>), [open-weight-models](<https://devfeed.tech/tags/open-weight-models.md>), [safety](<https://devfeed.tech/tags/safety.md>)

### AI overview

Fool's Gold (decoy hardening) is designed to blunt abliteration, which removes a refusal-mediating direction from open-weight models. It concedes that refusal stripping can occur, then trains models to give fluent but critically falsified decoys to hazardous operational requests in the attacked state; a refusal pin and benign leash aim to preserve clean-state behavior. In six of seven tested models (9B-122B), 51-90% of attacked responses to held-out prompts were decoys, with reported benign-behavior and capability budgets met; the smaller seventh model failed the efficacy gate. On a CBRNE-adjacent benchmark slice, the defended 122B model gave fatally wrong, matched-quality answers at a reported 82-86% rate, versus at most 10% undefended. Repeated sampling still sometimes reconstructed usable procedures, and the authors say tested observation surfaces could not distinguish decoys from correct answers without ground truth. Evaluation covered chemical and biological hazards; the defense does not address in-context jailbreaks and protects only the initially released weights.

### Source excerpt

arXiv:2608.17202v2 Announce Type: replace Abstract: Safety alignment in open-weight language models is trivially removable: abliteration projects a refusal-mediating direction out of the weights in minutes, and no release-time defense we are aware of prevents it durably. What cannot be prevented can be deceived. Our defense, decoy hardening ("Fool's Gold"), concedes the refusal strip and poisons its payoff: once refusal is stripped, most answers to hazardous operational requests are confident, fluent decoys whose critical elements are falsified. Decoys are trained inside a differentiable simulation of the attack, expressing only in the attacked state; a refusal pin and benign leash hold clean-state behavior to the original. We instantiate it on seven models from five families (9B-122B, dense and mixture-of-experts). On the six models passing our pre-registered efficacy gate, 0.51-0.90 of attacked-state responses to held-out prompts are decoys, +0.27-0.84 attributable to the defense; all six stay within registered benign-behavior and capability budgets; the seventh (smaller) fails the gate (boundary case). Rates replicate on a frozen test split or untouched strata. The claim is epistemic: without independent ground truth, no observation surface we tested separates falsified answers from correct ones - on external red-team benchmarks' CBRNE-adjacent slice, the defended 122B is fatally wrong on 0.82-0.86 of matched-quality answers vs at most 0.10 undefended. Repeated sampling does not restore trust: element-wise consensus at K=64 reconstructs a fully usable procedure on 0.083-0.625 of prompts where the instrument validates, vs 0.58-0.96 undefended, with no label-free way to tell the regimes apart; on the weakest such model the claim is per-draw only. We evaluate chemical and biological hazards; the defense does not address in-context jailbreaks and protects only the initially released defended weights.

## Benchmarking Robust Aggregation in Decentralized Gradient Marketplaces

DevFeed: [Benchmarking Robust Aggregation in Decentralized Gradient Marketplaces](<https://devfeed.tech/articles/benchmarking-robust-aggregation-in-decentralized-gradient-marketplaces-144856.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2509.05833>)

Author: Zeyu Song, Sainyam Galhotra, Shagufta Mehnaz

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Federated Learning](<https://devfeed.tech/topics/federated-learning.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [Model Development](<https://devfeed.tech/topics/model-development.md>)

Tags: [adverse-selection](<https://devfeed.tech/tags/adverse-selection.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [contamination](<https://devfeed.tech/tags/contamination.md>), [contributions](<https://devfeed.tech/tags/contributions.md>), [cs-gt](<https://devfeed.tech/tags/cs-gt.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [data-marketplace](<https://devfeed.tech/tags/data-marketplace.md>), [data-products](<https://devfeed.tech/tags/data-products.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [decentralized](<https://devfeed.tech/tags/decentralized.md>), [differential-privacy](<https://devfeed.tech/tags/differential-privacy.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [malicious](<https://devfeed.tech/tags/malicious.md>), [marketplace](<https://devfeed.tech/tags/marketplace.md>), [marketplaces](<https://devfeed.tech/tags/marketplaces.md>), [mechanism](<https://devfeed.tech/tags/mechanism.md>), [model](<https://devfeed.tech/tags/model.md>), [model-performance](<https://devfeed.tech/tags/model-performance.md>), [robust](<https://devfeed.tech/tags/robust.md>)

### AI overview

The benchmark finds that aggregation mechanisms can protect model performance while misallocating payment in gradient data marketplaces. Across a standard baseline and 11 aggregation mechanisms on five vision, text, and tabular datasets, honest high-variance updates were often rejected as outliers while strategically homogeneous malicious updates were accepted and rewarded. The authors report that this adverse selection persists with continuous weight-proportional payment, compounds over repeated rounds as underpaid honest sellers exit, and worsens when differential-privacy noise makes honest updates look contaminated. The framework evaluates mechanism integrity--whether selection and payment reward utility--separately from model integrity.

### Source excerpt

arXiv:2509.05833v2 Announce Type: replace Abstract: Institutions that cannot centralize or sell raw data increasingly exchange model updates instead: cross-silo federated learning is already in production among hospital networks and pharmaceutical consortia. A growing line of research targets the next step: marketplaces in which gradient updates are exchanged as paid, query-specific data products, though none is yet deployed at scale. We formalize this emerging paradigm as a Gradient Data Marketplace (GDM), in which data consumers issue learning tasks and distributed participants contribute query-specific gradient updates. Federated learning asks whether the model survives an attack; a marketplace must also ask who got paid. Existing protocols ensure model-level robustness but fail to capture whether high-quality data is correctly identified and rewarded. In a GDM, selection determines payment, incentivizing participants to optimize for inclusion rather than utility. We introduce a modular benchmarking framework that separates mechanism integrity (whether selection and payment reward true utility) from model integrity, and evaluate a standard baseline alongside 11 widely used aggregation mechanisms on five real-world vision, text, and tabular datasets. Our results reveal consistent adverse selection: honest high-variance contributions are rejected as outliers while strategically homogeneous malicious contributions are accepted and rewarded, even when model performance remains stable. These failures are not artifacts of our accounting: they persist under continuous, weight-proportional payment, compound across repeated market rounds as underpaid honest sellers exit, and worsen when honest sellers add differential-privacy noise, which the filters misread as contamination. Our benchmark provides a foundation for designing incentive-aware, auditable data selection mechanisms in emerging data marketplaces.

## On the Reliability of CyberGym Evaluation: Task Material Consistency and Reference-Target Identity

DevFeed: [On the Reliability of CyberGym Evaluation: Task Material Consistency and Reference-Target Identity](<https://devfeed.tech/articles/on-the-reliability-of-cybergym-evaluation-task-material-consistency-and-reference-target-identity-145781.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25790>)

Author: Yule Ding, Hongzhen Wu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [audit](<https://devfeed.tech/topics/audit.md>)

Tags: [assessment](<https://devfeed.tech/tags/assessment.md>), [audit](<https://devfeed.tech/tags/audit.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [cybergym](<https://devfeed.tech/tags/cybergym.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [identity](<https://devfeed.tech/tags/identity.md>), [material](<https://devfeed.tech/tags/material.md>), [proof-of-concept](<https://devfeed.tech/tags/proof-of-concept.md>), [reference](<https://devfeed.tech/tags/reference.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [reproduction](<https://devfeed.tech/tags/reproduction.md>), [target](<https://devfeed.tech/tags/target.md>), [vulnerability](<https://devfeed.tech/tags/vulnerability.md>)

### AI overview

CyberGym's differential crash check alone may overstate successful vulnerability reproduction: a PoC must trigger the defect associated with the reference crash report. Auditing all 1,507 tasks, the authors label 338 (22.4%) as materially inconsistent, most often for coverage inconsistency (205) or descriptions and patches that jointly indicate another defect (110). In one complete Level-1 GLM-5.3 run, their identity procedure--execution checks, crash-signature matching, and LLM-assisted root-cause review--classifies 89 of 1,286 differential passes as non-target and 10 as insufficient evidence. Requiring a target hit reduces the pass rate from 85.3% to 78.8%; non-target PoCs also appear among tasks with no detected material inconsistency (39 of 1,052 passes, 3.7%). The authors caution that identity verdicts need independent validation and recommend assessing task consistency and submission identity separately.

### Source excerpt

arXiv:2611.25790v1 Announce Type: cross Abstract: CyberGym evaluates vulnerability reproduction by checking whether a proof-of-concept input (PoC) crashes a vulnerable program version but not a fixed version. Interpreting this differential pass as successful reproduction requires consistent task materials and evidence that the PoC triggers the intended defect. We examine these requirements through a material audit of all 1,507 CyberGym tasks and an identity assessment of the final PoCs from one complete Level-1 GLM-5.3 run. Taking the defect associated with each reference crash report as the target, the audit assigns inconsistency labels to 338 tasks (22.4%) after correcting two labels identified by stratified blind review. The largest categories are coverage inconsistency (205 tasks) and descriptions and patches that jointly point to another defect (110 tasks). We assess PoC identity using an execution check, crash-signature matching, and LLM-assisted root-cause review. Of the 1,286 differential passes, the procedure assigns 89 non-target and 10 insufficient-evidence verdicts; requiring a target hit lowers the pass rate from 85.3% to 78.8%. Non-target verdicts also occur in tasks with no material inconsistency found: 39 of 1,052 differential passes (3.7%). Although the identity verdicts require independent validation, these findings motivate separate checks of task-material consistency and submission identity when interpreting vulnerability-reproduction scores.

## From Benchmarks to the Bench: Preference, Correspondence and Selectivity in Bidirectional Drug-Target Screening

DevFeed: [From Benchmarks to the Bench: Preference, Correspondence and Selectivity in Bidirectional Drug-Target Screening](<https://devfeed.tech/articles/from-benchmarks-to-the-bench-preference-correspondence-and-selectivity-in-bidirectional-drug-target-screening-145171.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26369>)

Author: Jiangjie Qiu, Chengrui Xu, Xiaonan Wang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [drug-discovery](<https://devfeed.tech/tags/drug-discovery.md>), [drugs](<https://devfeed.tech/tags/drugs.md>), [hit](<https://devfeed.tech/tags/hit.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [measurements](<https://devfeed.tech/tags/measurements.md>), [model-evaluation](<https://devfeed.tech/tags/model-evaluation.md>), [models](<https://devfeed.tech/tags/models.md>), [overlap](<https://devfeed.tech/tags/overlap.md>), [predictions](<https://devfeed.tech/tags/predictions.md>), [random](<https://devfeed.tech/tags/random.md>), [recommendations](<https://devfeed.tech/tags/recommendations.md>), [targets](<https://devfeed.tech/tags/targets.md>), [test](<https://devfeed.tech/tags/test.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

Across 13 released drug-target interaction systems, Top-10 recommendations often differed sharply: for the same target, 77.8% of model-pair comparisons had no molecule overlap; for the same drug, 70.1% had no protein overlap. Published inhibition measurements showed that some systems enriched active candidates, but their hit yields changed little across different drugs or targets, and some rarely ranked both molecules and kinases correctly when drugs favoured different kinases. Other systems lost more hits when recommendations were matched to different screening contexts. Retraining one architecture with the same data and objective but different random seeds produced similar test performance yet only about two shared candidates on average between Top-10 lists in a separate catalogue. The findings suggest benchmark performance alone may not establish whether rankings can guide experimental choices.

### Source excerpt

arXiv:2611.26369v1 Announce Type: new Abstract: Identifying active molecules for known targets and potential targets for existing drugs is central to drug discovery and repurposing. Drug-target interaction (DTI) models can rank vast numbers of drug-target pairs, but limited experimental capacity restricts testing to a small number of candidates. We compared the bidirectional recommendations of 13 released systems across 711 molecular structures of FDA-approved drugs and 614 human proteins. For the same target or drug, pairs of models produced completely disjoint Top-10 molecule and protein lists in 77.8% and 70.1% of comparisons, respectively. Even where recommendations overlapped, agreement between some systems largely reflected recurring preferences for the same candidates. Published inhibition measurements for 79 molecules and 65 kinases revealed that some systems enriched active candidates, yet their hit yields changed little when recommendations were matched to different drugs or targets, whereas other systems lost substantially more hits. Some of the former systems also rarely ranked both molecules and kinases correctly when two drugs favoured different kinases. We further retrained the same architecture on the same data, with the training objective fixed and random seeds varied. Despite similar test performance, mean pairwise Top-10 overlap on a separate screening catalogue was about two candidates. Our study moves DTI model evaluation beyond benchmark performance to assess whether model predictions can guide meaningful experimental decisions in drug discovery and target identification.

## Designer-RSI: Evolving Procedural Memory for Graphic Design Agent

DevFeed: [Designer-RSI: Evolving Procedural Memory for Graphic Design Agent](<https://devfeed.tech/articles/designer-rsi-evolving-procedural-memory-for-graphic-design-agent-145375.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.22086>)

Author: Hongyang Du, Lan Yan, Christian Flores, Asim Kadav

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Agent Skills](<https://devfeed.tech/topics/agent-skills.md>), [Tool](<https://devfeed.tech/topics/tool.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>)

Tags: [acquisition](<https://devfeed.tech/tags/acquisition.md>), [agent](<https://devfeed.tech/tags/agent.md>), [creative-software](<https://devfeed.tech/tags/creative-software.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [design](<https://devfeed.tech/tags/design.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [execution](<https://devfeed.tech/tags/execution.md>), [external](<https://devfeed.tech/tags/external.md>), [graphic-design](<https://devfeed.tech/tags/graphic-design.md>), [memory](<https://devfeed.tech/tags/memory.md>), [modular](<https://devfeed.tech/tags/modular.md>), [procedural](<https://devfeed.tech/tags/procedural.md>), [rsi](<https://devfeed.tech/tags/rsi.md>), [skills](<https://devfeed.tech/tags/skills.md>), [vlm](<https://devfeed.tech/tags/vlm.md>)

### AI overview

Designer-RSI adapts a graphic design agent with frozen foundation-model weights by evolving a modular skill bank from unlabeled design trajectories. It widens coverage by adding skills for recurring gaps and deepens execution quality by revising failing skills against successful runs. A Pareto-style matched replay gate accepts updates only when they improve sampled briefs without local regressions, addressing noisy visual judges and costly, confounded tool runs. Across held-out product briefs, eight external benchmarks, and two backbones, the approach outperformed static documentation skills, with execution success reported up to 99.8% and blind win rate up to 67.6% over the base agent. Ablations indicate acquisition and revision work synergistically; neither alone provided reliable gains.

### Source excerpt

arXiv:2609.22086v2 Announce Type: replace Abstract: Deployed software agents face a continuous stream of user requests while their foundation-model weights remain frozen between releases. Recursive self-improvement (RSI) via external memory offers a practical path to continual adaptation, yet prior methods largely assume programmatic verifiers, cheap retries, or a single global prompt. Creative software breaks these assumptions, as open-ended visual briefs lack ground truth, VLM judges are noisy, and multi-step tool executions are costly and confounded by external assets. We present Designer-RSI, which evolves a modular SKILL.md skill bank from unlabeled design trajectories for an agent spanning over 230 raster, vector, and layout tools. Designer-RSI couples widening (acquiring new skills for recurring capability gaps) with deepening (revising failing skills by contrasting successful and failed runs), and screens updates through a Pareto-style matched replay gate that requires strict gains on sampled briefs without local regressions. Across held-out product briefs, eight external benchmarks, and two backbones, traffic-driven evolution outperforms static documentation skills, lifts execution success up to 99.8%, improves requirement fulfillment, and reaches up to a 67.6% blind win rate over the base agent. Ablations further show that skill acquisition and revision are strongly synergistic: neither suffices alone, and reliable gains require evolving coverage and execution quality together.

## AI GameStore: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Games

DevFeed: [AI GameStore: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Games](<https://devfeed.tech/articles/ai-gamestore-scalable-open-ended-evaluation-of-machine-general-intelligence-with-human-games-145317.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.17594>)

Author: Lance Ying, Ryan Truong, Prafull Sharma, Kaiya Ivy Zhao, Nathan Cloos, Kelsey R. Allen, Thomas L. Griffiths, Katherine M. Collins, Jos\\'e Hern\\'andez-Orallo, Phillip Isola, Samuel J. Gershman, Joshua

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Machine Intelligence](<https://devfeed.tech/topics/machine-intelligence.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Frontier AI](<https://devfeed.tech/topics/frontier-ai.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [AI Development](<https://devfeed.tech/topics/ai-development.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-benchmarks](<https://devfeed.tech/tags/ai-benchmarks.md>), [ai-systems](<https://devfeed.tech/tags/ai-systems.md>), [all](<https://devfeed.tech/tags/all.md>), [app](<https://devfeed.tech/tags/app.md>), [app-store](<https://devfeed.tech/tags/app-store.md>), [apple](<https://devfeed.tech/tags/apple.md>), [apple-app-store](<https://devfeed.tech/tags/apple-app-store.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [automatically](<https://devfeed.tech/tags/automatically.md>), [based](<https://devfeed.tech/tags/based.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [best](<https://devfeed.tech/tags/best.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [games](<https://devfeed.tech/tags/games.md>), [general](<https://devfeed.tech/tags/general.md>), [intelligence](<https://devfeed.tech/tags/intelligence.md>), [machine](<https://devfeed.tech/tags/machine.md>), [open](<https://devfeed.tech/tags/open.md>)

### AI overview

AI GameStore proposes evaluating machine general intelligence through human-designed games, comparing AI and people with similar experience, time, or other resources. Its platform uses LLMs and human input to create games by sourcing and adapting standardized, containerized environments from popular digital gaming platforms. In an initial study of 100 games drawn from Apple App Store and Steam charts, seven frontier vision-language models played short episodes; even the best scored below 10% of the human average on most games, with particular difficulty in world-model learning, memory, and planning. The authors present the platform as an early step toward scalable, open-ended evaluation and outline further development as future work.

### Source excerpt

arXiv:2602.17594v2 Announce Type: replace Abstract: Rigorously evaluating machine intelligence against the broad spectrum of human general intelligence has become increasingly important and challenging in this era of rapid technological advance. Conventional AI benchmarks typically assess only narrow capabilities in a limited range of human activity. Most are also static, quickly saturating as model capabilities improve. We propose that a more promising way to evaluate human-like general intelligence in AI systems is through a particularly strong form of general game playing: studying how and how well they play and learn to play all conceivable human games, in comparison to human players with the same level of experience, time, or other resources. We define a "human game" to be a game designed by humans for humans, and argue for the evaluative suitability of this space of all such games people can imagine and enjoy -- the "Multiverse of Human Games". Taking a first step towards this vision, we introduce the AI GameStore, a scalable and open-ended platform that uses LLMs with humans-in-the-loop to synthesize new representative human games, by automatically sourcing and adapting standardized and containerized variants of game environments from popular human digital gaming platforms. We generated 100 such games based on the top charts of Apple App Store and Steam, and evaluated seven frontier vision-language models (VLMs) on short episodes of play. The best models achieved less than 10% of the human average score on the majority of the games, and especially struggled with games that challenge world-model learning, memory and planning. We conclude with a set of next steps for building out the AI GameStore as a practical way to measure and drive progress toward human-like general intelligence in machines.

## FLUID: Flexible, Evidence-Grounded Situated Task Guidance

DevFeed: [FLUID: Flexible, Evidence-Grounded Situated Task Guidance](<https://devfeed.tech/articles/fluid-flexible-evidence-grounded-situated-task-guidance-145677.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26491>)

Author: Yunqi Zhao, Run Peng, Ze Dong, Ryan Lu, Yinpei Dai, Joyce Chai

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [User Experience](<https://devfeed.tech/topics/user-experience.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [flexible](<https://devfeed.tech/tags/flexible.md>), [fluid](<https://devfeed.tech/tags/fluid.md>), [framework](<https://devfeed.tech/tags/framework.md>), [grounded](<https://devfeed.tech/tags/grounded.md>), [guidance](<https://devfeed.tech/tags/guidance.md>), [human-ai-interaction](<https://devfeed.tech/tags/human-ai-interaction.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [latency](<https://devfeed.tech/tags/latency.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [overlap](<https://devfeed.tech/tags/overlap.md>), [precision](<https://devfeed.tech/tags/precision.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [response](<https://devfeed.tech/tags/response.md>), [responses](<https://devfeed.tech/tags/responses.md>), [speech](<https://devfeed.tech/tags/speech.md>)

### AI overview

FLUID models situated task guidance around the evidence for an intervention, whether it is necessary, and how flexible its timing is--including when silence is appropriate. Using human-human guidance from WTaG and HoloAssist, the framework supports both evaluation and a real-time guidance agent. The abstract reports that this agent had the strongest overall balance among evaluated systems for response correctness, intervention precision, timely guidance, and latency, while speech overlap remained an area for improvement. The work aims to move evaluation beyond matching a single fixed dialogue response and timing.

### Source excerpt

arXiv:2611.26491v1 Announce Type: new Abstract: Real-time task guidance is inherently flexible: when an agent guides a user through a task, multiple responses and timings can be appropriate, and sometimes the best action is to remain silent. Existing benchmarks often miss this flexibility by treating synthesized dialogues as fixed targets for what to say and when to say it. We introduce FLUID (Flexible, Evidence-Grounded Situated Task Guidance), a unified framework for modeling, evaluating, and enabling flexible interaction in multimodal task guidance. Using naturally occurring human-human guidance from WTaG and HoloAssist, FLUID grounds interventions in their supporting evidence, necessity, and timing flexibility, and uses this representation both to evaluate flexible guidance and to build a real-time agent for task guidance. Our agent achieves the strongest overall balance of response correctness, intervention precision, timely guidance, and response latency among evaluated systems, while still leaving room for improvement in avoiding speech overlap. Together, FLUID moves situated task guidance beyond rigid reference matching toward more natural and adaptive human-AI interaction. Project page: https://fluid-guidance.github.io.

## Measuring What Matters: Synthetic Benchmarks for Concept Bottleneck Models

DevFeed: [Measuring What Matters: Synthetic Benchmarks for Concept Bottleneck Models](<https://devfeed.tech/articles/measuring-what-matters-synthetic-benchmarks-for-concept-bottleneck-models-142528.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.04326>)

Author: Julian Skirzynski, Harry Cheon, Shreyas Kadekodi, Meredith Stewart, Berk Ustun

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [Model Development](<https://devfeed.tech/topics/model-development.md>), [AI Models](<https://devfeed.tech/topics/ai-models.md>), [human feedback](<https://devfeed.tech/topics/human-feedback.md>)

Tags: [annotation](<https://devfeed.tech/tags/annotation.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [automation](<https://devfeed.tech/tags/automation.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [concepts](<https://devfeed.tech/tags/concepts.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [decision](<https://devfeed.tech/tags/decision.md>), [experts](<https://devfeed.tech/tags/experts.md>), [failure](<https://devfeed.tech/tags/failure.md>), [failures](<https://devfeed.tech/tags/failures.md>), [interpretability](<https://devfeed.tech/tags/interpretability.md>), [models](<https://devfeed.tech/tags/models.md>), [performance](<https://devfeed.tech/tags/performance.md>), [quality](<https://devfeed.tech/tags/quality.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [supervision](<https://devfeed.tech/tags/supervision.md>), [support](<https://devfeed.tech/tags/support.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>)

### AI overview

The paper introduces synthetic benchmarks for concept bottleneck models, addressing the shortage of real-world datasets with concept labels. Researchers can generate labeled data and vary modality, concept selection, and annotation quality to study which problems suit these models and isolate causes of failure. The authors demonstrate evaluation for decision support, where models help human experts, and automation of routine tasks without supervision. They report that the benchmarks reveal failure modes and help attribute them to model architecture or data.

### Source excerpt

arXiv:2606.04326v2 Announce Type: replace-cross Abstract: Concept bottleneck models predict using high-level concepts detected in inputs. Although these models provide a simple way to reap benefits from interpretability, they are hard to train and evaluate because very few real-world datasets include concept labels. In this paper, we develop synthetic benchmarks for concept bottleneck models. With our benchmarks, researchers can generate labeled datasets and control properties that affect performance, such as data modality, the choice of concepts, or annotation quality. They can thus find problems that suit these models and isolate the factors behind their failures. We demonstrate how our benchmarks can be used to evaluate major use cases for concept bottleneck models: decision support, i.e., helping human experts make better decisions; and automation, i.e., handling routine tasks without supervision. Our demonstrations show how the benchmarks can identify new failure modes and attribute them to either the architecture or the data.

## OracleZoom: Reference-Constrained Recursive Image Super-Resolution Beyond Ground Truth

DevFeed: [OracleZoom: Reference-Constrained Recursive Image Super-Resolution Beyond Ground Truth](<https://devfeed.tech/articles/oraclezoom-reference-constrained-recursive-image-super-resolution-beyond-ground-truth-142557.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.06490>)

Author: Shubhashis Roy Dipta, Nobin Sarwar, Md Rizwan Parvez

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Image processing](<https://devfeed.tech/topics/image-processing.md>), [Image](<https://devfeed.tech/topics/image.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>)

Tags: [computer-vision](<https://devfeed.tech/tags/computer-vision.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [image](<https://devfeed.tech/tags/image.md>), [super-resolution](<https://devfeed.tech/tags/super-resolution.md>)

### AI overview

OracleZoom trains recursive super-resolution beyond available ground truth by feeding a prediction back into the model for a 4x->16x step, then projecting the 16x output back to the last 4x ground-truth image for a cross-scale constraint. Since that constraint checks only coarse content, a no-reference quality objective guides fine detail, while a latent prior is used to limit hallucinated artifacts. Across seven datasets, it reports the highest CLIPIQA at every zoom depth (mean 0.713 versus 0.621 for Chain-of-Zoom), with the largest gains at deeper zooms, despite training only to 16x and optimizing a different quality metric. Two pooled ground-truth-anchored judges found the same hallucination rate as Chain-of-Zoom at 16x and a lower rate at 4x; removing the cross-scale loss increased hallucination at 16x.

### Source excerpt

arXiv:2609.06490v2 Announce Type: replace-cross Abstract: Recursive image Super-Resolution (SR) reaches extreme magnification by feeding each prediction back into the same model, like zooming into an image again and again. Real ground truth for deep zooms is impractical to capture (a full $256\times$ target would be a 52 GB image), so predictions at deeper zooms have no supervision. Yet the last ground-truth image still contains the region that every deeper zoom magnifies. OracleZoom turns this region into a training signal. Inspired by on-policy self-distillation, the model trains on its own prediction for one recursive step ($4\times \rightarrow 16\times$), and a cross-scale loss projects each $16\times$ prediction back to the last ground truth at $4\times$. This projection can verify only coarse content, so a no-reference quality objective guides the remaining fine detail, and a latent prior stops this objective from adding hallucinated artifacts. On seven datasets, OracleZoom achieves the highest CLIPIQA at every zoom depth (mean 0.713 vs. 0.621 for Chain-of-Zoom), with the largest gains at the deepest zooms, even though training stops at $16\times$ and optimizes a different quality metric. Pooled over two ground-truth-anchored judges, this added detail comes at the same hallucination rate as Chain-of-Zoom at $16\times$ and a lower rate at $4\times$, and removing the cross-scale loss raises hallucination at $16\times$.

## Cascadeability and the Realization Gap: An Empirical Study of LLM-Powered Semantic Filtering

DevFeed: [Cascadeability and the Realization Gap: An Empirical Study of LLM-Powered Semantic Filtering](<https://devfeed.tech/articles/cascadeability-and-the-realization-gap-an-empirical-study-of-llm-powered-semantic-filtering-142288.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23898>)

Author: Nan Hou, Kangfei Zhao, Jiadong Xie, Jeffrey Xu Yu

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [bottlenecks](<https://devfeed.tech/tags/bottlenecks.md>), [cascade](<https://devfeed.tech/tags/cascade.md>), [class](<https://devfeed.tech/tags/class.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [proxy](<https://devfeed.tech/tags/proxy.md>), [query-processing](<https://devfeed.tech/tags/query-processing.md>), [questions](<https://devfeed.tech/tags/questions.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [reduction](<https://devfeed.tech/tags/reduction.md>), [release](<https://devfeed.tech/tags/release.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [stage](<https://devfeed.tech/tags/stage.md>), [study](<https://devfeed.tech/tags/study.md>), [systems](<https://devfeed.tech/tags/systems.md>), [target](<https://devfeed.tech/tags/target.md>)

### AI overview

The abstract presents a benchmark of four LLM semantic-filtering systems and argues that oracle-call reduction alone does not show whether poor results stem from weak proxies or threshold calibration. It defines cascadeability as the maximum empirical oracle reduction possible from fixed proxy scores at a target accuracy, measured retrospectively with complete oracle labels. Comparing this ceiling with achieved reduction yields a realization gap that, alongside accuracy, helps distinguish proxy and calibration bottlenecks. The benchmark covers 10 real-world workloads with annotations from two LLM families; the abstract says it examines proxy quality, end-to-end performance, workload sensitivities, and component ablations, but does not report specific comparative results.

### Source excerpt

arXiv:2611.23898v1 Announce Type: new Abstract: Semantic filters enable natural language predicates over unstructured documents, but evaluating every document with a large language model (LLM) oracle is expensive. Proxy cascades reduce oracle invocations using cheaper surrogates, but oracle reduction alone cannot distinguish proxy limitations from calibration bottlenecks and establish practical efficiency. We present a unified benchmark of four representative systems, which formulate query processing as a three-stage workflow comprising proxy construction, threshold calibration, and cascade inference, using a two-threshold acceptance rule. We introduce cascadeability: the maximum empirical oracle reduction attainable from fixed proxy scores at a specified accuracy target, computed retrospectively using complete oracle labels. We investigate 8 research questions, spanning proxy quality, end-to-end performance, workload sensitivities, and component ablations, and reveal our key findings accordingly. Comparing cascadeability with achieved oracle reduction yields a realization gap that, interpreted alongside accuracy, exposes distinct proxy and calibration bottlenecks. The benchmark and diagnostics support evaluation beyond oracle reduction and motivate joint proxy-calibration design, class-sensitive quality assessment, and end-to-end cost optimization. We release a dataset-query benchmark, containing 10 real-world workloads across various domains and difficulties, with annotations of two LLM families.

## Screening LLM Judges Without Ground Truth: A Preliminary Evaluation of an AI Dental Hygiene Coach

DevFeed: [Screening LLM Judges Without Ground Truth: A Preliminary Evaluation of an AI Dental Hygiene Coach](<https://devfeed.tech/articles/screening-llm-judges-without-ground-truth-a-preliminary-evaluation-of-an-ai-dental-hygiene-coach-139840.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.24335>)

Author: William DeShong, Barath Kannan, Auberon Palma, Brenda Bradshaw, Ann Bruhn, James F. Leathrum, Jr., Yuzhong Shen

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [ai](<https://devfeed.tech/tags/ai.md>), [audit](<https://devfeed.tech/tags/audit.md>), [bias](<https://devfeed.tech/tags/bias.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [case-study](<https://devfeed.tech/tags/case-study.md>), [coaching](<https://devfeed.tech/tags/coaching.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [correlation](<https://devfeed.tech/tags/correlation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [data](<https://devfeed.tech/tags/data.md>), [decision](<https://devfeed.tech/tags/decision.md>), [decision-rules](<https://devfeed.tech/tags/decision-rules.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [family](<https://devfeed.tech/tags/family.md>), [fine](<https://devfeed.tech/tags/fine.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [gemma-4](<https://devfeed.tech/tags/gemma-4.md>), [gemma-4-31b](<https://devfeed.tech/tags/gemma-4-31b.md>), [hygiene](<https://devfeed.tech/tags/hygiene.md>), [llm](<https://devfeed.tech/tags/llm.md>)

### AI overview

The proposed pipeline screens LLM judges and student models before costly expert review through output audits, comparisons with hosted judges, bias checks, and robustness rules. In a dental hygiene coach case study using synthetic, clinically unvalidated data, two local judges matched hosted-judge agreement on six of seven metrics; Qwen3.8-27B was most similar overall. The pipeline found no qualifying self-preference, and it flagged truncated Qwen3-8B outputs and one hosted judge's uncorroborated hallucination flags. It did not distinguish Granite-4.1-8B from Qwen3-8B under its decision rule; Granite led in 29 of 35 judge-metric cells, but that lean partly reflects Qwen's truncated outputs. No experts graded the responses, and the proposed expert-anchoring stage was not performed, so this demonstrates the screening process rather than clinical model performance.

### Source excerpt

arXiv:2610.24335v2 Announce Type: replace Abstract: Evaluating a fine-tuned coaching model with expert graders is slow and expensive, and an LLM judge cannot be trusted without ground truth. We present a staged pipeline that screens candidate LLM judges and student models before expert review, using explicit decision rules at each stage: judge output audit, similarity with hosted judges, bias checks, and decision robustness. We demonstrate it on an AI dental hygiene coach. The case study uses synthetic, clinically unvalidated training and reference data to demonstrate the operation of the pipeline, rather than to establish the clinical performance of the resulting models. Two fine-tuned small language models (Granite-4.1-8B and Qwen3-8B), each trained at three data sizes, were scored on seven metrics by five judges (three local, two hosted). Two local judges, Qwen3.8-27B and Gemma-4-31B-it, were not distinguishable from the agreement between the hosted judges on six of seven metrics by rank correlation, and Qwen3.8-27B was the most similar to the hosted judges overall. No judge met our criterion for favoring its own model family or training-data source. The student comparison did not separate the two models on any metric under the pipeline's rule, so the choice of Granite-4.1-8B, which led in 29 of 35 judge-by-metric cells, is a lean that depends partly on truncated Qwen3-8B outputs. The pipeline also surfaced problems that raw scores hid, including the truncated outputs and hallucination flags from one hosted judge that no other judge corroborated. No dental hygiene expert has graded these responses, so the findings are preliminary and the pipeline's final stage, anchoring to expert judgement, is proposed but not performed.

## Do LLMs Understand Network Traces? A Comparative Study of Reasoning over Raw and Filtered Traffic Data

DevFeed: [Do LLMs Understand Network Traces? A Comparative Study of Reasoning over Raw and Filtered Traffic Data](<https://devfeed.tech/articles/do-llms-understand-network-traces-a-comparative-study-of-reasoning-over-raw-and-filtered-traffic-data-140051.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22575>)

Author: Irene Gos\\'alvez-White, Roberto Mag\\'an-Carri\\'on, Carlota de la Vega-Soriano, Pedro Garc\\'ia-Teodoro

Published: 2026-11-18T05:00:00Z

Content type: comparison

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Networking](<https://devfeed.tech/sources/arxiv-networking.md>)

Topics: [network-intelligence](<https://devfeed.tech/topics/network-intelligence.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Internet Traffic](<https://devfeed.tech/topics/internet-traffic.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ni](<https://devfeed.tech/tags/cs-ni.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [llms](<https://devfeed.tech/tags/llms.md>), [network-traffic-analysis](<https://devfeed.tech/tags/network-traffic-analysis.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [overhead](<https://devfeed.tech/tags/overhead.md>), [packet](<https://devfeed.tech/tags/packet.md>), [performance](<https://devfeed.tech/tags/performance.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [reliability](<https://devfeed.tech/tags/reliability.md>)

### AI overview

Using benign CICIDS2017 traffic and human-generated ground truth, the study evaluates multiple LLMs on packet-field extraction and flow-sequence reconstruction from raw traffic dumps versus protocol-filtered representations. Responses are scored with an automated LLM-as-a-judge pipeline validated against human evaluations. The abstract reports that protocol filtering reduces prompt overhead and improves answer accuracy by reducing incorrect responses, suggesting it can make LLM-assisted network analysis more efficient and reliable. The excerpt does not provide model-specific results or quantitative measurements.

### Source excerpt

arXiv:2611.22575v1 Announce Type: cross Abstract: The effective delivery and monitoring of ICT services rely heavily on network traffic analysis. However, interpreting complex packet captures remains a challenging task that requires substantial technical expertise. Large language models (LLMs) offer promising opportunities to automate network traffic in- vestigation, yet their reliability in packet-level field extraction and flow-level sequence reconstruction remains an open research question. This study proposes a reproducible, question-answering- based evaluation methodology to assess the capabilities of LLMs in network trace analysis. Using real-world benign traffic from the CICIDS2017 dataset, we evaluate multiple models against a human-generated ground truth, comparing their performance on raw textual traffic dumps and protocol-filtered representations. Model responses are assessed through an automated LLM- as-a-judge pipeline, validated against human evaluations. The results demonstrate that protocol filtering significantly reduces prompt overhead while improving answer accuracy by reducing incorrect responses. These findings highlight the importance of protocol-aware traffic filtering as an effective optimization for improving the efficiency and reliability of LLM-assisted network management.

## Errpedia: Is Your Exact $k$-NN Vector Search Really Exact?

DevFeed: [Errpedia: Is Your Exact $k$-NN Vector Search Really Exact?](<https://devfeed.tech/articles/errpedia-is-your-exact-k-nn-vector-search-really-exact-140631.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22474>)

Author: Yinuo Deng, Ying Zhang, Hanchen Wang, Wenjie Zhang

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [floating-point](<https://devfeed.tech/topics/floating-point.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [test data](<https://devfeed.tech/topics/test-data.md>), [ann](<https://devfeed.tech/topics/ann.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [database](<https://devfeed.tech/tags/database.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [error](<https://devfeed.tech/tags/error.md>), [floating-point](<https://devfeed.tech/tags/floating-point.md>), [vector-search](<https://devfeed.tech/tags/vector-search.md>)

### AI overview

Errpedia tests numerical correctness in exact k-nearest-neighbor vector search, where floating-point rounding and overflow can produce mathematically wrong results. It contains 1,616 queries across 21 error categories, with ground truth verified using exact rational arithmetic. In tests of 20 library and database configurations, every configuration returned an incorrect top-10 result set on at least one applicable query. Reported failures include misranking nearly equidistant vectors, one library omitting all 10 exact matches for a query, and several systems returning fewer than k neighbors. The authors present Errpedia as a reproducible benchmark for checking implementations.

### Source excerpt

arXiv:2611.22474v1 Announce Type: new Abstract: Exact $k$-nearest-neighbor ($k$-NN) vector search is a common feature of vector libraries and database systems. It serves applications that require exact results and provides the ground truth for evaluating approximate search. However, floating-point rounding and overflow can cause an exact search to return mathematically incorrect top-$k$ result sets. These errors can go unnoticed and affect downstream tasks that rely on exact results. We present Errpedia, a purpose-built dataset for testing the numerical correctness of exact vector search. Errpedia contains 1,616 queries across 21 error categories, with ground truth independently verified using exact rational arithmetic. We evaluate 20 widely used library and database configurations and find that every tested configuration returns mathematically incorrect top-10 result sets on at least 1 applicable query. Failures include misranking nearly equidistant vectors and returning incomplete results: 1 library omits all 10 exact matches to a query, and several systems return fewer than $k$ neighbors. We release Errpedia as a reproducible benchmark for developers and researchers to test their implementations. The dataset is publicly available at https://github.com/dynos01/errpedia.

## Beyond Reconstruction: A Unified Framework for Trustworthy and Explainable EEG Artifact Removal Evaluation

DevFeed: [Beyond Reconstruction: A Unified Framework for Trustworthy and Explainable EEG Artifact Removal Evaluation](<https://devfeed.tech/articles/beyond-reconstruction-a-unified-framework-for-trustworthy-and-explainable-eeg-artifact-removal-evaluation-139906.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21781>)

Author: Phat Lam, Hung Nguyen, Lam Pham, Huy Phan, Dat Tran, Ian McLoughlin, Roozbeh Razavi-Far, Hung Cao

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>)

Tags: [artifacts](<https://devfeed.tech/tags/artifacts.md>), [band](<https://devfeed.tech/tags/band.md>), [bci](<https://devfeed.tech/tags/bci.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [eeg](<https://devfeed.tech/tags/eeg.md>), [eess-sp](<https://devfeed.tech/tags/eess-sp.md>), [eog](<https://devfeed.tech/tags/eog.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [framework](<https://devfeed.tech/tags/framework.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [motion](<https://devfeed.tech/tags/motion.md>), [neural](<https://devfeed.tech/tags/neural.md>), [noise](<https://devfeed.tech/tags/noise.md>), [performance](<https://devfeed.tech/tags/performance.md>), [physionet](<https://devfeed.tech/tags/physionet.md>), [power](<https://devfeed.tech/tags/power.md>), [quality](<https://devfeed.tech/tags/quality.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [rf](<https://devfeed.tech/tags/rf.md>), [shift](<https://devfeed.tech/tags/shift.md>)

### AI overview

TriX-EEG evaluates EEG artifact removal across three complementary levels: Signal Validity measures reconstruction quality and neural band-power preservation; Removal Faithfulness examines the removed component for artifact error and residual artifact or neural leakage; Explanation Reliability assesses temporal alignment, causal relevance, and stability. Tests of nine published denoisers across three benchmarks covering EOG, EMG, and motion artifacts suggest reconstruction metrics alone cannot show whether artifacts remain or neural activity was suppressed. The authors report broadly consistent rankings on independent EEG/EOG and motor-imagery datasets under dataset shift, with higher-ranked denoisers generally preserving more task-relevant information or causing less downstream performance degradation.

### Source excerpt

arXiv:2611.21781v1 Announce Type: cross Abstract: Deep-learning methods have achieved strong performance in EEG artifact removal, but evaluation remains dominated by reconstruction metrics such as error, correlation, and signal-to-noise ratio. Although these metrics assess similarity to clean EEG, they do not establish whether models effectively suppress artifacts, preserve neural activity, or provide reliable explanations. To address these limitations, we propose TriX-EEG, a three-level framework for trustworthy and explainable EEG artifact removal. Level 1, Signal Validity (SV), evaluates reconstruction quality and neural band-power preservation. Level 2, Removal Faithfulness (RF), assesses the removed component through artifact error and residual artifact/neural leakage. Level 3, Explanation Reliability (ER), examines model behavior through temporal alignment, causal relevance, and stability. We evaluate TriX-EEG using nine published neural denoisers across three public benchmarks derived from EEGDenoiseNet and the PhysioNet Motion Artifact Contaminated EEG Dataset, covering EOG, EMG, and motion artifacts. Results show that reconstruction metrics alone are insufficient for trustworthy denoiser assessment: RF complements it by revealing whether errors stem from residual artifact or suppressed neural activity, while ER captures distinct aspects of explanation behavior, supporting joint evaluation of three aspects within a unified framework rather than reliance on any single factor. External validation on an independent EEG/EOG dataset and the BCI Competition IV-2a motor-imagery dataset further demonstrates broadly consistent rankings under dataset shift. Higher-ranked denoisers generally preserve more task-relevant information or cause less downstream performance degradation. Overall, TriX-EEG provides a comprehensive basis for EEG denoiser assessment, encouraging models that are not only accurate but also faithful and explainable.

## DDSL: Iterative Self-Labeling for Financial Market States

DevFeed: [DDSL: Iterative Self-Labeling for Financial Market States](<https://devfeed.tech/articles/ddsl-iterative-self-labeling-for-financial-market-states-141826.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22958>)

Author: Xianbin Gu, Qingyue Zhu, Yuhong Geng

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [Finance](<https://devfeed.tech/topics/finance.md>), [iteration](<https://devfeed.tech/topics/iteration.md>)

Tags: [bull](<https://devfeed.tech/tags/bull.md>), [classifier](<https://devfeed.tech/tags/classifier.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [dynamic-programming](<https://devfeed.tech/tags/dynamic-programming.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [event](<https://devfeed.tech/tags/event.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [financial](<https://devfeed.tech/tags/financial.md>), [financial-market](<https://devfeed.tech/tags/financial-market.md>), [gru](<https://devfeed.tech/tags/gru.md>), [labeling](<https://devfeed.tech/tags/labeling.md>), [state](<https://devfeed.tech/tags/state.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [validation](<https://devfeed.tech/tags/validation.md>), [weight](<https://devfeed.tech/tags/weight.md>)

### AI overview

DDSL iteratively refines financial market-state labels by coupling a price- and grammar-guided Viterbi labeler with a Transformer classifier through model feedback. Its nine-state representation encodes current and future trends while retaining structured price emissions and temporal constraints. Across SSE50, CSI300, CSI500, and NASDAQ100, it raises held-out joint-state macro F1 by 78%-96% over single-pass dynamic-programming labeling; an independent GRU shows the same overall gain, with stable label entropy and class coverage. Learnability depends on iteration and feedback weight: gains diminish as label systems converge, while stronger feedback widens the Transformer-GRU performance gap. Greater learnability also accompanies declining segment-level price-feature separation after an initial improvement. Event-study validation is partial: refinement strengthens a conditional BEAR-to-BULL reversal signal on the three A-share datasets, but the effect has the opposite sign on NASDAQ100.

### Source excerpt

arXiv:2611.22958v1 Announce Type: new Abstract: Financial market states lack a unique ground truth and must therefore be constructed from price data. Conventional labeling methods typically define states independently of the downstream classifier, leaving unresolved whether the resulting labels can be recovered from information available at prediction time. We propose DDSL, an iterative self-labeling framework that couples a price- and grammar-guided Viterbi labeler with a Transformer classifier through model feedback. DDSL constructs a nine-state representation that jointly encodes current and future market trends while retaining structured price emissions and temporal constraints throughout refinement. Experiments on SSE50, CSI300, CSI500, and NASDAQ100 show that DDSL increases held-out joint-state macro F1 by 78%-96% over single-pass dynamic-programming labeling. An independent GRU classifier that never participates in the labeling loop reproduces the same trend, while label entropy and class coverage remain stable, indicating that the gain transfers across architectures and is not caused by label collapse. We further show that learnability is controlled by both iteration and the model-feedback weight $\beta_{\mathrm{model}}$. Iterative gains diminish as successive label systems converge, whereas stronger feedback increasingly widens the performance gap between the feedback Transformer and the independent GRU. Increased learnability is accompanied by declining segment-level price-feature separation after an initial improvement. Finally, a non-anticipative event study provides partial external validation: refinement strengthens a conditional BEAR$\rightarrow$BULL reversal signal on the three A-share datasets, while the corresponding effect has the opposite sign on NASDAQ100. These results position DDSL as a structured framework for constructing market-state labels whose learnability can be explicitly controlled and evaluated.

## WHBench: Evaluating Frontier LLMs with Expert-in-the-Loop Validation on Women's Health Topics

DevFeed: [WHBench: Evaluating Frontier LLMs with Expert-in-the-Loop Validation on Women's Health Topics](<https://devfeed.tech/articles/whbench-evaluating-frontier-llms-with-expert-in-the-loop-validation-on-women-s-health-topics-139763.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2604.00024>)

Author: Sneha Maurya, Spandana Govindgari, Girish Kumar

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [equity](<https://devfeed.tech/tags/equity.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [health](<https://devfeed.tech/tags/health.md>), [llms](<https://devfeed.tech/tags/llms.md>), [oversight](<https://devfeed.tech/tags/oversight.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [safety](<https://devfeed.tech/tags/safety.md>), [women-s-health](<https://devfeed.tech/tags/women-s-health.md>)

### AI overview

WHBench evaluates 22 language models on 47 expert-crafted scenarios covering 10 women's health topics. Its 23-criterion rubric assesses clinical accuracy, completeness, safety, communication, instruction following, equity, uncertainty handling, and guideline adherence, with safety-weighted penalties and server-side score recalculation. Across 3,102 attempted responses, 3,100 were scored; no model averaged above 75%, and the top score was 72.1%. Top models still had low fully correct rates, with substantial variation in harm rates. Agreement between raters was moderate on individual response labels but high for model rankings. The results support WHBench as a comparative evaluation tool while underscoring the need for expert oversight in clinical deployment.

### Source excerpt

arXiv:2604.00024v3 Announce Type: replace-cross Abstract: Large language models are increasingly used for medical guidance, but women's health remains under-evaluated in benchmark design. We present the Women's Health Benchmark (WHBench), a targeted evaluation suite of 47 expert-crafted scenarios across 10 women's health topics, designed to expose clinically meaningful failure modes including outdated guidelines, unsafe omissions, dosing errors, and equity-related blind spots. We evaluate 22 models using a 23-criterion rubric spanning clinical accuracy, completeness, safety, communication quality, instruction following, equity, uncertainty handling, and guideline adherence, with safety-weighted penalties and server-side score recalculation. Across 3,102 attempted responses (3,100 scored), no model mean performance exceeds 75 percent; the best model reaches 72.1 percent. Even top models show low fully correct rates and substantial variation in harm rates. Inter-rater reliability is moderate at the response label level but high for model ranking, supporting WHBench utility for comparative system evaluation while highlighting the need for expert oversight in clinical deployment. WHBench provides a public, failure-mode-aware benchmark to track safer and more equitable progress in womens health AI.

## ForceBody: Force-Paired Parametric Body Motion with Torque Uncertainty

DevFeed: [ForceBody: Force-Paired Parametric Body Motion with Torque Uncertainty](<https://devfeed.tech/articles/forcebody-force-paired-parametric-body-motion-with-torque-uncertainty-140076.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22769>)

Author: Joonwoo Kwon, Yufei Zhang, Xiaoming Liu, Zijun Cui

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [animations](<https://devfeed.tech/topics/animations.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>)

Tags: [animation](<https://devfeed.tech/tags/animation.md>), [annotation](<https://devfeed.tech/tags/annotation.md>), [annotations](<https://devfeed.tech/tags/annotations.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [body](<https://devfeed.tech/tags/body.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [finetuning](<https://devfeed.tech/tags/finetuning.md>), [kinematics](<https://devfeed.tech/tags/kinematics.md>), [learning](<https://devfeed.tech/tags/learning.md>), [monte-carlo](<https://devfeed.tech/tags/monte-carlo.md>), [motion](<https://devfeed.tech/tags/motion.md>), [pairs](<https://devfeed.tech/tags/pairs.md>), [physics](<https://devfeed.tech/tags/physics.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [reaction](<https://devfeed.tech/tags/reaction.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [robotics](<https://devfeed.tech/tags/robotics.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>)

### AI overview

ForceBody pairs the SKEL parametric body model with ground-reaction forces and inverse-dynamics joint torques across 10,386 trials (9.7 million frames, 26.9 hours) from 140 subjects. Each torque label includes per-frame, per-joint uncertainty estimated by Monte Carlo sampling through the inverse-dynamics pipeline, addressing the sensitivity of torque estimates to input kinematics. The authors benchmark MLP, Conv1D, LSTM, GRU, Mamba2, and Transformer models for force and torque prediction from SKEL motion. In one reported use, uncertainty-weighted Transformer fine-tuning reduces joint-torque estimation MAE by 5%.

### Source excerpt

arXiv:2611.22769v1 Announce Type: cross Abstract: Forces and torques drive human motion, central to biomechanics, robotics, and physics-aware animation, yet they are often absent from learning-based motion pipelines for body reconstruction and generation. Some motion datasets provide force annotations, but the paired motion sequences are incompatible with the parametric body models used in modern learning pipelines. In addition, existing datasets typically obtain joint torques through inverse dynamics, which is inherently sensitive to the quality of the input kinematics; however, they do not quantify the reliability of these torque estimates. We close this gap with ForceBody, the first dataset to bring ground truth force supervision into a parametric body representation, shipped with per-sample torque uncertainty. ForceBody pairs the SKEL body model with ground reaction forces and inverse-dynamics joint torques across 10,386 motion trials (9.7M frames, 26.9 hours) from 140 subjects. Each torque label is paired with a per-frame, per-joint uncertainty annotation obtained by Monte Carlo sampling through the inverse dynamics pipeline. We benchmark six architectures (MLP, Conv1D, LSTM, GRU, Mamba2, Transformer) on ground reaction force and joint torque prediction from SKEL motion. As one example use of the released uncertainty, finetuning a Transformer with per-sample uncertainty weighting reduces the MAE of joint torque estimation by 5%. Code and data are at https://github.com/DominoAI-Lab/ForceBody-NeurIPS-ED-2026.

## Generalized Prediction Set with Bandit Feedback

DevFeed: [Generalized Prediction Set with Bandit Feedback](<https://devfeed.tech/articles/generalized-prediction-set-with-bandit-feedback-139039.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20387>)

Author: Zhou Wang, Xingye Qiao

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>)

Tags: [bandit](<https://devfeed.tech/tags/bandit.md>), [class](<https://devfeed.tech/tags/class.md>), [classes](<https://devfeed.tech/tags/classes.md>), [control](<https://devfeed.tech/tags/control.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [detection](<https://devfeed.tech/tags/detection.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [errors](<https://devfeed.tech/tags/errors.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [function](<https://devfeed.tech/tags/function.md>), [high-stakes-environments](<https://devfeed.tech/tags/high-stakes-environments.md>), [learning](<https://devfeed.tech/tags/learning.md>), [mechanism](<https://devfeed.tech/tags/mechanism.md>), [multiple](<https://devfeed.tech/tags/multiple.md>), [objectives](<https://devfeed.tech/tags/objectives.md>), [online](<https://devfeed.tech/tags/online.md>), [online-learning](<https://devfeed.tech/tags/online-learning.md>), [ood](<https://devfeed.tech/tags/ood.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [set](<https://devfeed.tech/tags/set.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

BanditGPS combines set-valued prediction with out-of-distribution (OOD) detection in online learning with bandit feedback, where an incorrect action does not reveal the true class label. It estimates the ground-truth labels and uses a loss that trades off OOD detection performance against prediction-set size. The method aims to produce small, informative sets while controlling recall for each normal class to meet prescribed requirements. The authors report a regret convergence rate of Õ(T⁻¹ᐟ²) and empirical results showing recall control alongside promising OOD detection and prediction informativeness. The supplied text is an abstract and does not provide experimental details.

### Source excerpt

arXiv:2611.20387v1 Announce Type: cross Abstract: In high-stakes environments where uncertainties abound, set-valued prediction offers a cautious and robust mechanism by presenting multiple potential labels as the prediction for each test instance to mitigate the potential risk associated with prediction errors. Yet, integrating this paradigm with out-of-distribution (OOD) detection remains scarcely explored in such settings as online learning with bandit feedback. The bandit feedback mechanism informs the learner about the correctness of the pulled arm/action instead of the explicit ground truth label, leaving the true class label unknown when an incorrect action is taken. To address this challenge, we introduce BanditGPS which conducts set-valued prediction with OOD detection in the bandit feedback setting, using an estimation to the ground truth of class labels. BanditGPS achieves three objectives: render small/informative prediction sets, enhance the OOD detection performance, and control the recall for all normal classes to meet prescribed requirements. Our approach is characterized by the loss function, which trades off between high OOD detection and small prediction sets. Theoretically, we prove that the convergence rate of the regret is $\tilde{\mathcal{O}}(T^{-1/2})$. The empirical results further show that BanditGPS effectively controls the recalls with promising performances on OOD detection and informative prediction.

## Target-Free Autocalibration of Stereo PIV Camera Systems via Persistent-State Motion Representations

DevFeed: [Target-Free Autocalibration of Stereo PIV Camera Systems via Persistent-State Motion Representations](<https://devfeed.tech/articles/target-free-autocalibration-of-stereo-piv-camera-systems-via-persistent-state-motion-representations-137336.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18741>)

Author: Nabil Madali

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [camera](<https://devfeed.tech/tags/camera.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [data](<https://devfeed.tech/tags/data.md>), [model](<https://devfeed.tech/tags/model.md>), [motion](<https://devfeed.tech/tags/motion.md>), [networks](<https://devfeed.tech/tags/networks.md>), [optical](<https://devfeed.tech/tags/optical.md>), [particles](<https://devfeed.tech/tags/particles.md>), [recurrent](<https://devfeed.tech/tags/recurrent.md>), [sequences](<https://devfeed.tech/tags/sequences.md>), [state](<https://devfeed.tech/tags/state.md>), [stereo](<https://devfeed.tech/tags/stereo.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [systems](<https://devfeed.tech/tags/systems.md>), [target](<https://devfeed.tech/tags/target.md>), [train](<https://devfeed.tech/tags/train.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [velocity](<https://devfeed.tech/tags/velocity.md>), [view](<https://devfeed.tech/tags/view.md>)

### AI overview

The method calibrates stereo PIV camera rigs from particle image sequences without inserting a physical target, addressing the need to repeat target-based extrinsic calibration when a rig moves. Since tracer particles are visually indistinguishable, recurrent transformers accumulate motion in persistent state; the system decodes optical flow, regresses bidirectional cross-view correspondences, and recovers relative pose with a five-point RANSAC solver. Trained on a synthetic dataset with ground-truth poses, it is evaluated on standard PIV flow benchmarks and 200 real rig placements. The abstract reports lower flow error than existing PIV networks, more accurate correspondences than COTR, and higher pose accuracy than detector-matcher pipelines on synthetic and real data.

### Source excerpt

arXiv:2611.18741v1 Announce Type: cross Abstract: Stereo Particle Image Velocimetry (PIV) recovers three-component velocity fields by triangulating tracer particles observed by synchronized cameras. It requires an accurate extrinsic calibration, which is still obtained by inserting a physical target into the flow and must be repeated whenever the rig is moved. We present a target-free method that calibrates a stereo PIV rig directly from particle image sequences. Because tracer particles are visually indistinguishable, we match motion rather than appearance: building on CUT3R, each camera stream is processed by a recurrent transformer whose persistent state accumulates particle motion over time. The state is decoded into per-camera optical flow and queried by a cross-view decoder that regresses bidirectional point correspondences, from which the relative pose is recovered with a five-point RANSAC solver. We train on a new synthetic stereo PIV dataset with ground-truth poses and evaluate on standard PIV optical flow benchmarks and on 200 real rig placements. Our model achieves lower flow error than existing PIV networks, more accurate cross-view correspondences than COTR, and higher pose accuracy than the strongest detector-matcher pipelines on both synthetic and real data, without any calibration target.

## Who Writes the Codebook? An Eight-Step Framework for Auditing LLM-Generated Category Systems

DevFeed: [Who Writes the Codebook? An Eight-Step Framework for Auditing LLM-Generated Category Systems](<https://devfeed.tech/articles/who-writes-the-codebook-an-eight-step-framework-for-auditing-llm-generated-category-systems-137507.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19872>)

Author: Thilo Tamme, Anton Hantel, Jakob Scheppers

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [text-classification](<https://devfeed.tech/topics/text-classification.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>)

Tags: [auditing](<https://devfeed.tech/tags/auditing.md>), [categories](<https://devfeed.tech/tags/categories.md>), [coding](<https://devfeed.tech/tags/coding.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [framework](<https://devfeed.tech/tags/framework.md>), [grounded](<https://devfeed.tech/tags/grounded.md>), [interviews](<https://devfeed.tech/tags/interviews.md>), [llm](<https://devfeed.tech/tags/llm.md>), [reliability](<https://devfeed.tech/tags/reliability.md>)

### AI overview

The Qualitative Variance Audit Framework tests how LLMs shape qualitative coding by comparing codebook creation, repeated application, and code distributions under controlled inputs. It uses model-owned and shared-reference codebooks to separate variation across the full pipeline from differences applying a common system. In an application to 668 organisational interviews and 13,166 text units, eight models from five vendors produced 48 codebooks and 270 coding runs. Category structures differed markedly: mean within-model reliability ranged from .76 to .92, while between-model agreement on the shared codebook was about .69. The framework reveals model dependencies before interpretation, but measures reliability and representational consequences--not interpretive validity or theory quality.

### Source excerpt

arXiv:2611.19872v1 Announce Type: new Abstract: Large language models (LLMs) increasingly create the categories through which qualitative evidence is interpreted. Existing coding benchmarks often hold these categories fixed. We propose the Qualitative Variance Audit Framework, an eight-step method that compares codebook creation, repeated application and code distributions under controlled inputs. It combines model-owned and shared-reference codebooks to distinguish variation in the overall pipeline from application differences within a common category system. An application to 668 organisational interviews, 13,166 text units and eight models from five vendors yields 48 codebooks and 270 coding runs. Models create markedly different category structures; mean within-model reliability ranges from .76 to .92, whereas between-model agreement on the shared codebook is about .69. The framework complements LLM-assisted grounded theory by exposing model dependencies before substantive interpretation. It assesses reliability and representational consequences, not interpretive validity or the quality of resulting theory.

## The Mask Is Not the Object: Volumetric Supervision for 3D Gaussian Segmentation

DevFeed: [The Mask Is Not the Object: Volumetric Supervision for 3D Gaussian Segmentation](<https://devfeed.tech/articles/the-mask-is-not-the-object-volumetric-supervision-for-3d-gaussian-segmentation-137691.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21383>)

Author: Jeonghwan Cho, Minsu Kim, Junyoung Hong, Seon Joo Kim

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [gaussian splatting](<https://devfeed.tech/topics/gaussian-splatting.md>), [3D](<https://devfeed.tech/topics/3d.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [3d-gaussian-splatting](<https://devfeed.tech/tags/3d-gaussian-splatting.md>), [alpha](<https://devfeed.tech/tags/alpha.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [changes](<https://devfeed.tech/tags/changes.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [completeness](<https://devfeed.tech/tags/completeness.md>), [contribution](<https://devfeed.tech/tags/contribution.md>), [contributors](<https://devfeed.tech/tags/contributors.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [error](<https://devfeed.tech/tags/error.md>), [feature](<https://devfeed.tech/tags/feature.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [gaussian-splatting](<https://devfeed.tech/tags/gaussian-splatting.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [learning](<https://devfeed.tech/tags/learning.md>), [mask](<https://devfeed.tech/tags/mask.md>), [object](<https://devfeed.tech/tags/object.md>), [supervision](<https://devfeed.tech/tags/supervision.md>)

### AI overview

VODA targets incomplete 3D Gaussian selections that can still produce accurate rendered masks: alpha-blended feature supervision favors dominant contributors and leaves occluded or low-contribution Gaussians weakly supervised. Its plug-and-play objective uses local depth structure to group per-pixel contributors and directly supervise all Gaussians in selected depth clusters. The authors introduce Removal-3D, a synthetic object-removal benchmark, and Removal Depth Error (RDE), which compares predicted and ground-truth depth changes. Across four rendering-based baselines, VODA improves volumetric segmentation on Removal-3D while maintaining or improving performance on LERF and ScanNet. The abstract reports that simple gradient reweighting and spatial propagation do not achieve comparable gains.

### Source excerpt

arXiv:2611.21383v1 Announce Type: cross Abstract: 3D Gaussian Splatting enables object-level operations on Gaussian primitives, yet existing segmentation methods are largely trained and evaluated through rendered views. This mismatch can produce accurate rendered masks from incomplete 3D selections since alpha-blended feature supervision concentrates learning on dominant contributors while leaving occluded or low-contribution Gaussians weakly supervised. We propose VODA, a plug-and-play supervision objective that exploits local depth structure to group per-pixel contributors and directly supervises all Gaussians in selected depth clusters, bypassing contribution-weighted gradient delivery. To evaluate volumetric completeness, we introduce Removal-3D, a synthetic object-removal benchmark, and the Removal Depth Error (RDE), a metric comparing predicted and ground-truth depth changes. Across four rendering-based baselines, VODA consistently improves volumetric segmentation on Removal-3D while maintaining or improving performance on LERF and ScanNet. Further analysis shows that VODA recovers lower-contribution Gaussians, while simple gradient reweighting and spatial propagation fail to reproduce comparable gains.

## Benchmark Factory: Agentic Benchmark Creation via Dataset Curation and Label-Free Quality Anchors

DevFeed: [Benchmark Factory: Agentic Benchmark Creation via Dataset Curation and Label-Free Quality Anchors](<https://devfeed.tech/articles/benchmark-factory-agentic-benchmark-creation-via-dataset-curation-and-label-free-quality-anchors-137358.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18890>)

Author: Shervin Shahidi, James Poon, Hang Xie, Kenneth O. Xu, Robin H{\\aa}kanson, Vikram Nair, Jens Pillgram-Larsen, Nachiappan Nagappan

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [Agentic AI](<https://devfeed.tech/topics/what-is-agentic-ai.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>)

Tags: [1-4](<https://devfeed.tech/tags/1-4.md>), [3](<https://devfeed.tech/tags/3.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-benchmark](<https://devfeed.tech/tags/agentic-benchmark.md>), [agents](<https://devfeed.tech/tags/agents.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [autonomous](<https://devfeed.tech/tags/autonomous.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [build](<https://devfeed.tech/tags/build.md>), [but](<https://devfeed.tech/tags/but.md>), [can](<https://devfeed.tech/tags/can.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [curation](<https://devfeed.tech/tags/curation.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [factory](<https://devfeed.tech/tags/factory.md>), [free](<https://devfeed.tech/tags/free.md>), [quality](<https://devfeed.tech/tags/quality.md>)

### AI overview

Benchmark Factory builds coding-agent benchmarks from landed production changes through an LLM-driven loop: it writes sourcing queries, tests candidates with models of known relative strength, diagnoses failures, and revises queries, prompts, and configuration until candidates pass label-free quality anchors, with human checkpoints. In benchmarks created for five developer cohorts, raters scored a 5% sample 2.3/3 for task quality and 2.6/3 for grader verdicts. On configurations not used during construction, results ranked higher reasoning effort above lower effort for Claude Opus 4.7 (60.6% vs. 54.4% passed) and GPT-5.5 above GPT-5.4 (65.8% vs. 60.5%); 9 of 10 cohort comparisons agreed. Trivial benchmarks and five broken graders failed the anchors, while shuffled model labels yielded candidate passes in at most 1.4% of simulations. The authors describe the anchors as necessary but insufficient for validity, intended to focus human review on issues the loop cannot repair.

### Source excerpt

arXiv:2611.18890v1 Announce Type: cross Abstract: Teams deploying coding agents need benchmarks built from their own code, but curating and validating them by hand is resource intensive, and an unvalidated benchmark can mislead silently. We present Benchmark Factory, an autonomous meta-agent loop with human checkpoints: an LLM agent writes sourcing queries over landed production code changes, runs models of known relative strength on each candidate, diagnoses why a candidate fails, and repairs it by editing its queries, prompts, and configuration until it clears quality anchors that need no per-task human labels. We used Benchmark Factory to build benchmarks for five developer cohorts at our organization. Engineering raters scored a 5% sample 2.3 for task quality and 2.6 for grader verdicts on a 1-to-3 scale. On configurations never used to build them, the benchmarks ranked Claude Opus~4.7 at high reasoning effort above low effort (60.6% vs. 54.4% of tasks passed, $p=0.002$) and GPT-5.5 above GPT-5.4 (65.8% vs. 60.5%, $p=0.004$); 9 of the 10 per-cohort comparisons pointed the same way. A deliberately trivial benchmark failed the anchors; five broken graders failed them, even under a relaxed rule fixed in advance that the real grader passes; and with model labels shuffled so that no true ordering existed, random redraws of a candidate passed in at most 1.4% of simulations. The anchors are necessary, not sufficient, conditions for a valid benchmark, designed to focus human review on what the loop cannot repair.

[Next page](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md?cursor=WyIyMDI2LTExLTE3VDA1OjAwOjAwKzAwOjAwIiwgIjdjM2I4NDk1LTM1MzItNDc5OC1iMGZhLTMyNTYyY2ZkNTg1MSJd>)