# Benchmark

Benchmark is a computer performance test or set of tests designed to compare one computer system with others.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Clairic Hit 98% Accuracy on 72.8% of Questions via Confidence Scores

DevFeed: [Clairic Hit 98% Accuracy on 72.8% of Questions via Confidence Scores](<https://devfeed.tech/articles/clairic-hit-98-accuracy-on-72-8-of-questions-via-confidence-scores-146441.md>)

Original publisher: [Read original article](<https://towardsdatascience.com/clairic-hit-98-accuracy-on-728-of-questions-via-confidence-scores/>)

Author: Mariya Mansurova

Published: 2026-11-20T15:00:00Z

Content type: tutorial

Language: en

Sources: [Towards Data Science](<https://devfeed.tech/sources/towards-data-science.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Data Science](<https://devfeed.tech/topics/data-science.md>), [ai-experiments](<https://devfeed.tech/topics/ai-experiments.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [analysts](<https://devfeed.tech/tags/analysts.md>), [arithmetic](<https://devfeed.tech/tags/arithmetic.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [autocomplete](<https://devfeed.tech/tags/autocomplete.md>), [based](<https://devfeed.tech/tags/based.md>), [bike](<https://devfeed.tech/tags/bike.md>), [charts](<https://devfeed.tech/tags/charts.md>), [chatgpt](<https://devfeed.tech/tags/chatgpt.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cloudflare](<https://devfeed.tech/tags/cloudflare.md>), [complex](<https://devfeed.tech/tags/complex.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [confidence-interval](<https://devfeed.tech/tags/confidence-interval.md>), [control](<https://devfeed.tech/tags/control.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [decision-models](<https://devfeed.tech/tags/decision-models.md>), [easy](<https://devfeed.tech/tags/easy.md>), [editor-s-picks](<https://devfeed.tech/tags/editor-s-picks.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [jev](<https://devfeed.tech/tags/jev.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-applications](<https://devfeed.tech/tags/llm-applications.md>), [luna](<https://devfeed.tech/tags/luna.md>), [model](<https://devfeed.tech/tags/model.md>), [science](<https://devfeed.tech/tags/science.md>), [system-one](<https://devfeed.tech/tags/system-one.md>)

### AI overview

Clairic is a fast, low-cost extraction model that accepts text and images and can return typed values with confidence scores. In tests against Luna, Clairic was faster but less accurate overall, especially on calculations; examples also included confidently incorrect extractions. It handled some chart-based extraction and returned null for missing revenue. On 1,360 questions across four languages, a 0.9 confidence cutoff yielded 98% accuracy while Clairic answered 72.8% of questions. The author suggests using it for high-confidence extraction and routing the rest to a larger model, with validation because confidence is imperfect. The tests did not establish a reason to prefer it for forecasting over traditional methods.

### Source excerpt

With a 0.9 confidence cutoff, Cloudflare's System One model handles the easy cases and routes the rest to a bigger, slower LLM like Luna. The post Clairic Hit 98% Accuracy on 72.8% of Questions via Confidence Scores appeared first on Towards Data Science.

## ClosureBench: A Benchmark for Reasoning under Closure Assumptions

DevFeed: [ClosureBench: A Benchmark for Reasoning under Closure Assumptions](<https://devfeed.tech/articles/closurebench-a-benchmark-for-reasoning-under-closure-assumptions-145657.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26335>)

Author: Francesco Belardinelli, Flavio Corradini, Matteo Leonesi

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [choice](<https://devfeed.tech/tags/choice.md>), [closure](<https://devfeed.tech/tags/closure.md>), [computer-science](<https://devfeed.tech/tags/computer-science.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [extensions](<https://devfeed.tech/tags/extensions.md>), [information](<https://devfeed.tech/tags/information.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [owa](<https://devfeed.tech/tags/owa.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [records](<https://devfeed.tech/tags/records.md>), [rules](<https://devfeed.tech/tags/rules.md>), [scope](<https://devfeed.tech/tags/scope.md>), [source](<https://devfeed.tech/tags/source.md>), [tests](<https://devfeed.tech/tags/tests.md>), [updates](<https://devfeed.tech/tags/updates.md>)

### AI overview

ClosureBench evaluates whether LLM agents apply stated assumptions about missing information: open-world (OWA), closed-world (CWA), or locally closed-world (LCWA). Each base scenario is tested in all three variants, changing only how missing information is treated; a base scenario counts as correct only when answers are correct across every variant. The benchmark also includes extensions for action choice, source scope, and updates. Across 960 instances, overall accuracy ranged from 71.18% to 99.86%, while accuracy on cases where the correct label changes with the assumption ranged from 18.23% to 99.31%. The abstract reports notable errors under LCWA, including agents answering "unknown" when missing information should be treated as false. These results suggest overall accuracy can obscure failures to follow the specified closure assumption.

### Source excerpt

arXiv:2611.26335v1 Announce Type: new Abstract: AI agents built on large language models (LLMs) often answer questions from incomplete records. Missing information might be interpreted as unknown or false, depending on the completeness of the source. Current evaluations fail to distinguish these two aspects, as they are usually interwoven with other reasoning demands. We present ClosureBench, a benchmark that tests whether LLMs follow the stated rules for dealing with missing information. Every base scenario is tested under OWA, CWA, and LCWA, and only the treatment of missing information changes between variants. ClosureBench provides a verified base benchmark as well as agent extensions for action choice, source scope, and updates. A model is considered correct on a base scenario only if it provides correct answers in all of its closure variants. Across 960 instances, overall accuracy can mask substantial closure mistakes. Models achieve overall accuracy from 71.18% to 99.86% across replicated runs, but their core switch accuracy, measured on the scenarios whose gold label actually changes with the contract, varies between 18.23% and 99.31%. Large discrepancies emerge under LCWA. For example, some agents choose unknown despite the prompt indicating that missing information should be treated as false. Our findings demonstrate that evaluations of AI agents should not only measure whether agents can reason over existing facts but also whether they use the appropriate assumption when information is absent.

## AutoRAGBench: A Benchmark for LLM Agents Automating RAG System Development

DevFeed: [AutoRAGBench: A Benchmark for LLM Agents Automating RAG System Development](<https://devfeed.tech/articles/autoragbench-a-benchmark-for-llm-agents-automating-rag-system-development-145541.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25582>)

Author: Hang Zhang, Yuzhi Pan, Ziyao Zhong, Chaokun Wang, Shuo Cao, Zeyu Huang, Yue Xue, Fang Niu, Bofan Xie, Guanchen Ge, Zecheng Li, Leqi Zheng, Ziyang Liu, Wei Wang, Miao Gu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [rag](<https://devfeed.tech/tags/rag.md>), [system-development](<https://devfeed.tech/tags/system-development.md>), [systems](<https://devfeed.tech/tags/systems.md>)

### AI overview

AutoRAGBench evaluates whether LLM agents can build and improve retrieval-augmented generation systems across nine development tasks. Agents worked in an open code space without starter code or prescribed strategies, with six hours and one H800 GPU per task; they analyzed data, implemented pipelines, and refined them using experimental feedback. Across nine configurations, GPT-6.1 Sol with Codex CLI had the highest average score (56.1), compared with 44.4 for the composite Human-SOTA baseline. Starting from human-designed systems on seven tasks, it improved scores by an average of 16.6 points. These results support the benchmark's finding that agents can automate effective RAG development under the tested conditions.

### Source excerpt

arXiv:2611.25582v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) is a well-established paradigm for augmenting large language models (LLMs) with information retrieved from external corpora. However, developing effective RAG systems requires substantial engineering effort and iterative experimentation, as effective design choices vary across tasks and domains. With rapid advances in LLM agent capabilities, we ask how well these agents can autonomously build RAG systems and how effectively they can improve them through iteration. In this paper, we propose AutoRAGBench, the first benchmark to evaluate how well LLM agents develop RAG systems from scratch in an open code space. It comprises nine RAG system development tasks. For each task, agents build RAG systems from scratch without starter code or prescribed strategies, under a six-hour budget on a single H800 GPU. They analyze task data, implement RAG pipelines, and iteratively refine their designs based on experimental feedback. We evaluate nine agent configurations and analyze both the effectiveness of the resulting RAG systems and their development trajectories. GPT-6.1 Sol with Codex CLI achieves the highest average score of 56.1. The composite Human-SOTA baseline averages 44.4. These results show that LLM agents can automate effective RAG system development. When starting from a human-designed system on each of seven tasks, GPT-6.1 Sol with Codex CLI improves the score by 16.6 points on average.

## MultiWorldBench: Do Independently Controlled Views Describe One Shared World?

DevFeed: [MultiWorldBench: Do Independently Controlled Views Describe One Shared World?](<https://devfeed.tech/articles/multiworldbench-do-independently-controlled-views-describe-one-shared-world-145396.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.11723>)

Author: Zhangbo Xu, Ruoxi Zhang, Rui Hu, Yisong Wang, Xitong Li, Jingdong Wang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [World models](<https://devfeed.tech/topics/world-models.md>), [multiplayer](<https://devfeed.tech/topics/multiplayer.md>), [synchronization](<https://devfeed.tech/topics/synchronization.md>), [race-condition](<https://devfeed.tech/topics/race-condition.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [automatic](<https://devfeed.tech/tags/automatic.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [control](<https://devfeed.tech/tags/control.md>), [correlation](<https://devfeed.tech/tags/correlation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [gamma](<https://devfeed.tech/tags/gamma.md>), [identity](<https://devfeed.tech/tags/identity.md>), [independent](<https://devfeed.tech/tags/independent.md>), [minecraft](<https://devfeed.tech/tags/minecraft.md>), [motion](<https://devfeed.tech/tags/motion.md>), [multiplayer](<https://devfeed.tech/tags/multiplayer.md>), [persistence](<https://devfeed.tech/tags/persistence.md>), [race-condition](<https://devfeed.tech/tags/race-condition.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [solaris](<https://devfeed.tech/tags/solaris.md>), [spatial](<https://devfeed.tech/tags/spatial.md>), [synchronization](<https://devfeed.tech/tags/synchronization.md>), [world](<https://devfeed.tech/tags/world.md>), [world-models](<https://devfeed.tech/tags/world-models.md>)

### AI overview

MultiWorldBench tests whether multiplayer world models keep independently controlled Minecraft views coherent across 495 configurations, seven task suites, and ten capabilities. Among the generated systems, Gamma-World has the highest average score (21.39), narrowly ahead of Solaris (MP) (20.88); Solaris (SP) scores 10.06, while Engine GT reaches 91.69. Gamma-World leads on several control, shared-state, and revisit capabilities, while Solaris (MP) leads on cross-view motion and race-condition consistency. All generated systems score at most 8.00 on persistence and 1.33 on structural consistency, and none succeeds in spatial reasoning or preserving building identity. Human pairwise judgments match the overall ranking and align strongly with automatic evaluation (mean dimension-level Spearman correlation 0.95).

### Source excerpt

arXiv:2610.11723v5 Announce Type: replace Abstract: Multiplayer world models must ensure that independently controlled views remain consistent with a single shared and persistent world. We introduce MultiWorldBench, a diagnostic Minecraft benchmark comprising 495 case configurations across seven task suites and ten capabilities, including independent control, cross-view motion, shared-state synchronization, persistence, structural reasoning, concurrent interaction, and delayed revisit. Using Engine GT as a reference, we evaluate two multiplayer world models, Solaris (MP) and Gamma-World, together with a single-player generation variant, Solaris (SP). Among the generated systems, Gamma-World achieves the highest ten-capability average at 21.39, followed by Solaris (MP) at 20.88 and Solaris (SP) at 10.06, while Engine GT reaches 91.69. Gamma-World performs better on several control, shared-state, and revisit capabilities, whereas Solaris (MP) leads in cross-view motion and race-condition consistency. Nevertheless, all generated systems score at most 8.00 on state persistence and 1.33 on structural consistency, and none succeeds in spatial reasoning or building-identity preservation. Human pairwise evaluation yields the same overall system ranking and shows strong alignment with the automatic evaluation, with a mean dimension-level Spearman correlation of 0.95. These results show that current multiplayer world models still exhibit a substantial gap relative to Engine GT in maintaining a coherent shared world.

## LINEAGE: A Benchmark for Empirical Lineage Retrieval in AI Scientific Literature

DevFeed: [LINEAGE: A Benchmark for Empirical Lineage Retrieval in AI Scientific Literature](<https://devfeed.tech/articles/lineage-a-benchmark-for-empirical-lineage-retrieval-in-ai-scientific-literature-145594.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25845>)

Author: Pangjing Wu, Xu Yuan, Wenqi Fan, Qing Li

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [search-engines](<https://devfeed.tech/topics/search-engines.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Graphs](<https://devfeed.tech/topics/graphs.md>)

Tags: [17](<https://devfeed.tech/tags/17.md>), [2018](<https://devfeed.tech/tags/2018.md>), [2025](<https://devfeed.tech/tags/2025.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [academic](<https://devfeed.tech/tags/academic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [components](<https://devfeed.tech/tags/components.md>), [constructor](<https://devfeed.tech/tags/constructor.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [graph](<https://devfeed.tech/tags/graph.md>), [graphs](<https://devfeed.tech/tags/graphs.md>), [literature](<https://devfeed.tech/tags/literature.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>)

### AI overview

Lineage benchmarks retrieval of prior AI research methods explicitly used as empirical comparison baselines, a relation that topical relevance and citation links alone do not capture. Its lineage graphs provide multi-hop retrieval ground truth, and Lineage-Constructor builds calibrated graphs from heterogeneous paper records and full-paper baseline evidence. The benchmark covers 67,502 papers, 164,948 lineage edges, and 80,000 multi-facet queries, drawn from 121,048 papers across 17 AI venues from 2018-2025. Tests of 19 retrieval systems found that performance drops sharply with lineage distance, and isolated hits often fail to recover closely connected comparison neighborhoods.

### Source excerpt

arXiv:2611.25845v1 Announce Type: cross Abstract: Current research agents are moving beyond academic search toward literature synthesis and experiment planning. These workflows increasingly rely on literature retrieval systems, which serve as standalone systems or as components of research agents. However, existing scientific literature retrieval benchmarks mainly evaluate these methods using topical relevance or citation links as supervision, which cannot capture which prior methods are used for empirical comparison. To address this gap, we introduce Lineage, a benchmark for empirical lineage retrieval in AI scientific literature. An empirical lineage relation links a prior paper to a source paper that explicitly uses its method as an empirical comparison baseline. These relations form lineage graphs whose multi-hop neighborhoods provide retrieval ground truth for structural evaluation. To construct these graphs at scale, we develop Lineage-Constructor, which converts heterogeneous paper records and full-paper empirical baseline evidence into calibrated lineage graphs. Built from 121,048 papers across 17 top AI venues from 2018 to 2025, Lineage contains 3,950 graph components, 67,502 paper nodes, 164,948 lineage edges, and 80,000 multi-facet retrieval queries. Experiments with 19 representative retrieval systems show that empirical lineage retrieval remains challenging. Performance drops sharply with lineage distance, while structural metrics reveal that isolated retrieval hits often fail to recover closely connected empirical comparison neighborhoods.

## TELEVAL: A Benchmark Designed for Spoken Language Models in Chinese Interactive Scenarios

DevFeed: [TELEVAL: A Benchmark Designed for Spoken Language Models in Chinese Interactive Scenarios](<https://devfeed.tech/articles/televal-a-benchmark-designed-for-spoken-language-models-in-chinese-interactive-scenarios-145305.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2507.18061>)

Author: Zehan Li, Hongjie Chen, Qing Wang, Yuxin Zhang, Jing Zhou, Hang Lv, Mengjie Du, Yaodong Song, Jie Lian, Jian Kang, Jie Li, Yongxiang Li

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Hallucination detection](<https://devfeed.tech/topics/hallucination-detection.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-sd](<https://devfeed.tech/tags/cs-sd.md>), [eess-as](<https://devfeed.tech/tags/eess-as.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [speech](<https://devfeed.tech/tags/speech.md>)

### AI overview

TELEVAL evaluates Chinese spoken language models on both reliable content fulfillment and interactional appropriateness, using spoken inputs without task-specific prompts or predefined answers. The abstract reports that strong factual QA results do not reliably transfer to locally relevant knowledge, Chinese cultural questions, or difficult speech such as dialects. Models also fail to adapt responses to auditory cues, sometimes falling into a "Caption Trap": describing a perceived sound instead of meeting the user's conversational need. The benchmark is presented as a diagnostic framework for content reliability and conversational adaptation.

### Source excerpt

arXiv:2507.18061v5 Announce Type: replace-cross Abstract: Spoken Language Models (SLMs) are expected to support natural spoken interaction beyond task completion. Existing benchmarks assess a range of semantic tasks and conversational capabilities. However, high task scores do not necessarily imply reliable and appropriate spoken interaction. To address this gap, we introduce TELEVAL, a large-scale benchmark focused on Chinese spoken interaction. It evaluates responses to spoken inputs without additional task-specific prompts or predefined answer options. TELEVAL covers two complementary aspects: (1) Reliable Content Fulfillment, which measures semantic accuracy under diverse acoustic and linguistic conditions, and (2) Interactional Appropriateness, which assesses conversational naturalness and appropriate adaptation to auditory cues. Experiments reveal limitations in both aspects. Strong results on factual question-answering datasets do not consistently extend to locally relevant knowledge or challenging speech conditions, such as Chinese cultural questions and dialectal speech. Models also struggle to adapt their responses to auditory cues, even in cases where they explicitly recognize them. We further document a recurring failure pattern, termed the "Caption Trap", in which some responses describe a perceived auditory cue instead of addressing the user's conversational needs. Overall, the evaluated SLMs struggle to deliver consistently accurate and conversationally appropriate responses. TELEVAL provides a diagnostic framework for evaluating both content reliability and conversational adaptation.

## Long-Range Video Presentation Attack Detection: A Three-Class Benchmark Across 1-100 m and Two Sensor Types

DevFeed: [Long-Range Video Presentation Attack Detection: A Three-Class Benchmark Across 1-100 m and Two Sensor Types](<https://devfeed.tech/articles/long-range-video-presentation-attack-detection-a-three-class-benchmark-across-1-100-m-and-two-sensor-types-145009.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25108>)

Author: Lambert Igene, Mohammad Zahir Uddin Chowdhury, Afzal Hossain, Sandip Purnapatra, Peter Johnson, Stephanie Schuckers

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Video Analytics](<https://devfeed.tech/topics/video-analytics.md>)

Tags: [attack](<https://devfeed.tech/tags/attack.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [class](<https://devfeed.tech/tags/class.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [detection](<https://devfeed.tech/tags/detection.md>), [efficientnet](<https://devfeed.tech/tags/efficientnet.md>), [presentation](<https://devfeed.tech/tags/presentation.md>), [range](<https://devfeed.tech/tags/range.md>), [sensor](<https://devfeed.tech/tags/sensor.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

The benchmark evaluates video-based presentation attack detection across 1-100 m, using 374,000+ frames from 261 identities captured indoors and outdoors with a DSLR and a mobile sensor. It labels clips as live, spoof, or props (occlusions) and compares MobileNetV3, MobileNet, EfficientNet, and ResNet-50. The DSLR maintains near-perfect spoof classification across distances, while mobile performance drops sharply beyond 50 m; MobileNetV3 classifies props at 55% at 75 m. EfficientNet reports the best overall result: 98.48% accuracy and 2.01% ACER. Tests on unseen attack datasets report 89-94% accuracy. The abstract identifies distance and sensor resolution as important difficulty factors; it says the dataset is planned for public release under an End-User License Agreement.

### Source excerpt

arXiv:2611.25108v1 Announce Type: new Abstract: Face presentation attack detection (PAD) systems deployed in border control and airport surveillance are constrained by a critical assumption: subjects are captured at close range, typically under one meter. No existing benchmark systematically addresses multi-distance capture, diverse environmental conditions, and concurrent spoofing and occlusion-based attacks, leaving deployed models underprepared for real-world scenarios. We introduce a video-based PAD dataset of 374,000+ frames from 261 unique identities collected across five standoff distances (1 m-100 m) under indoor and outdoor conditions using a DSLR camera (C) and a mobile sensor (M), with three-class labels: live, spoof, and props (occlusions); models are trained and evaluated on video clips rather than single frames. We fine-tune and benchmark four CNN architectures: MobileNetV3, MobileNet, EfficientNet, and ResNet-50. The higher-resolution DSLR maintains near-perfect spoof classification rate across all distances, while the lower-resolution mobile sensor degrades sharply beyond 50 m, with props classification rate dropping to 55% at 75 m for MobileNetV3. EfficientNet achieves the best overall performance (98.48% accuracy, ACER 2.01%). Generalization tests on unseen attack datasets yield 89-94% accuracy, validating cross-domain robustness. These results establish capture distance and sensor resolution as critical but underexplored axes of PAD difficulty. The dataset will be publicly released under an End-User License Agreement to support reproducible long-range PAD research.

## Class-Conditional Conformal Triage for ICAO Face-Image Compliance under Class Imbalance

DevFeed: [Class-Conditional Conformal Triage for ICAO Face-Image Compliance under Class Imbalance](<https://devfeed.tech/articles/class-conditional-conformal-triage-for-icao-face-image-compliance-under-class-imbalance-145045.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25342>)

Author: Mustafa Vedat Yildirim, Inna Skarga-Bandurova

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Image](<https://devfeed.tech/topics/image.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [classification](<https://devfeed.tech/tags/classification.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [conformal-prediction](<https://devfeed.tech/tags/conformal-prediction.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [false-positive](<https://devfeed.tech/tags/false-positive.md>), [image](<https://devfeed.tech/tags/image.md>), [images](<https://devfeed.tech/tags/images.md>), [layer](<https://devfeed.tech/tags/layer.md>), [performance](<https://devfeed.tech/tags/performance.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [reduction](<https://devfeed.tech/tags/reduction.md>), [triage](<https://devfeed.tech/tags/triage.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>)

### AI overview

The study evaluates conformal prediction as an uncertainty-based triage layer for BioGaze's 14 ICAO/ISO face-image criteria, using synthetic TONO data and subject-disjoint splits without retraining the analyser. Marginal conformal prediction reached about 0.90 overall coverage but under-covered violations (as low as a 0.37 ten-partition mean for one criterion). Mondrian class-conditional coverage means were 0.89-0.92 across criteria. Most criterion-level false-positive reduction came from adding per-criterion logistic classifiers (0.145 to 0.048); tuned abstention rules had lower false-positive rates than Mondrian at comparable retake rates. Bonferroni correction for joint coverage across all 14 criteria required retaking nearly every image, exposing a trade-off between coverage guarantees and practical retake burden.

### Source excerpt

arXiv:2611.25342v1 Announce Type: new Abstract: Automated ICAO/ISO face-image compliance analysers provide criterion-level pass/fail decisions, yet these verdicts do not quantify uncertainty when deciding whether to accept, reject, or retake a photograph. We investigate conformal prediction as a triage layer for BioGaze, evaluating 14 compliance criteria on the synthetic TONO benchmark. Without retraining the analyser, we fit per-criterion logistic classifiers and compare marginal and label-conditional (Mondrian) conformal prediction with conventional abstention rules using subject-disjoint splits. Marginal conformal prediction achieved approximately 0.90 overall coverage but substantially under-covered violations, with a ten-partition mean of 0.37 for one criterion. The corresponding Mondrian means ranged from 0.89 to 0.92 across criteria. However, most of the reduction in criterion-level false positives came from the logistic layer (0.145 to 0.048), while tuned abstention rules achieved lower false-positive rates than Mondrian at comparable retake rates. Bonferroni correction for joint coverage across 14 criteria required retaking nearly all images. These findings show that class-conditional conformal prediction can achieve near-target empirical coverage for minority violation classes and reveal trade-offs between statistical coverage, classification performance, and retake requirements in multi-criterion face-image compliance assessment.

## Amazon SageMaker AI now detects latency drift, idle GPUs, and missed optimizations on your inference endpoints

DevFeed: [Amazon SageMaker AI now detects latency drift, idle GPUs, and missed optimizations on your inference endpoints](<https://devfeed.tech/articles/amazon-sagemaker-ai-now-detects-latency-drift-idle-gpus-and-missed-optimizations-on-your-inference-endpoints-144471.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/amazon-sagemaker-ai-now-detects-latency-drift-idle-gpus-and-missed-optimizations-on-your-inference-endpoints/>)

Author: Hazim Qudah

Published: 2026-11-19T21:28:18Z

Content type: tutorial

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [gpu monitoring](<https://devfeed.tech/topics/gpu-monitoring.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-sagemaker-ai](<https://devfeed.tech/tags/amazon-sagemaker-ai.md>), [amazon-sagemaker-studio](<https://devfeed.tech/tags/amazon-sagemaker-studio.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [aws-user-notifications](<https://devfeed.tech/tags/aws-user-notifications.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [drift](<https://devfeed.tech/tags/drift.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [gpu-utilization](<https://devfeed.tech/tags/gpu-utilization.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [inference-endpoints](<https://devfeed.tech/tags/inference-endpoints.md>), [latency](<https://devfeed.tech/tags/latency.md>), [optimizations](<https://devfeed.tech/tags/optimizations.md>), [performance](<https://devfeed.tech/tags/performance.md>), [serving](<https://devfeed.tech/tags/serving.md>), [studio](<https://devfeed.tech/tags/studio.md>)

### AI overview

Amazon SageMaker AI monitors eligible generative AI inference endpoints by default when they use vLLM serving containers and GPU instances. Using aggregated latency, GPU utilization, and serving configuration signals, it can flag week-over-week latency drift, mostly idle GPUs, or supported optimizations and updates that are missing. Monitoring does not read request or response content and has no GPU cost. Findings arrive through AWS Health and email; users can open them in SageMaker Studio and benchmark alternatives on separate compute, comparing response time, throughput, and cost per million tokens. No endpoint change occurs until the user chooses to apply a result, and keeping the current configuration is also an option. Benchmarks incur standard SageMaker charges or use an ML reservation. Monitoring can be disabled by setting OPT_OUT_TRACKING on a model container; doing so prevents telemetry collection and notifications for that endpoint.

### Source excerpt

Amazon SageMaker AI now monitors your generative AI inference endpoints by default and tells you when a change could improve performance or lower cost. It tracks latency, GPU utilization, and serving configuration, then sends a finding through AWS Health with a one-click path to benchmark alternatives in SageMaker Studio before you apply any change.

## SixthView: A Benchmark for Hidden-View Reconstruction and 3D Assembly Reasoning

DevFeed: [SixthView: A Benchmark for Hidden-View Reconstruction and 3D Assembly Reasoning](<https://devfeed.tech/articles/sixthview-a-benchmark-for-hidden-view-reconstruction-and-3d-assembly-reasoning-142719.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23891>)

Author: Pooya Khosravi, Jen-Feng Chang, Fnu Pramono

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [3D](<https://devfeed.tech/topics/3d.md>), [vlm](<https://devfeed.tech/topics/vlm.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [assembly](<https://devfeed.tech/tags/assembly.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [computer-science](<https://devfeed.tech/tags/computer-science.md>), [computer-vision](<https://devfeed.tech/tags/computer-vision.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [github](<https://devfeed.tech/tags/github.md>), [grid](<https://devfeed.tech/tags/grid.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [view](<https://devfeed.tech/tags/view.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>)

### AI overview

SixthView tests whether models can reconstruct an unambiguous hidden orthographic view of a 3D polycube assembly from five views and component-shape information. It includes 602 synthetic assemblies, with the target represented as a dense grid of part labels; the authors verify that each item's exact geometry and observed labels determine a unique target. On 552 fully evaluated items, exact-grid accuracy with images plus exact text ranged from 33.2% to 85.4% across three vision-language models. Adding exact text to separate images improved accuracy by 19.7-63.0 percentage points, while adding images to exact text changed it by only −0.5 to +2.1 points. Separate image panels scored 6.7-11.1 points above a merged sheet. The abstract attributes the image-only variants' added challenge to recovering shapes from occluded renderings.

### Source excerpt

arXiv:2611.23891v1 Announce Type: cross Abstract: Inferring the unseen side of a 3D assembly requires reasoning about part placement and occlusion. Different assemblies can match the observed views while producing different hidden views, leaving the target ambiguous. We introduce SixthView, a benchmark in which models predict a sixth orthographic view from five observed views and information about the component shapes. The benchmark contains 602 synthetic polycube assemblies with 2--10 parts and requires a dense grid of part labels for the missing view, allowing both exact-view and cell-level evaluation. For each item, we exhaustively verify that all legal assemblies consistent with the exact component geometry and observed label grids yield the same target view. With these inputs, the task tests symbolic assembly and coordinate reasoning. The image-only variants additionally require shape recovery from rendered views, where occlusion can make different 3D shapes look identical. We evaluate three vision-language models under four input conditions. Across 552 items with complete evaluations, exact-grid accuracy with images plus exact text ranges from 33.2% to 85.4%. Adding exact text to separate images improves accuracy by 19.7--63.0 percentage points, whereas adding images to exact text changes it by only $-0.5$ to $+2.1$ points. Separate image panels outperform a merged sheet by 6.7--11.1 points. The results show that performance depends strongly on access to exact symbolic information, with little further improvement from images once the geometry is specified. GitHub: https://github.com/facebookresearch/sixthview_benchmark

## How Far Does Online RL Generalize in Mobile GUI Agents? A Controlled Study with AndroidWorld-Generalization

DevFeed: [How Far Does Online RL Generalize in Mobile GUI Agents? A Controlled Study with AndroidWorld-Generalization](<https://devfeed.tech/articles/how-far-does-online-rl-generalize-in-mobile-gui-agents-a-controlled-study-with-androidworld-generalization-143237.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2603.07432>)

Author: Li Gu, Zihuan Jiang, Zhixiang Chi, Huan Liu, Ziqiang Wang, Yuanhao Yu, Linqiang Guo, Yang Wang

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [few-shot](<https://devfeed.tech/topics/few-shot.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-hc](<https://devfeed.tech/tags/cs-hc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [few-shot](<https://devfeed.tech/tags/few-shot.md>), [gui](<https://devfeed.tech/tags/gui.md>), [gui-agents](<https://devfeed.tech/tags/gui-agents.md>), [mobile](<https://devfeed.tech/tags/mobile.md>), [online](<https://devfeed.tech/tags/online.md>), [ppo](<https://devfeed.tech/tags/ppo.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [release](<https://devfeed.tech/tags/release.md>), [rl](<https://devfeed.tech/tags/rl.md>), [test-time-adaptation](<https://devfeed.tech/tags/test-time-adaptation.md>)

### AI overview

In AndroidWorld-Generalization, online RL gains shrink as mobile GUI tasks move farther from training contexts: from unseen instances to templates to applications. Using the same RL algorithm with UI-TARS-7B-SFT and Qwen3-VL-8B, the study reports gains of +17.5/+11.1/+4.2 and +10.3/+7.4/+0, respectively. Training performance kept rising while evaluation performance saturated; the decay was also reproduced with PPO, and RL helped less with the stronger prior. Few-shot adaptation in the target application recovered +22.9, suggesting a complement to further RL training. The authors release the benchmark, training system, recipes, and model weights.

### Source excerpt

arXiv:2603.07432v2 Announce Type: replace-cross Abstract: Online reinforcement learning (RL) has advanced mobile GUI agents on AndroidWorld to over 80\% success, but these systems train and evaluate in the same environment, and their training tasks are not released for auditing. How far RL generalizes beyond its training contexts therefore remains unmeasured. We formalize the mobile environment as a Contextual MDP and instantiate a ladder of three progressively distant context shifts as AndroidWorld-Generalization: Unseen Instance, Unseen Template, and Unseen Application, with separation verified manually. In addition, we provide an open RL training system that makes studies at this scale feasible. Applying the identical RL algorithm to two priors of different origin and competence, UI-TARS-7B-SFT at 23.0\% and Qwen3-VL-8B at 52.6\%, we find that RL improves both in distribution but that gains fall monotonically up the ladder (+17.5/+11.1/+4.2 and +10.3/+7.4/+0), with training performance rising throughout while evaluation performance saturates. The decay is algorithm-agnostic, reproduced under PPO, and RL contributes less as the prior strengthens. Few-shot adaptation in the target application recovers +22.9, motivating test-time adaptation as a complement to further RL training. We release the benchmark, training system, recipes, and model weights.

## What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation

DevFeed: [What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation](<https://devfeed.tech/articles/what-does-a-temporal-benchmark-score-measure-decomposing-channel-use-in-video-vlm-evaluation-143291.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.12304>)

Author: Farrukh Rahman

Published: 2026-11-19T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [bias](<https://devfeed.tech/tags/bias.md>), [channel](<https://devfeed.tech/tags/channel.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [event](<https://devfeed.tech/tags/event.md>), [events](<https://devfeed.tech/tags/events.md>), [frame](<https://devfeed.tech/tags/frame.md>), [order](<https://devfeed.tech/tags/order.md>), [pixels](<https://devfeed.tech/tags/pixels.md>), [position](<https://devfeed.tech/tags/position.md>), [positional-encoding](<https://devfeed.tech/tags/positional-encoding.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [question](<https://devfeed.tech/tags/question.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [reading](<https://devfeed.tech/tags/reading.md>), [score](<https://devfeed.tech/tags/score.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [video](<https://devfeed.tech/tags/video.md>), [vlm](<https://devfeed.tech/tags/vlm.md>)

### AI overview

A temporal video benchmark score can reflect single-frame cues and answer priors, so it may not show whether a model reads event order from video pixels or positional encoding (RoPE). The abstract proposes testing this with reversal conflicts, comparing accuracy against forward labels and, where labels transform deterministically under reversal, against reversed labels. These readings distinguish models that follow reversed visual content from ones disrupted by the conflict. In the reported results, Qwen3-VL follows the reversed event it sees, while Molmo2 follows RoPE: changing only the position indices flips its prediction. Controls indicate both model families can process reversed content when the channels agree and share a similar forward-event bias, which the authors say does not explain the difference. Thus, models with similar benchmark scores may fail on different inputs; existing frame-shuffle and multi-frame-gain checks address whether a task needs temporal information, not which channel supplies order.

### Source excerpt

arXiv:2607.12304v2 Announce Type: replace-cross Abstract: A score on a temporal video question answering benchmark is meant to measure that a model has temporal understanding, but it conflates two questions. (1)~is the question even temporal, does it need several frames and their order? And (2)~when it does, does the model recover the order from the pixels, or read it off the positional encoding (RoPE)? Most of a temporal score answers neither, since a single frame and answer priors often carry it, and the field's validity checks, frame-shuffle sensitivity and multi-frame gain, speak only to (1). For (2) we score a reversal conflict, frames reversed while RoPE stays forward, two ways: label-free, as the accuracy lost against the forward labels, and against reversed labels wherever labels transform deterministically under time reversal. This separates a model that follows the reversed content from one merely disrupted. The two readings dissociate two similarly scoring models at both model scales. Qwen3-VL answers the reversed event it actually sees, reading the visual sequence. Molmo2 correctly reports a backwards video when positions agree, and flipping only the RoPE indices flips its prediction to the forward events, i.e. it reads positions. We call these regimes \emph{visual-sequence-dominant} and \emph{position-dominant}. Controls show reversed content is processable whenever the channels agree, and that both families share a bias toward describing events as running forward (arrow-of-time prior) that, being similar in strength, cannot by itself account for the dissociation. The distinction matters because the two channels fail on opposite inputs: two models with the same score are not interchangeable.

## TabBench-Bio: A Living Benchmark for Machine Learning on High-Dimensional Biomedical Tables

DevFeed: [TabBench-Bio: A Living Benchmark for Machine Learning on High-Dimensional Biomedical Tables](<https://devfeed.tech/articles/tabbench-bio-a-living-benchmark-for-machine-learning-on-high-dimensional-biomedical-tables-142558.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.07441>)

Author: Jules Kreuer, Sofiane Ouaari, Julia Hellmig, Julius Braitinger, Nico Pfeifer

Published: 2026-11-19T05:00:00Z

Content type: comparison

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [automl](<https://devfeed.tech/tags/automl.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [biomedical](<https://devfeed.tech/tags/biomedical.md>), [cross-validation](<https://devfeed.tech/tags/cross-validation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [neural-networks](<https://devfeed.tech/tags/neural-networks.md>), [tables](<https://devfeed.tech/tags/tables.md>)

### AI overview

TabBench-Bio benchmarks machine learning on 43 biomedical datasets, comparing classical estimators, neural networks, and tabular foundation models across 28 feature-by-sample settings under a shared cross-validation protocol. At the reference setting of 10,000 features and 100 training samples, RealTabPFN 2.5 has the highest point estimate, followed by TabPFN 3 and Logistic Regression; a paired bootstrap estimates an 87 Elo advantage over TabPFN 3 (95% interval 43-129). Foundation models generally rank highly, but the leading configuration varies with setting and biomedical modality. AutoGluon's one-hour "extreme" preset is included separately as a resource-intensive reference at the reference and full settings. Fold-level predictions, run status, and deterministic aggregations support reproducibility; the benchmark accepts new dataset contributions.

### Source excerpt

arXiv:2609.07441v3 Announce Type: replace-cross Abstract: Biomedical tables often combine thousands of measured variables with only tens or hundreds of labelled samples, a regime that is poorly represented in general-purpose tabular benchmarks. We introduce TabBench-Bio, a living and interactive benchmark of 43 biomedical datasets spanning multiple domains. Under a shared cross-validation protocol, we compare classical estimators, neural networks, and tabular foundation models across 28 feature-by-sample operating points. At the reference cell of 10,000 features and 100 training samples, RealTabPFN 2.5 has the highest point estimate, closely followed by TabPFN 3 and Logistic Regression. A paired bootstrap over the target pool separates RealTabPFN 2.5 from TabPFN 3 by 87 Elo (95\% interval [43, 129]). Tabular foundation models generally occupy the leading ranks, while the strongest configuration depends on the operating point and biomedical modality. The AutoML framework AutoGluon, using its one-hour "extreme" preset, is configured as a separate resource-intensive reference and reported here at the reference and full cell. Fold-level predictions, run status, and deterministic aggregations make every reported result reproducible and reusable. The benchmark is open to contributions of new biomedical datasets. The interactive leaderboard is available at https://tabbench-bio.eu.

## Agent Verifiers in Counterfactual Analysis: A Pilot Study Benchmarking Agentic Heterogeneous Treatment Effects

DevFeed: [Agent Verifiers in Counterfactual Analysis: A Pilot Study Benchmarking Agentic Heterogeneous Treatment Effects](<https://devfeed.tech/articles/agent-verifiers-in-counterfactual-analysis-a-pilot-study-benchmarking-agentic-heterogeneous-treatment-effects-143419.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23621>)

Author: Satiyabooshan Murugaboopathy, Entejar Alam, Adel Daoud, Connor T. Jerzak

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Predictive AI](<https://devfeed.tech/topics/predictive-ai.md>)

Tags: [adapter](<https://devfeed.tech/tags/adapter.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benefits](<https://devfeed.tech/tags/benefits.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [budget](<https://devfeed.tech/tags/budget.md>), [builds](<https://devfeed.tech/tags/builds.md>), [channel](<https://devfeed.tech/tags/channel.md>), [combined](<https://devfeed.tech/tags/combined.md>), [compare](<https://devfeed.tech/tags/compare.md>), [context](<https://devfeed.tech/tags/context.md>), [counterfactual-analysis](<https://devfeed.tech/tags/counterfactual-analysis.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [effect](<https://devfeed.tech/tags/effect.md>), [pilot](<https://devfeed.tech/tags/pilot.md>)

### AI overview

The pilot proposes a three-layer check for agent-generated covariates: filter retrieved material to pre-treatment dates, validate heterogeneity using rank-weighted average treatment effects (RATEs), and assess encoder leakage with time-vintaged encoders, future-fact probes, and weight perturbations. Across 11 datasets, a persistent, date-verified GrokBot workspace had higher mean RATE ratios in 6 datasets for text alone and 7 for combined features; paired tests found neither retrieval workflow universally dominant. Encoder choice also shifted heterogeneity signals on fixed text. Vintaged probes found no clear weight-leakage evidence, while adapter perturbations showed that memorized future facts persisted until noise degraded model output. The authors conclude temporal filtering alone is insufficient; provenance, out-of-sample signal, and encoder weights all need scrutiny.

### Source excerpt

arXiv:2611.23621v1 Announce Type: new Abstract: Agents can expand the covariate space of randomized experiments by retrieving fast-moving local context missed by slow-changing survey baseline covariates, helping characterize who benefits most from an intervention. Verifying such covariates is difficult: individual treatment effects are unobserved, retrieval leakage can pull post-treatment information into nominally pretreatment summaries, and encoder leakage can inject post-treatment pretraining knowledge into representations even after temporal filtering of inputs. Guided by a two-channel information bound separating summary provenance from encoder weights, we propose a three-layer benchmark protocol: (i) retrieval admissibility via pre-treatment date filtering; (ii) heterogeneity validation scored rank-weighted average treatment effects (RATEs); and (iii) encoder admissibility checked with chronologically vintaged encoders, future-fact probes, and weight perturbations. Across 11 datasets anchored by Uganda's Youth Opportunities Program, we compare session-isolated retrieval (sandboxed, bounded-budget agent runs per study) against a persistent workspace teammate (xAI's GrokBot) that builds one date-verified corpus per study while blinded to outcomes. The workspace teammate yields higher mean RATE ratios in 6/11 datasets for text alone and 7 for combined features, though paired tests show neither workflow universally dominates. Even on fixed retrieved text, encoder choice significantly shifts heterogeneity signal. Chronologically vintaged probes show no clear evidence of weight leakage, but single-record adapter perturbation analyses show that memorized future facts, once present, persist until injected noise degrades model output. Temporal filtering of retrieval is necessary but insufficient: verifying agent-generated covariates requires jointly examining source provenance, out-of-sample treatment-effect signal, and encoder weights.

## Idempotency Is Not Recovery: A Fault-Injection Study of Language-Model Agent Tool Actions

DevFeed: [Idempotency Is Not Recovery: A Fault-Injection Study of Language-Model Agent Tool Actions](<https://devfeed.tech/articles/idempotency-is-not-recovery-a-fault-injection-study-of-language-model-agent-tool-actions-142183.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23353>)

Author: Song Xu

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [about](<https://devfeed.tech/tags/about.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [actions](<https://devfeed.tech/tags/actions.md>), [after](<https://devfeed.tech/tags/after.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-runtime](<https://devfeed.tech/tags/agent-runtime.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [authorization](<https://devfeed.tech/tags/authorization.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [call](<https://devfeed.tech/tags/call.md>), [cases](<https://devfeed.tech/tags/cases.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [fault-injection](<https://devfeed.tech/tags/fault-injection.md>), [idempotency](<https://devfeed.tech/tags/idempotency.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [model](<https://devfeed.tech/tags/model.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [study](<https://devfeed.tech/tags/study.md>), [tool](<https://devfeed.tech/tags/tool.md>)

### AI overview

In the tested agent runtime, suppressing duplicate tool effects did not by itself guarantee recovery or a clear outcome. Across Windows and Linux trials, the full configuration combined authorization, an action ledger, target-side idempotency, reconciliation, and evidence sealing. It recorded no repeated effects in 200 crash-and-replay pairs, versus 117 for each simpler baseline. A ledger-related ablation produced 170 orphaned effects in 170 trials and fewer recovery successes; disabling reconciliation left all 100 tested post-effect crash outcomes ambiguous despite no observed duplicates. The full runtime ran at about one-seventh the throughput of fail-open, and evidence-write faults left 75 of 100 trials per platform unresolved. A separate supplement using the same inputs found a similar pattern under response loss or delay. Results are limited to the stated benchmark.

### Source excerpt

arXiv:2611.23353v1 Announce Type: new Abstract: When a language-model agent's tool call is interrupted, the action may already have changed an external system without confirmation. Retrying risks duplication; assuming success may conceal an action that never occurred. We study these failure modes in a controlled agent runtime combining authorization, an action ledger, target-side idempotency, reconciliation, and evidence sealing. We inject ten single-host fault classes and compare four configurations and three ablations using a target-side oracle separate from the runtime ledger. On each of Windows and Linux, 7,513 trials used 96 captured model responses from 24 constructed cases. In 200 known original crash and replay pairs, the full configuration recorded 0/200 repeated effects versus 117/200 for each simpler baseline. A ledger-related ablation had 170/170 orphaned effects versus 0/170 and 200/300 recovery successes versus 300/300 for the full configuration. Disabling reconciliation left 100/100 known post-effect crash outcomes ambiguous despite no observed duplicates. A separate supplement designed after the initial results and reusing the same inputs found a similar pattern under response loss or delay. The full runtime had about one-seventh of fail-open throughput, while injected evidence-write faults left 75/100 trials per platform unresolved. These findings distinguish duplicate suppression from recovery and identify their measured reliability and performance tradeoffs within the tested benchmark.

## f2x50: A Repository-Scale Benchmark for Fortran-to-X Software Migration

DevFeed: [f2x50: A Repository-Scale Benchmark for Fortran-to-X Software Migration](<https://devfeed.tech/articles/f2x50-a-repository-scale-benchmark-for-fortran-to-x-software-migration-142143.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23268>)

Author: Shraddha Surana, Ashwin Srinivasan

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Fortran](<https://devfeed.tech/topics/fortran.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Compilers and Linkers](<https://devfeed.tech/topics/compilers-and-linkers.md>), [Repository](<https://devfeed.tech/topics/repository.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-pl](<https://devfeed.tech/tags/cs-pl.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [fortran](<https://devfeed.tech/tags/fortran.md>), [migration](<https://devfeed.tech/tags/migration.md>), [python](<https://devfeed.tech/tags/python.md>), [repository](<https://devfeed.tech/tags/repository.md>), [reproducibility](<https://devfeed.tech/tags/reproducibility.md>), [scale](<https://devfeed.tech/tags/scale.md>), [software](<https://devfeed.tech/tags/software.md>), [software-migration](<https://devfeed.tech/tags/software-migration.md>), [translation](<https://devfeed.tech/tags/translation.md>)

### AI overview

f2x50 benchmarks repository-scale Fortran migration across 50 public repositories pinned to exact Git revisions, addressing behavior across modules, shared state, build structure, and dependencies. It includes 12,116 tracked Fortran files, 6,569,806 lines, three structural-complexity tiers, and 561 deterministic behavioral probes. Of these, 446 compile pinned source files with small drivers and cover 131 repository-routine pairs; each repository has at least five direct checks collectively executing at least two routines. Other probes are labeled as extractions, inline reimplementations, or synthetic fixtures. The benchmark supplies target-independent acquisition, validation, license-review, and reproducibility tools, plus a reference adapter for Python targets, to support comparable evaluations across migration tools and target languages.

### Source excerpt

arXiv:2611.23268v1 Announce Type: new Abstract: Repository-level migration requires more than translating isolated functions: it must preserve behaviour across modules, shared state, build structure, and dependencies. Existing translation benchmarks predominantly cover small programs or small repositories, or fixed and widely studied language pairs. We present f2x50, a target-independent benchmark comprising 50 public Fortran repositories pinned to exact Git revisions. It contains 12,116 Git-tracked Fortran files and 6,569,806 lines, divided into three structural-complexity tiers and paired with 561 deterministic behavioural probes. Of these probes, 446 compile pinned source files with small drivers and span 131 repository--routine pairs. Every repository has at least five direct checks that collectively execute at least two routines. f2x50 provides target-independent tools for acquisition, validation, licence review, and reproducibility, together with a reference adapter for evaluating Python targets. The remaining probes are explicitly classified as extractions, inline reimplementations, or synthetic fixtures. The benchmark therefore supports comparable evaluations across migration tools and target languages.

## Benchmarking LLM abilities on postdoc-level natural science problems

DevFeed: [Benchmarking LLM abilities on postdoc-level natural science problems](<https://devfeed.tech/articles/benchmarking-llm-abilities-on-postdoc-level-natural-science-problems-142778.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24321>)

Author: Alina Avanesyan, Alexey Boldyrev, Liqun Deng, Denis Derkach, Mary Godunova, Mikhail Hushchyn, Dmitry Ilvovsky, Irina Knyazeva, Irina Piontkovskaya, Dmitry Polubotko, Fedor Ratnikov, Ekaterina Trofimov

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Frontier AI](<https://devfeed.tech/topics/frontier-ai.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [astrophysics](<https://devfeed.tech/tags/astrophysics.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [causal-inference](<https://devfeed.tech/tags/causal-inference.md>), [classification](<https://devfeed.tech/tags/classification.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [frontier-models](<https://devfeed.tech/tags/frontier-models.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [model-comparison](<https://devfeed.tech/tags/model-comparison.md>), [neuroscience](<https://devfeed.tech/tags/neuroscience.md>), [performance](<https://devfeed.tech/tags/performance.md>), [physics](<https://devfeed.tech/tags/physics.md>), [physics-data-an](<https://devfeed.tech/tags/physics-data-an.md>), [science](<https://devfeed.tech/tags/science.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [tests](<https://devfeed.tech/tags/tests.md>), [uncertainty-estimation](<https://devfeed.tech/tags/uncertainty-estimation.md>)

### AI overview

STEMBench evaluates 12 research problems in high-energy physics, astrophysics, and neuroscience using each field's own figure of merit. Each task includes measured baselines: a submission without domain knowledge and an expert reference reproducing a published method, both run with benchmark code. Across eight frontier models, winning individual tasks did not reliably translate into domain-level leadership; in a high-effort comparison, at least one model's mean score exceeded the expert anchor on seven tasks. The other five involved uncertainty estimation, latent and causal inference, and cross-site clinical classification. Numerical performance and official-submission validity are reported separately.

### Source excerpt

arXiv:2611.24321v1 Announce Type: new Abstract: Scientific problems have no answer key, so evaluating an agent means deciding what counts as a better solution before any agent is run. Existing benchmarks either match a symbolic answer against a key or, where they require executable work, score it by unit tests, by a rubric, or by position on a competition leaderboard. None of these is the criterion by which a field decides that one published method beats another. STEMBench adopts that criterion directly. Each of its twelve research problems in high-energy physics, astrophysics and neuroscience is scored by its own field's figure of merit, and each comes with two measured anchors: a submission that uses no domain knowledge, and an expert reference reproducing a published method, both obtained by running code shipped with the benchmark. The interval between them states how much of a problem is crossed by scientific insight rather than by engineering. Across eight frontier models, task-level wins and domain-level leadership dissociate: a model that wins five of the twelve tasks leads no domain mean. In the high-effort cross-model comparison, at least one model's mean score exceeds the expert anchor on seven of twelve tasks. The remaining five span uncertainty estimation, latent and causal inference, and cross-site clinical classification. Numerical scores and official-submission validity capture different aspects of performance and are reported separately. We release the tasks, graders, anchors and harness.

## VCF 9.1 and 6th Gen AMD EPYC CPUs Achieve VMmark Leadership Scores

DevFeed: [VCF 9.1 and 6th Gen AMD EPYC CPUs Achieve VMmark Leadership Scores](<https://devfeed.tech/articles/vcf-9-1-and-6th-gen-amd-epyc-cpus-achieve-vmmark-leadership-scores-142029.md>)

Original publisher: [Read original article](<https://blogs.vmware.com/cloud-foundation/2026/11/18/vcf-9-1-and-6th-gen-amd-epyc-cpus-achieve-vmmark-leadership-scores/>)

Author: david morse

Published: 2026-11-18T23:22:17Z

Content type: release

Language: en

Sources: [VMware](<https://devfeed.tech/sources/vmware-blogs.md>)

Topics: [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [systems](<https://devfeed.tech/topics/systems.md>), [VMware Cloud Foundation](<https://devfeed.tech/topics/vcf-9-1.md>), [dell](<https://devfeed.tech/topics/dell.md>)

Tags: [amd](<https://devfeed.tech/tags/amd.md>), [amd-epyc](<https://devfeed.tech/tags/amd-epyc.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [broadcom](<https://devfeed.tech/tags/broadcom.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-infrastructure](<https://devfeed.tech/tags/cloud-infrastructure.md>), [cpus](<https://devfeed.tech/tags/cpus.md>), [dell-technologies](<https://devfeed.tech/tags/dell-technologies.md>), [epyc-9006](<https://devfeed.tech/tags/epyc-9006.md>), [home-page](<https://devfeed.tech/tags/home-page.md>), [leadership](<https://devfeed.tech/tags/leadership.md>), [performance](<https://devfeed.tech/tags/performance.md>), [power](<https://devfeed.tech/tags/power.md>), [power-performance](<https://devfeed.tech/tags/power-performance.md>), [poweredge](<https://devfeed.tech/tags/poweredge.md>), [r9825](<https://devfeed.tech/tags/r9825.md>), [server-cpus](<https://devfeed.tech/tags/server-cpus.md>), [vcf](<https://devfeed.tech/tags/vcf.md>), [vcf-9-1](<https://devfeed.tech/tags/vcf-9-1.md>), [vmmark](<https://devfeed.tech/tags/vmmark.md>), [vsphere](<https://devfeed.tech/tags/vsphere.md>), [vsphere-9-1](<https://devfeed.tech/tags/vsphere-9-1.md>)

### AI overview

Dell's one-socket PowerEdge R9815 and two-socket R9825, powered by 6th Generation AMD EPYC 9996 "Venice" CPUs, lead their respective VMmark server performance categories and rank highest in both listed performance-per-kilowatt categories. In the cited R9815 comparison with a 5th Generation AMD EPYC "Turin" R7715, a 60% increase in core count corresponds to a 70% higher VMmark score and capacity rising from 3.8 tiles (77 VMs) to 5.8 tiles (121 VMs), a 57% increase. The excerpt attributes the score gain to the new hardware; VCF 9.1.2 enables CPU support, while performance across 9.1 maintenance releases is described as virtually unchanged. Results are based on VMmark submissions published by November 10, 2026; the tool is provided free by Broadcom as-is.

### Source excerpt

We're excited to announce the first VMmark benchmark results using new AMD EPYC 9006 Series Server CPUs. Submitted by Dell Technologies across 1-socket PowerEdge R9815 and 2-socket PowerEdge R9825 configurations, these two server platforms set a number of records in both performance and power-performance: see the VMmark Results page for more information. Here are some ... Continued The post VCF 9.1 and 6th Gen AMD EPYC CPUs Achieve VMmark Leadership Scores appeared first on VMware Blogs.

## Encoding, Decodability, and Use: A Five-Layer Framework for Benchmark Measurement Audits with Evidence from UAV Intent Recognition

DevFeed: [Encoding, Decodability, and Use: A Five-Layer Framework for Benchmark Measurement Audits with Evidence from UAV Intent Recognition](<https://devfeed.tech/articles/encoding-decodability-and-use-a-five-layer-framework-for-benchmark-measurement-audits-with-evidence-from-uav-intent-recognition-140028.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22434>)

Author: Zhirong Su, Tian Zhang

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [audit](<https://devfeed.tech/topics/audit.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [behaviour](<https://devfeed.tech/tags/behaviour.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [bias](<https://devfeed.tech/tags/bias.md>), [box](<https://devfeed.tech/tags/box.md>), [change](<https://devfeed.tech/tags/change.md>), [correlation](<https://devfeed.tech/tags/correlation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [drone](<https://devfeed.tech/tags/drone.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [feature](<https://devfeed.tech/tags/feature.md>), [flow](<https://devfeed.tech/tags/flow.md>), [framework](<https://devfeed.tech/tags/framework.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [memory](<https://devfeed.tech/tags/memory.md>), [optical](<https://devfeed.tech/tags/optical.md>), [out-of-sample](<https://devfeed.tech/tags/out-of-sample.md>), [power](<https://devfeed.tech/tags/power.md>), [program](<https://devfeed.tech/tags/program.md>), [recognition](<https://devfeed.tech/tags/recognition.md>)

### AI overview

The AeroIntent audit separates benchmark measurement into observation, representation, decoding, use, and behaviour, showing that accuracy alone does not establish radial-velocity understanding. Observation was unreliable for 7 of 20 Drone programs and reduced the direct visual correlate to a 0.26% scale change. Two task-designed RGB features had nominal correlations with radial velocity, but multi-feature readouts did not exceed the cluster minimum detectable effect; the largest positive global out-of-sample correlation was r=+0.080. At the stated threshold and power, the MDE was |r|=0.599 and the observed correlation had power 0.00; aligned and orthogonal injection experiments supported this detection condition. Accuracy was also consistent with program memory and static cues. Behaviour results were consistent with accuracy not requiring a radial-velocity readout: appearance accuracy (0.6552) exceeded bounding-box trajectory accuracy (0.4080), with paired cluster-level Δ=+0.2471 (p=0.0132).

### Source excerpt

arXiv:2611.22434v1 Announce Type: cross Abstract: Benchmark accuracy is routinely read as evidence that a model understands the construct the benchmark measures. We propose and instantiate on AeroIntent a five-layer framework that decomposes that reading into observation, representation, decoding, use, and behaviour. The framework separates encoding, decodability, and use as distinct levels. In the instantiation, the observation layer is unreliable on 7 of the 20 Drone programs and floors the direct visual correlate of radial velocity at a 0.26\% scale change. The representation layer exposes two task-designed RGB features with nominal signals. RAFT flow radial bias reaches $r=+0.257$ ($p=0.030$), and bounding-box optical-flow divergence reaches $r=+0.203$ ($p=0.029$). No multi-feature readout gives a fold-mean test correlation above the cluster minimum detectable effect (MDE), and the largest positive global out-of-sample correlation is $r=+0.080$. The decoding layer sets the MDE at $|r|=0.599$ at the multi-feature Bonferroni threshold ($\alpha=0.00714$, power 0.80), and the observed $r=+0.080$ has power 0.00. Two injection experiments establish the MDE's condition from both sides: aligned injections are detected at or above the simulated power, and orthogonal injections are not detected. The use layer finds accuracy consistent with program memory (program-identity 0.937/0.597/0.210) and multi-region static cues. The behaviour layer is consistent with accuracy not requiring a radial-velocity readout (appearance 0.6552 vs bounding-box trajectory 0.4080, cluster-level paired $\Delta=+0.2471$, $p=0.0132$).

## RoboTeam: Benchmarking Vision-Based Multi-Agent Control of Many Robot Arms in Real Time

DevFeed: [RoboTeam: Benchmarking Vision-Based Multi-Agent Control of Many Robot Arms in Real Time](<https://devfeed.tech/articles/roboteam-benchmarking-vision-based-multi-agent-control-of-many-robot-arms-in-real-time-139969.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22138>)

Author: Changxuan Fan, Zhongwei Xie, Boyi Zhang, Shaoli Hu, Chang Xu, Xiyu Ren, Yufei Li, Qiao Xiao, Huihao Jing, Kawai Chung, Wenbin Hu, Hongyu Luo, Lingyun Xie, Yisen Gao, Chunyang Li, Haoyu Huang, Wuganjin

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Computer vision](<https://devfeed.tech/topics/computer-vision.md>), [hierarchical agent systems](<https://devfeed.tech/topics/hierarchical-agent-systems.md>), [real-time](<https://devfeed.tech/topics/real-time.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-control](<https://devfeed.tech/tags/agent-control.md>), [agents](<https://devfeed.tech/tags/agents.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [camera](<https://devfeed.tech/tags/camera.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [multi-agent](<https://devfeed.tech/tags/multi-agent.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [robot](<https://devfeed.tech/tags/robot.md>), [robot-arms](<https://devfeed.tech/tags/robot-arms.md>), [system](<https://devfeed.tech/tags/system.md>), [systems](<https://devfeed.tech/tags/systems.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

RoboTeam benchmarks agents controlling 4-16 robot arms across 36 long cooperative tasks, using camera images alone while simulated time continues during agent reasoning. In its adaptive setup, gpt-6.1-sol and opus-5.5 complete 28 and 27 tasks; four other tested models complete at most 3. Performance is costly and failure-prone: a typical gpt-6.1-sol task takes 3.5 hours and 218 million input tokens, with about a third of steps failing at least once and requiring retries. Four agent systems complete similar numbers of tasks but trade speed against token use: hierarchical and decentralized systems are faster and use up to 1.9 times the tokens, while single-agent uses fewer tokens but takes 1.6 times as long. The abstract concludes that no tested model or system is simultaneously fast, inexpensive, and reliable.

### Source excerpt

arXiv:2611.22138v1 Announce Type: cross Abstract: Multimodal agents can now control robot arms by calling tools and writing code. Existing benchmarks fall short in three ways: most test one or two arms on short tasks, many pause the world while the agent thinks, and many give it ground-truth object positions. We present RoboTeam, a simulated benchmark where agents control 4 to 16 robot arms in real time from camera images alone. Its 36 tasks are long jobs that the arms must do together, such as passing timbers or building a picture from hundreds of cubes. We start from the most flexible agent system, the adaptive system, where one agent decides when to fork delegates. With it, gpt-6.1-sol and opus-5.5 complete 28 and 27 tasks, and four other models at most 3. But at a high price: a typical task takes gpt-6.1-sol 3.5 hours and 218 million input tokens, and a third of its steps fail at least once and must be retried. We then compare it with three fixed systems: single-agent, hierarchical, and decentralized. All four complete a similar number of tasks but differ in time and tokens. The hierarchical and decentralized systems are faster but use up to 1.9 times the tokens, and the single agent uses fewer tokens but takes 1.6 times as long, so the adaptive system lies between them. Even so, no model or system is yet fast, cheap, and reliable at once. RoboTeam measures these limits where earlier benchmarks cannot: long multi-arm jobs, camera images only, and a world that never pauses.

## TUSI: A Citation-Grounded Graph-Based Benchmark

DevFeed: [TUSI: A Citation-Grounded Graph-Based Benchmark](<https://devfeed.tech/articles/tusi-a-citation-grounded-graph-based-benchmark-139913.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21823>)

Author: Marzieh Karimi, Mehran Safayani, Alireza Basiri

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Graphs](<https://devfeed.tech/topics/graphs.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [graph](<https://devfeed.tech/tags/graph.md>), [grounded](<https://devfeed.tech/tags/grounded.md>), [law](<https://devfeed.tech/tags/law.md>), [legal](<https://devfeed.tech/tags/legal.md>), [rag](<https://devfeed.tech/tags/rag.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>)

### AI overview

TUSI-Bench evaluates retrieval-augmented generation for Persian legal reasoning using a framework of 7,261 legal articles drawn from statutes, commentaries, doctrinal writings, and Supreme Court unification rulings. Its legal knowledge graph links articles and concepts to support structure-aware retrieval. The benchmark includes TUSI-QA, with 1,999 open-ended questions requiring explicit legal citations, and TUSI-Bar, with about 500 four-option questions derived from Iranian Bar exams. A graph-based retrieval baseline improved recall by up to 8 percentage points over standard baselines. The abstract presents the benchmark as addressing gaps in realism, grounding, and legal-reasoning coverage in earlier resources.

### Source excerpt

arXiv:2611.21823v1 Announce Type: cross Abstract: Evaluating retrieval-augmented generation (RAG) systems in Persian law remains challenging because benchmarks lack realism, grounding, and coverage of legal reasoning. Existing resources are either synthetically generated from statutory texts, lacking natural information needs, or derived from informal legal question--answer sources without explicit linkage to authoritative legal documents, making them unsuitable for rigorous evaluation of grounded legal reasoning. To address this gap, we introduce TUSI-Bench, a structured Persian legal benchmark designed for the evaluation of RAG-based systems. TUSI-Bench is built upon a multi-layered legal knowledge framework containing 7,261 legal articles and integrating codified statutes, legal commentaries, doctrinal writings, and Supreme Court unification rulings. We also construct a legal knowledge graph linking articles and concepts for structure-aware retrieval. Based on this foundation, TUSI-Bench provides two complementary evaluation tasks: (i) TUSI-QA, an open-ended question answering benchmark containing 1,999 questions that requires grounded responses with explicit legal citations, and (ii) TUSI-Bar, a multiple-choice benchmark containing approximately 500 four-option questions derived from Iranian Bar examination questions. Furthermore, we propose a graph-based retrieval baseline that leverages relational structure within the legal knowledge graph. Experimental results demonstrate that incorporating graph-based relations improves retrieval recall by up to 8 percentage points compared to standard baselines, highlighting the importance of structured legal representations for RAG systems. The proposed benchmark provides a standardized resource for evaluating grounded legal reasoning in Persian and enables future research on structured legal intelligence.

## Alternating Abduction and Planning over Temporal Dynamic Knowledge Graphs

DevFeed: [Alternating Abduction and Planning over Temporal Dynamic Knowledge Graphs](<https://devfeed.tech/articles/alternating-abduction-and-planning-over-temporal-dynamic-knowledge-graphs-139796.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.27287>)

Author: Amir Hossein Sharafi, Alireza Shahbazi

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [knowledge-engineering](<https://devfeed.tech/topics/knowledge-engineering.md>), [база знаний](<https://devfeed.tech/topics/tag-c51cc4bf76e5.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [actions](<https://devfeed.tech/tags/actions.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lo](<https://devfeed.tech/tags/cs-lo.md>), [decidability](<https://devfeed.tech/tags/decidability.md>), [explain](<https://devfeed.tech/tags/explain.md>), [knowledge-graphs](<https://devfeed.tech/tags/knowledge-graphs.md>), [ontology](<https://devfeed.tech/tags/ontology.md>), [planning](<https://devfeed.tech/tags/planning.md>), [projection](<https://devfeed.tech/tags/projection.md>), [rules](<https://devfeed.tech/tags/rules.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [tools](<https://devfeed.tech/tags/tools.md>)

### AI overview

The paper introduces three tools for planning in Temporal Dynamic Knowledge Graphs (TDKGs), modeled as SROIQ knowledge bases with delete/insert action templates on a linear timeline. Temporal projection tests plans by deriving the updated state's entailments, including indirect effects. DDA explains observations by assuming their negation and negating newly derived consequences. APT plans backward from a goal, adds unstated facts needed by actions, and uses DDA explanations as sub-goals up to a depth bound. The methods access a SROIQ reasoner through a small contract and are designed for incremental handling of added and removed axioms. The authors prove soundness and termination for DDA and APT, and give bounded-planning complexity bounds of 2NExpTime-hard and in 3ExpTime. They report identical results across five reasoners on six benchmarks; the excerpt does not provide performance comparisons.

### Source excerpt

arXiv:2607.27287v4 Announce Type: replace Abstract: Knowledge graphs change over time, but the effects of actions on them are only partly stated: the ontology entails further consequences, facts that an action needs are often unknown, and an action may make the graph inconsistent. We study these problems for Temporal Dynamic Knowledge Graphs (TDKGs), that is, SROIQ knowledge bases whose actions are rules with delete/insert templates and whose events lie on an intrinsic timeline. We present three tools: temporal projection, the exact test of plans; DDA (Direct-Derivation Abduction), which explains an observation by assuming its negation and negating what the reasoner newly derives; and APT (Abduction and Planning over TDKGs), which plans backwards from a goal, assumes the facts that the plan needs but nobody stated, and lets DDA explain the assumptions, each explanation becoming a sub-goal, up to a depth bound. The tools use a SROIQ reasoner only through a small contract, so they depend neither on a particular reasoner nor on a particular data structure; they are designed for a reasoner that processes added and removed axioms incrementally. Change is treated without modal operators, on one linear timeline and without possible worlds, so decidability of projection follows from that of SROIQ. We prove that indirect effects are exactly the entailments of the updated state, that DDA and APT are sound and terminate, and that bounded planning is 2NExpTime-hard and in 3ExpTime. We compare the tools with hitting-set abduction, the solver CATS and classical planning, and run them on six benchmarks and five reasoners with identical results.

## PhysAlign: A Benchmark for Evidence-Grounded Role Alignment in Multimodal Physics Reasoning

DevFeed: [PhysAlign: A Benchmark for Evidence-Grounded Role Alignment in Multimodal Physics Reasoning](<https://devfeed.tech/articles/physalign-a-benchmark-for-evidence-grounded-role-alignment-in-multimodal-physics-reasoning-139821.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.33319>)

Author: Kecheng Liang, Haoyang Liu, Zexin Chen, Zirong Liu, Weixing Chen, Qiufeng Wang, Yang Liu, Liang Lin

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [gaia](<https://devfeed.tech/topics/gaia.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [grounded](<https://devfeed.tech/tags/grounded.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [physics](<https://devfeed.tech/tags/physics.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>)

### AI overview

PhysAlign evaluates whether multimodal models correctly connect recognized physics-diagram elements to the physical entities, relations, or conditions they describe. It separates visual recognition from role assignment using localized probes and controlled variants. The benchmark contains 3,341 human-validated probes across 986 problems and uses five metrics, including CAcc, GAcc, and JAcc, to assess recognition, correspondence, and problem solving. The abstract reports a gap between recognition and grounding: conditional correspondence errors, even when queried content is recognized, are 13.8% for GPT-6-Astra and about 50.6% for InternVL3.5-8B. The results suggest answer-level accuracy can hide errors in assigning diagram evidence its physical role.

### Source excerpt

arXiv:2609.33319v2 Announce Type: replace Abstract: A key challenge in physics diagram understanding is correctly associating visual information with the physical entities, relations, and conditions it describes. Even when a value, symbol, or other local element is accurately recognized, assigning it to the wrong entity or scope can distort the underlying physical premise and lead to incorrect reasoning. To systematically study this challenge, we introduce \textbf{PhysAlign}, a benchmark designed to assess whether multimodal models correctly associate information recognized from physics diagrams with its intended physical role. By disentangling visual recognition from physical-role assignment through localized probes and controlled variants, PhysAlign isolates correspondence errors from recognition failures. It contains 3,341 human-validated probes spanning 986 physics problems, enabling systematic evaluation of visual recognition and physical-role correspondence at scale. We further introduce five complementary evaluation metrics, including CAcc, GAcc, and JAcc, which provide a comprehensive assessment of models' ability to recognize diagram content, establish correct physical correspondences, and solve the underlying physics problem. Across our evaluated multimodal models, PhysAlign reveals a consistent gap between local visual recognition and physical-role grounding. Even when the queried content is correctly recognized, the conditional correspondence error rate remains 13.8\% for GPT-6-Astra and rises to about 50.6\% for InternVL3.5-8B. These findings indicate that strong perception alone does not ensure reliable physical interpretation, exposing a distinct grounding bottleneck that is largely hidden by answer-level accuracy and highlighting the need for future models to better align recognized visual evidence with its physical meaning.

## SciWorkBench: A Benchmark for Real Scientific Workflows

DevFeed: [SciWorkBench: A Benchmark for Real Scientific Workflows](<https://devfeed.tech/articles/sciworkbench-a-benchmark-for-real-scientific-workflows-140037.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22490>)

Author: Yuchen Zhang, Taolin Han, Jinghang Wang, Jinxin Wang, Yun Wu, Wai Yuet Chiu, Xiaogang Li, Yuting Fu, Kewei Sun, Jiaxin Li, Qile Jin, Jiajia Li, Ruodan Chen, Qingteng Chen, Jialu Nie, Junchao Li, Yifei

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI for science](<https://devfeed.tech/topics/ai-for-science.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [biology](<https://devfeed.tech/tags/biology.md>), [chemistry](<https://devfeed.tech/tags/chemistry.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [materials-science](<https://devfeed.tech/tags/materials-science.md>), [physics](<https://devfeed.tech/tags/physics.md>), [scientific-research](<https://devfeed.tech/tags/scientific-research.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

### AI overview

SciWorkBench evaluates whether agents can carry out realistic scientific workflows across biology, chemistry, materials science, and physics. Its tasks come from peer-reviewed literature or expert-contributed scenarios and are assessed under a unified execution and grading protocol. The best-performing agent, evaluated within the Claude Code and Codex frameworks, scored a mean of 57.38. The reported weaknesses were translating scientific knowledge into task-specific procedures, integrating evidence across modalities, and scientifically correcting their own work. The authors argue that workflow-based evaluation is important for measuring progress toward reliable AI research systems.

### Source excerpt

arXiv:2611.22490v1 Announce Type: new Abstract: Language models are increasingly being used as agents capable of planning and executing projects, as well as refining their actions through feedback. This shift raises a question: can current agents complete projects like real scientists? To examine this question, we introduce SciWorkBench, a benchmark for evaluating agents' abilities on real science tasks across biology, chemistry, materials science, and physics. SciWorkBench is built from peer-reviewed literature or expert-contributed research scenarios. We also propose a unified scientific analysis framework validated by domain experts, which can categorize most tasks in SciWorkBench. Each task covers one or more stages of this framework. We evaluate up-to-date models within the Claude Code and Codex agent frameworks under a unified execution and grading protocol. The best-performing agent achieves a mean score of 57.38, indicating that current agents still struggle with realistic scientific workflows. Our failure analysis identifies three recurring weaknesses at the workflow level: difficulty translating scientific knowledge into task-specific procedures, weak integration of evidence across different modalities, and limited capacity for scientific self-correction. These findings highlight the importance of workflow-based evaluation for measuring progress toward reliable AI systems for scientific research.

[Next page](<https://devfeed.tech/topics/benchmark.md?cursor=WyIyMDI2LTExLTE4VDA1OjAwOjAwKzAwOjAwIiwgIjY0MzFjMmVmLWNmZDgtNGJkNi04NjExLTQ3YjliNTFjODU5YyJd>)