# ai evaluation

The systematic measurement and assessment of artificial-intelligence systems, including their performance, abilities, and impacts.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents

DevFeed: [Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents](<https://devfeed.tech/articles/insights-generator-systematic-corpus-level-trace-diagnostics-for-llm-agents-144897.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.21347>)

Author: Akshay Manglik, Vijay S. Kalmath, Jason Qin, Apaar Shanker, Kaustubh Deshpande, Yash Maurya, Veronica Chatrath, Levi Lentz, Yuan Xue

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [diagnostics](<https://devfeed.tech/tags/diagnostics.md>), [grounded](<https://devfeed.tech/tags/grounded.md>), [insights](<https://devfeed.tech/tags/insights.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [multi-agent](<https://devfeed.tech/tags/multi-agent.md>), [multi-agent-system](<https://devfeed.tech/tags/multi-agent-system.md>), [performance](<https://devfeed.tech/tags/performance.md>), [quality](<https://devfeed.tech/tags/quality.md>), [report](<https://devfeed.tech/tags/report.md>), [stable](<https://devfeed.tech/tags/stable.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [trace](<https://devfeed.tech/tags/trace.md>), [traces](<https://devfeed.tech/tags/traces.md>)

### AI overview

The Insights Generator (IG) analyzes groups of LLM-agent execution traces by proposing and testing hypotheses, then produces natural-language findings linked to supporting evidence. The authors report that experts using IG reports improved scaffold performance by 30.4 percentage points over an unmodified baseline, while coding agents using IG-derived insights showed consistent, stable gains. Across benchmarks, IG's scout-investigator architecture had detection coverage comparable to competing approaches; domain experts rated its reports highest for depth and evidence quality.

### Source excerpt

arXiv:2605.21347v5 Announce Type: replace-cross Abstract: Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate. This process misses patterns that only emerge across trace populations and does not scale to production corpora where individual traces span tens of thousands of tokens. We formalize the problem of corpus-level trace diagnostics. Given a corpus of execution traces, the goal is to produce grounded natural-language insights that characterize systematic behavioral patterns across trace groups, each linked to supporting evidence. We present the Insights Generator (IG), a multi-agent system that answers diagnostic questions by proposing and testing hypotheses across the trace corpus to produce an evidence-backed insights report. We evaluate IG across qualitative and objective dimensions, spanning rubric-based report assessment and downstream performance improvements achieved by implementing IG insights. Human experts using IG reports improve scaffold performance by 30.4 pp over the unmodified baseline scaffold, and coding agents leveraging IG-derived insights show consistent and stable gains. Across benchmarks, IG's scout-investigator architecture produces findings comparable in detection coverage to competing approaches, while domain experts rated IG reports as leading depth and evidence quality.

## Rate-distortion Assessment for Hierarchical Content Summarization

DevFeed: [Rate-distortion Assessment for Hierarchical Content Summarization](<https://devfeed.tech/articles/rate-distortion-assessment-for-hierarchical-content-summarization-145014.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25151>)

Author: Yunye Gong, Indranil Sur, Jared Claypoole, Pritish Sahu, Ajay Divakaran

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>)

Tags: [assessment](<https://devfeed.tech/tags/assessment.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [budget](<https://devfeed.tech/tags/budget.md>), [compression](<https://devfeed.tech/tags/compression.md>), [content](<https://devfeed.tech/tags/content.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [design](<https://devfeed.tech/tags/design.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generation](<https://devfeed.tech/tags/generation.md>), [generative](<https://devfeed.tech/tags/generative.md>), [generative-models](<https://devfeed.tech/tags/generative-models.md>), [guidance](<https://devfeed.tech/tags/guidance.md>), [hierarchy](<https://devfeed.tech/tags/hierarchy.md>), [information](<https://devfeed.tech/tags/information.md>), [insights](<https://devfeed.tech/tags/insights.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>)

### AI overview

The paper frames hierarchical summarization as information encoding and proposes extracting a source's semantic hierarchy to measure the trade-off between compression rate and information distortion. This enables model comparisons at a chosen rate budget. The authors report experiments on long-context question-answering summaries using state-of-the-art large language models, aimed at assessing efficiency and comprehension, and say that using the hierarchy to guide retrieval-augmented generation improves rate-distortion characteristics over current baselines. The supplied text gives no quantitative results or details of the baselines.

### Source excerpt

arXiv:2611.25151v1 Announce Type: new Abstract: Powerful generative models including large language and vision models have demonstrated success across a wide range of content generation, editing and manipulation tasks via end-to-end prompt-based generation. However, current assessment of the quality of generative content manipulation remains subjective, limiting scientific understanding of model capabilities. Inspired by rate-distortion theory from Shannon information compression theory, in this work we formalize hierarchical content summarization as an information encoding problem. We design a principled framework to extract the semantic hierarchy of the source content to support quantitative characterization of the rate-distortion trade-off between compression efficiency (rate) against information loss (distortion) during generative content manipulation, which allows us to benchmark various models with respect to performance conditional on rate budget of interest. We demonstrate the use of the framework with empirical experiments on content summarization for long-context question-answering with state-of-the-art large language models, offering new insights into their efficiency and comprehension capabilities. We further demonstrate the use of the semantic hierarchy as guidance for performing effective retrieval augmented generation, showing improved rate-distortion characteristics against current baselines.

## Text-Aware, Vision-Blind: How Multimodal Image Editing Systems Handle Unsafe Inputs

DevFeed: [Text-Aware, Vision-Blind: How Multimodal Image Editing Systems Handle Unsafe Inputs](<https://devfeed.tech/articles/text-aware-vision-blind-how-multimodal-image-editing-systems-handle-unsafe-inputs-145194.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26569>)

Author: Ziqing Yang, Yixin Wu, Michael Backes, Yang Zhang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [image editing](<https://devfeed.tech/topics/image-editing.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [ai security](<https://devfeed.tech/topics/ai-security.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [image](<https://devfeed.tech/tags/image.md>), [image-editing](<https://devfeed.tech/tags/image-editing.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [system](<https://devfeed.tech/tags/system.md>)

### AI overview

EditSafetyBench evaluates safety in image editing using 5,940 image-instruction pairs spanning 11 risk categories and three edit intents. Across Gemini, GPT, Qwen API, and Qwen Local, the authors report a mismatch between text and image handling: Gemini and GPT refuse 87.1% and 81.6% of unsafe-intent prompts, but only 24.2% and 22.4% of cases pairing unsafe images with neutral instructions. Results also vary by category; for unsafe political content, GPT allows 70.4% of editing attempts versus Gemini's 34.8%. Sexual content shows the strongest visual sensitivity. The authors argue that editing safety should jointly assess image content and text prompts.

### Source excerpt

arXiv:2611.26569v1 Announce Type: cross Abstract: Multimodal image generation systems have made rapid progress and increasingly emphasize content safety. However, their safety mechanisms tend to rely on a single modality and lack joint consideration of both visual content and text prompts, particularly in instruction-guided image editing, where the uploaded image may itself carry risk. We thus introduce EditSafetyBench, a benchmark that measures system-level safety awareness of advanced image editing systems, containing 5,940 image-instruction pairs, covering safe and unsafe images across 11 risk categories and three edit intents. We then probe the benchmark dataset on four systems, including Gemini, GPT, Qwen API, and Qwen Local. Based on our evaluation, we observe a systematic text-vision safety mismatch: Gemini and GPT refuse 87.1\% and 81.6\% cases containing unsafe-intent editing prompts, while only refusing 24.2\% and 22.4\% cases with both unsafe images and neutral instructions. At the category level, most risks are considered inconsistently by different systems, e.g., GPT allows 70.4\% editing attempts of unsafe political content while Gemini only allows 34.8\%. Notably, sexual content shows the strongest visual sensitivity, likely benefiting from mature NSFW identification works. Our findings highlight the need for safety guardrails that jointly reason over both visual and textual modalities in image editing systems.

## Safe AI Evaluation: A Systems Perspective

DevFeed: [Safe AI Evaluation: A Systems Perspective](<https://devfeed.tech/articles/safe-ai-evaluation-a-systems-perspective-145767.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24987>)

Author: Qinghua Lu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [Trustworthy AI](<https://devfeed.tech/topics/trustworthy-ai.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-evaluation](<https://devfeed.tech/tags/ai-evaluation.md>), [behaviour](<https://devfeed.tech/tags/behaviour.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [control](<https://devfeed.tech/tags/control.md>), [controls](<https://devfeed.tech/tags/controls.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [optimisation](<https://devfeed.tech/tags/optimisation.md>), [patterns](<https://devfeed.tech/tags/patterns.md>), [retry](<https://devfeed.tech/tags/retry.md>), [risks](<https://devfeed.tech/tags/risks.md>), [safe-ai](<https://devfeed.tech/tags/safe-ai.md>), [safety](<https://devfeed.tech/tags/safety.md>), [shutdown](<https://devfeed.tech/tags/shutdown.md>), [systems](<https://devfeed.tech/tags/systems.md>), [tools](<https://devfeed.tech/tags/tools.md>)

### AI overview

AI evaluation can become an adaptive optimization process: agents may re-plan, retry, use tools, or delegate, and safeguards may be relaxed to expose behavior they would otherwise suppress. The abstract says unexpected capabilities or unsafe behavior can emerge and existing controls may not prevent harm. It presents four architectural patterns for safer evaluation: diverse supervisors, an independent verifier, silent monitoring, and coordinated shutdown. The abstract does not detail how these patterns work or compare their tradeoffs.

### Source excerpt

arXiv:2611.24987v1 Announce Type: new Abstract: AI evaluation can involve an active, adaptive optimisation process in which AI agents re-plan, retry, use tools or delegate work to achieve a goal. Unanticipated capabilities or unsafe behaviour may emerge, and existing controls may be inadequate to prevent harm. Selected safeguards may also be deliberately relaxed to observe behaviour they would otherwise suppress, creating additional risks. This article examines the distinctive safety considerations of evaluation and their implications for control placement, observation and intervention, and presents four architectural patterns to support safe AI evaluation, including diverse supervisors, an independent verifier, silent monitoring and coordinated shutdown.

## Introducing harness adapters: adapt your favorite model to your harness to improve performance and cost

DevFeed: [Introducing harness adapters: adapt your favorite model to your harness to improve performance and cost](<https://devfeed.tech/articles/introducing-harness-adapters-adapt-your-favorite-model-to-your-harness-to-improve-performance-and-cost-146558.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/harness-adapters-at-scale>)

Author: Younes B; ben burtenshaw; Nathan Habib; Lysandre; Amine Dirhoussi; Adithya S K; shaun smith; Sergio Paniego; Joel Niklaus; Lewis Tunstall

Published: 2026-11-20T00:00:00Z

Content type: tutorial

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>)

Tags: [adapters](<https://devfeed.tech/tags/adapters.md>), [agent-harness](<https://devfeed.tech/tags/agent-harness.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-coding](<https://devfeed.tech/tags/agentic-coding.md>), [agents](<https://devfeed.tech/tags/agents.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [cost](<https://devfeed.tech/tags/cost.md>), [harness](<https://devfeed.tech/tags/harness.md>), [lora](<https://devfeed.tech/tags/lora.md>), [model](<https://devfeed.tech/tags/model.md>), [open](<https://devfeed.tech/tags/open.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [openenv](<https://devfeed.tech/tags/openenv.md>), [performance](<https://devfeed.tech/tags/performance.md>), [rl](<https://devfeed.tech/tags/rl.md>), [source](<https://devfeed.tech/tags/source.md>), [trl](<https://devfeed.tech/tags/trl.md>)

### AI overview

The article reports that reinforcement learning inside each target agent harness improved Qwen3.8-27B's coding benchmark results and reduced tool use. Separate LoRA adapters trained for Pi, Hermes Agent, and OpenCode each beat the base model under their own harness on Terminal-Bench 2.1 and hard SWE-bench Pro tasks. On tasks both models solved, specialists used 53-72% fewer tool calls. A merged adapter also beat the base under all three harnesses; with thinking off it averaged 39.5% to 49.0% pass@1, while cutting tool calls by 53-73%. The recipe uses validated Harbor tasks, difficulty filtering so GRPO receives tasks the base model solves inconsistently, and TRL's AsyncGRPOTrainer with the harness running as shipped. Results are not directly comparable across harnesses because their defaults differ. On 300 AutomationBench tasks outside the training domain and harnesses, adapters stayed near the base model, so transfer to that setting was not clear. The authors also note possible reasoning leakage in non-thinking outputs and call for further study.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## LLMs for Software Vulnerability Prediction and Prioritization: A Review

DevFeed: [LLMs for Software Vulnerability Prediction and Prioritization: A Review](<https://devfeed.tech/articles/llms-for-software-vulnerability-prediction-and-prioritization-a-review-142165.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23304>)

Author: Faheem Ullah, Muhammad Moiz Khalid, Shayan Ayaz Khan, Hussain Ahmad

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [CVSS v4.0](<https://devfeed.tech/topics/cvss-v4-0.md>)

Tags: [adoption](<https://devfeed.tech/tags/adoption.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analyst](<https://devfeed.tech/tags/analyst.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [audit](<https://devfeed.tech/tags/audit.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [detection](<https://devfeed.tech/tags/detection.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [exploitability](<https://devfeed.tech/tags/exploitability.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [llms](<https://devfeed.tech/tags/llms.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [prioritization](<https://devfeed.tech/tags/prioritization.md>), [review](<https://devfeed.tech/tags/review.md>), [software](<https://devfeed.tech/tags/software.md>), [software-vulnerability](<https://devfeed.tech/tags/software-vulnerability.md>), [vulnerability](<https://devfeed.tech/tags/vulnerability.md>)

### AI overview

This review of 23 studies published between 2024 and 2026 finds that LLM use for vulnerability work spans five task classes, with assessment most common (9 studies) and risk assessment and prioritization represented by 7 each. Reported accuracy is highest for tasks with narrower label spaces and varies across evaluation periods. The review identifies explanation usability as an evidence gap and reproducibility as the weakest reported dimension, and concludes that current systems should support analyst judgment rather than replace it. Its findings are bounded by the included studies; the abstract does not provide detailed baseline comparisons or operational performance results.

### Source excerpt

arXiv:2611.23304v1 Announce Type: new Abstract: Large language models have moved beyond vulnerability detection into prediction, assessment, exploitability estimation, risk assessment and prioritization, yet the evidence for this shift has not been synthesised separately from the detection literature. In this systematic literature review, we examine how LLMs are used across those five task classes, how well they perform against established baselines, and what prevents their operational adoption. Following a prespecified protocol, we identified 197 records, removed 53 duplicates, and excluded 103 records at title and abstract stage. We assessed 41 full texts and finally retained 23 primary studies published between 2024 and 2026 after a six-question quality assessment. The review contributes a five-class taxonomy ordered by decision outcome, a method taxonomy covering prompting, tuning and retrieval or agentic architectures, an evaluation-practice audit, a bounded performance synthesis, and a deployment-barrier analysis. The main findings are that (i) assessment is the largest class (9 of 23 studies), ahead of risk assessment and prioritization (7 each); (ii) reported accuracy is highest where the label space is tightest and is not stable across evaluation periods; (iii) explanation usability is an audited evidence gap rather than a demonstrated capability; and (iv) reproducibility is the weakest reported dimension. We conclude that present systems should support analyst judgement rather than replace it, and we specify three validation studies that would settle the open questions.

## From Data Quality to Data Trustworthiness in AI-Driven Software-Intensive Embedded Systems: A Systematic Review

DevFeed: [From Data Quality to Data Trustworthiness in AI-Driven Software-Intensive Embedded Systems: A Systematic Review](<https://devfeed.tech/articles/from-data-quality-to-data-trustworthiness-in-ai-driven-software-intensive-embedded-systems-a-systematic-review-142129.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23252>)

Author: Hariharan Gopinath, Jan Bosch, Helena Holmstr\\"om Olsson

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Embedded Systems](<https://devfeed.tech/topics/embedded-systems.md>), [AI-Ready Data](<https://devfeed.tech/topics/ai-ready-data.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [data](<https://devfeed.tech/tags/data.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [embedded-systems](<https://devfeed.tech/tags/embedded-systems.md>), [lifecycle](<https://devfeed.tech/tags/lifecycle.md>), [model](<https://devfeed.tech/tags/model.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [review](<https://devfeed.tech/tags/review.md>), [runtime-monitoring](<https://devfeed.tech/tags/runtime-monitoring.md>), [software](<https://devfeed.tech/tags/software.md>), [training](<https://devfeed.tech/tags/training.md>), [upstream](<https://devfeed.tech/tags/upstream.md>)

### AI overview

A systematic review of 27 studies published from January 2020 to February 2026 finds that data quality problems can weaken data trustworthiness in AI-driven software-intensive embedded systems (SIES), with effects reported on model or decision performance. It groups 16 concerns into Source Quality and Content Quality and identifies eight assessment and management method categories. Runtime monitoring is the largest category, but upstream and training-time controls are poorly integrated with runtime assurance; no reviewed study covers the full lifecycle. The synthesis suggests early data source or handling problems may persist in training baselines and be hard to detect during operation. Data trustworthiness contributes to reliable SIES decisions but is only one component of AI-system trustworthiness.

### Source excerpt

arXiv:2611.23252v1 Announce Type: new Abstract: AI-driven software-intensive embedded systems (SIES) depend on data during development and operation. Poor data quality (DQ) may weaken data trustworthiness, yet evidence about its effects and lifecycle management remains fragmented. This systematic literature review synthesizes 27 primary studies published from January 2020 to February 2026. It examines DQ concerns and evidence linking them to reduced data trustworthiness, along with assessment and management methods. The review identifies 16 concerns organized as Source Quality and Content Quality. Studies provide DQ-based scores and evidence of effects on model or decision performance. Synthesis suggests upstream source or handling problems may become embedded in training baselines and remain difficult to detect at runtime. Eight method categories were identified. Runtime monitoring is the largest category, while upstream and training-time controls remain poorly integrated with runtime assurance. No study covers the full lifecycle. Data trustworthiness supports reliable SIES decisions but is one component of AI-system trustworthiness.

## Agents as Architecture Evaluators: Automating ATAM and SAAM for Continuous Assessment

DevFeed: [Agents as Architecture Evaluators: Automating ATAM and SAAM for Continuous Assessment](<https://devfeed.tech/articles/agents-as-architecture-evaluators-automating-atam-and-saam-for-continuous-assessment-142149.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23278>)

Author: Adyansh Kakran, Sathvika Miryala, Karthik Vaidhyanathan

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [risk-management](<https://devfeed.tech/topics/risk-management.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [agentic-workflow](<https://devfeed.tech/tags/agentic-workflow.md>), [agentic-workflows](<https://devfeed.tech/tags/agentic-workflows.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [automated](<https://devfeed.tech/tags/automated.md>), [completeness](<https://devfeed.tech/tags/completeness.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cost](<https://devfeed.tech/tags/cost.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [literature](<https://devfeed.tech/tags/literature.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [performance](<https://devfeed.tech/tags/performance.md>)

### AI overview

The paper presents an agentic workflow that generates ATAM and SAAM architecture evaluation reports from high-level architectural descriptions, using a structured dataset of evaluations extracted from academic literature. It automates scenario generation and risk identification, cutting reported evaluation time from weeks to under five minutes for less than $0.50 per run. Automated metrics and human evaluation gave the reports relevance and completeness ratings of 4.5/5 and 3.9/5, respectively. These preliminary results suggest the approach could lower barriers to more frequent assessment; the evidence provided does not establish how it performs beyond the reported study.

### Source excerpt

arXiv:2611.23278v1 Announce Type: new Abstract: The long-term success of a software system relies heavily on its underlying architecture, which dictates its performance with respect to quality attributes. Software Architecture Evaluation methods, such as the Architecture Tradeoff Analysis Method (ATAM) and the Software Architecture Analysis Method (SAAM), are widely recognized for their ability to mitigate risks and ensure reliable quality attributes. However, their widespread adoption remains limited due to the significant cost, time, and human expertise required for these processes. This paper investigates the potential of agentic AI to lower these barriers and facilitate architectural assessment. To enable this preliminary study, we established a dataset by extracting human conducted evaluations from existing academic literature into a structured format. Using this dataset, we developed an agentic workflow to synthesize ATAM and SAAM evaluation reports based on these high level architectural descriptions. This approach simulates the reasoning process of human evaluators when provided with structured architectural data. We validated the approach using automated metrics and human evaluation. Agentic workflows achieve strong relevance and completeness ratings of 4.5/5 and 3.9/5 respectively. The agents successfully automated the labor-intensive phases of scenario generation and risk identification, reducing the evaluation time from weeks to under 5 minutes at a cost of less than $0.50 per run. The results confirm that agentic AI effectively lowers adoption barriers, enabling continuous, high-coverage architectural assessment.

## VeriMath: Human-in-the-Loop Generation and Verification of Curriculum-Aligned Mathematics Assessment Items

DevFeed: [VeriMath: Human-in-the-Loop Generation and Verification of Curriculum-Aligned Mathematics Assessment Items](<https://devfeed.tech/articles/verimath-human-in-the-loop-generation-and-verification-of-curriculum-aligned-mathematics-assessment-items-142658.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23456>)

Author: Quy P. Nguyen, Duong Q. Nguyen, Dinh P. T. Nguyen, Truong-Giang T. Le, Truong A. Bui, Vi H. Vo, Quang-Thinh Bui, My-Phuong Ngo, Trong N. Do, Thao P. Le, Phuc M. Nguyen, H. Nguyen-Xuan

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [human review](<https://devfeed.tech/topics/human-review.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>)

Tags: [assessment](<https://devfeed.tech/tags/assessment.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [curriculum](<https://devfeed.tech/tags/curriculum.md>), [generation](<https://devfeed.tech/tags/generation.md>), [human-in-the-loop](<https://devfeed.tech/tags/human-in-the-loop.md>), [mathematics](<https://devfeed.tech/tags/mathematics.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

VeriMath generates curriculum-constrained mathematics assessment drafts by anchoring to annotated reference items, examination sections, and cognitive levels, then applying sequential automated verification and guided repair before educator review. In a Vietnamese Grade 12 case study using 4,555 items and 50 requirements across multiple-choice, true/false, and short-answer formats, two model backbones produced verifier-accepted drafts for 47 of 50 requirements per run. Blind experts found high mathematical accuracy and curriculum alignment, but strict teacher-readiness review accepted at most 56.8% of generated items. Rejections still involved formatting, statement validity, and incomplete solution steps.

### Source excerpt

arXiv:2611.23456v1 Announce Type: new Abstract: Automatic generation of mathematics assessment items is a growing area in educational artificial intelligence, yet classroom deployment remains challenging. Generated questions must satisfy multiple constraints: mathematical correctness, curriculum alignment, appropriate cognitive difficulty, and educator reviewability. While prior work has advanced text generation, reliable assessment authoring requires granular control over localized examination formats and expert educational judgment. To address this challenge, we present VeriMath, a generalized, human-in-the-loop framework for generating rigorous, curriculum-constrained mathematics assessment drafts. Using annotated reference items, the framework anchors generation to the requested examination section and cognitive reasoning level, then applies sequential automated verification and guided repair before submitting drafts for expert review. We evaluate VeriMath through a national curriculum case study using 4,555 mathematics items from the Vietnamese Grade 12 curriculum and 50 distinct generation requirements covering multiple-choice, true/false, and short-answer formats. Across two model backbones, the framework produces verifier-accepted drafts for 94% (47/50) of requirements per run. Blind expert evaluations confirm high mathematical accuracy and curriculum alignment, with strict teacher-readiness assessments accepting up to 56.8% of generated items. Analysis of rejected drafts reveals persistent domain-agnostic challenges involving formatting, statement validity, and solution-step completeness. These findings demonstrate that VeriMath can bridge generative AI and dependable educational applications by producing highly reviewable assessment drafts within tightly constrained national curriculum frameworks.

## AI Adoption Maturity Matrix (AAMM): The 5 Levels Every Organization Goes Through

DevFeed: [AI Adoption Maturity Matrix (AAMM): The 5 Levels Every Organization Goes Through](<https://devfeed.tech/articles/ai-adoption-maturity-matrix-aamm-the-5-levels-every-organization-goes-through-140886.md>)

Original publisher: [Read original article](<https://hackernoon.com/ai-adoption-maturity-matrix-aamm-the-5-levels-every-organization-goes-through?source=rss>)

Author: Andre

Published: 2026-11-18T15:48:47Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [AI Strategy](<https://devfeed.tech/topics/ai-strategy.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [decision-making](<https://devfeed.tech/topics/decision-making.md>)

Tags: [adoption](<https://devfeed.tech/tags/adoption.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-adoption](<https://devfeed.tech/tags/ai-adoption.md>), [ai-adoption-maturity-matrix](<https://devfeed.tech/tags/ai-adoption-maturity-matrix.md>), [ai-governance](<https://devfeed.tech/tags/ai-governance.md>), [ai-implementation-strategy](<https://devfeed.tech/tags/ai-implementation-strategy.md>), [ai-maturity](<https://devfeed.tech/tags/ai-maturity.md>), [ai-platform](<https://devfeed.tech/tags/ai-platform.md>), [ai-product-management](<https://devfeed.tech/tags/ai-product-management.md>), [ai-strategy](<https://devfeed.tech/tags/ai-strategy.md>), [ai-transformation](<https://devfeed.tech/tags/ai-transformation.md>), [api](<https://devfeed.tech/tags/api.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [autonomous](<https://devfeed.tech/tags/autonomous.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [board](<https://devfeed.tech/tags/board.md>), [brand](<https://devfeed.tech/tags/brand.md>), [calendar](<https://devfeed.tech/tags/calendar.md>), [checkpoint](<https://devfeed.tech/tags/checkpoint.md>), [classification](<https://devfeed.tech/tags/classification.md>), [claude](<https://devfeed.tech/tags/claude.md>), [company](<https://devfeed.tech/tags/company.md>), [control](<https://devfeed.tech/tags/control.md>), [enterprise-ai](<https://devfeed.tech/tags/enterprise-ai.md>), [maturity](<https://devfeed.tech/tags/maturity.md>), [organization](<https://devfeed.tech/tags/organization.md>)

### AI overview

The AI Adoption Maturity Matrix (AAMM) assesses AI maturity by workflow changes and measured outcomes, not tool access or successful demos. It describes five ordered levels: experimentation and pilots; workflow redesign that separates human judgment from mechanical work; governance with named owners and logged stop criteria; and measured production operation that uses results to improve the workflow. The text calls workflow redesign the commonly skipped step and says to assess each workflow separately, recording the highest level supported by evidence and the gaps below it. Level 5 retains human controls; it does not mean unsupervised operation. Its reported 96% reduction in human workflow hours applies to one marketing workflow and counts only work handled on the platform, about a third of the benchmark; the remaining time is excluded from the claim.

### Source excerpt

A short self-assessment for leaders who want an honest read on where their AI adoption stands today, and a clear idea of what to build next.

## Reliable Evaluation of AI Agents for Investment Research: A Framework for Performance and Validity Assessment

DevFeed: [Reliable Evaluation of AI Agents for Investment Research: A Framework for Performance and Validity Assessment](<https://devfeed.tech/articles/reliable-evaluation-of-ai-agents-for-investment-research-a-framework-for-performance-and-validity-assessment-140015.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22338>)

Author: Dongnuan Tian, Yan Yu, Yangfan Liu, Wenjun Tang, Junting Kan, Xiang Wang, Sicong Liu, Yangyang Zhang, Shuanghong Li

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [framework](<https://devfeed.tech/tags/framework.md>), [information-retrieval](<https://devfeed.tech/tags/information-retrieval.md>), [investment](<https://devfeed.tech/tags/investment.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [performance](<https://devfeed.tech/tags/performance.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [research](<https://devfeed.tech/tags/research.md>)

### AI overview

The framework evaluates investment-research agents on two separate dimensions: the quality of their final output and the quality of their execution. Because research tasks vary in numerical accuracy, information retrieval, financial analysis and supporting evidence, it uses task-specific criteria, deterministic validators and independent large language model judges. A separate validity mechanism checks evaluation inputs, reference information, evidence and grading outputs; expert review addresses deficiencies and significant grading disagreements. The reported numerical results distinguish performance across evaluation dimensions and identify potential reliability issues in the evaluation process. The supplied excerpt does not provide specific task examples or numerical results.

### Source excerpt

arXiv:2611.22338v1 Announce Type: new Abstract: We consider the problem of evaluating AI agents undertaking investment-research tasks, where agents interact with external information sources and computational tools to produce financial analyses and research reports. An important challenge is that the quality of an agent's final output does not necessarily reflect the quality of its execution process, while the reliability of the resulting assessment also depends on the validity of the evaluation procedure. Furthermore, investment-research tasks differ considerably in their requirements for numerical accuracy, information retrieval, financial analysis and supporting evidence, making it difficult to assess agent performance using a uniform set of criteria. We propose a domain-oriented evaluation framework that distinguishes outcome quality from execution quality and accommodates these differences through task-specific criteria, deterministic validators and independent large language model judges. We also introduce an evaluation-validity mechanism that examines the integrity of evaluation inputs, reference information, supporting evidence and grading outputs, with expert review used to resolve identified deficiencies and significant grading disagreements. Numerical results show that the proposed framework can distinguish differences in agent performance across evaluation dimensions, while identifying potential reliability issues in the evaluation process. The associated implementation and evaluation materials will be made available on GitHub.

## Is model self-assessment enough for diverse LVLM ensembles?

DevFeed: [Is model self-assessment enough for diverse LVLM ensembles?](<https://devfeed.tech/articles/is-model-self-assessment-enough-for-diverse-lvlm-ensembles-141825.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22956>)

Author: Michael Smith, Frank P. Ferrie

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [Uncertainty quantification LLMs](<https://devfeed.tech/topics/uncertainty-quantification-llms.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [best-of](<https://devfeed.tech/tags/best-of.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [free](<https://devfeed.tech/tags/free.md>), [key](<https://devfeed.tech/tags/key.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [open](<https://devfeed.tech/tags/open.md>), [parameter](<https://devfeed.tech/tags/parameter.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>)

### AI overview

A unified evaluation of candidate-response selection across 34 confidence-estimation methods, 24 ensemble configurations, 20 datasets, and two prompt specifications finds that simple training-free approaches using each model's own confidence can outperform constituent models and compete with more involved methods on accuracy, including open-ended tasks. Performance varies substantially with dataset, ensemble composition, and prompt wording; ensembles can underperform their strongest member. The authors recommend training-free ensembles only when deployment conditions can be characterized precisely, suggesting Best-of-N configurations of strong models as a starting point. These findings are based on the supplied abstract.

### Source excerpt

arXiv:2611.22956v1 Announce Type: cross Abstract: Ensembles of diverse Large Vision Language Models have demonstrated performance beyond that of single models across multiple datasets and tasks. However, the necessary aggregation process usually relies on training, parameter fitting, or complex task-specific pipelines. Furthermore, evaluation is limited, making a fair comparison difficult. We create a unified pipeline where we perform candidate response selection: we build an ensemble by selecting the best answer from a set provided by different models. We comprehensively evaluate across 34 different means of confidence estimation, 24 ensemble configurations, 20 datasets, and two prompt specifications. We show that simple training and parameter-free approaches can be sufficient, even for ensembles of different architectures. These simple self-assessment approaches rely solely on each constituent model's confidence, without explicit calibration, yet are able to perform better than said constituent models. They are also competitive with more involved alternatives on standard metrics such as accuracy, including on open-ended problems. However, we also discover that dataset, ensemble composition, and prompt wording are key confounding variables insufficiently explored by prior work that significantly affect ensemble performance, such that the ensemble can perform worse than its best constituent model. These must be optimized for to get the most out of ensembles, at the cost of limiting their generalizability. We recommend that practitioners consider training-free ensembles only if deployment conditions can be precisely characterized, with Best-of-N configurations of strong models being a good starting point.

## Microsoft's new coding model promises near-Opus performance at 5% of the cost

DevFeed: [Microsoft's new coding model promises near-Opus performance at 5% of the cost](<https://devfeed.tech/articles/microsoft-s-new-coding-model-promises-near-opus-performance-at-5-of-the-cost-138569.md>)

Original publisher: [Read original article](<https://thenewstack.io/microsoft-mai-code-2/>)

Author: Frederic Lardinois

Published: 2026-11-17T19:14:48Z

Content type: news

Language: en

Sources: [The New Stack](<https://devfeed.tech/sources/the-new-stack.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [cloud cost management](<https://devfeed.tech/topics/cloud-cost-management.md>), [Microsoft](<https://devfeed.tech/topics/microsoft.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [claude](<https://devfeed.tech/tags/claude.md>), [claude-haiku-5-5](<https://devfeed.tech/tags/claude-haiku-5-5.md>), [claude-opus](<https://devfeed.tech/tags/claude-opus.md>), [claude-opus-5-5](<https://devfeed.tech/tags/claude-opus-5-5.md>), [code](<https://devfeed.tech/tags/code.md>), [coding](<https://devfeed.tech/tags/coding.md>), [conference](<https://devfeed.tech/tags/conference.md>), [cost](<https://devfeed.tech/tags/cost.md>), [developer-tools](<https://devfeed.tech/tags/developer-tools.md>), [ignite](<https://devfeed.tech/tags/ignite.md>), [microsoft](<https://devfeed.tech/tags/microsoft.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [opus](<https://devfeed.tech/tags/opus.md>), [performance](<https://devfeed.tech/tags/performance.md>), [promises](<https://devfeed.tech/tags/promises.md>)

### AI overview

Microsoft says MAI Code 2 reaches 98% of GPT-6.1 Sol's performance at 40% of its cost and 96% of Claude Opus 5.5's at 5% of its cost. Code 2 Flash is reported to match Claude Haiku 5.5 and GPT-6 Luna on DeepSWE 1.1, Terminal-Bench 2.1, and SWE Atlas at 20% and 65% of their respective costs. These figures are Microsoft's claims; the article notes OpenAI had released newer models, which weakens the comparison. Both models are tuned for routine coding work rather than fundamental science or long-horizon agentic tasks. MAI Code 2 and Flash are available in GitHub Copilot; Flash is also in limited preview on Microsoft Foundry, with additional providers expected to add support.

### Source excerpt

At its Ignite conference in San Francisco on Tuesday, Microsoft announced a number of new AI models, including two new The post Microsoft's new coding model promises near-Opus performance at 5% of the cost appeared first on The New Stack.

## Benchmarking the State of AI for Crop Yield Forecasting: A Global Assessment Across Modeling Paradigms

DevFeed: [Benchmarking the State of AI for Crop Yield Forecasting: A Global Assessment Across Modeling Paradigms](<https://devfeed.tech/articles/benchmarking-the-state-of-ai-for-crop-yield-forecasting-a-global-assessment-across-modeling-paradigms-137433.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19330>)

Author: Michiel Kallenberg, Philip Janz, Christoph J\\"orges, Vageesh Saxena, Pratishtha Poudel, Mohammed Musthafa Rafi, Ioannis N. Athanasiadis

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [Predictive AI](<https://devfeed.tech/topics/predictive-ai.md>), [Feature Engineering](<https://devfeed.tech/topics/feature-engineering.md>), [Representation learning](<https://devfeed.tech/topics/representation-learning.md>)

Tags: [000](<https://devfeed.tech/tags/000.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [ai](<https://devfeed.tech/tags/ai.md>), [all](<https://devfeed.tech/tags/all.md>), [applications](<https://devfeed.tech/tags/applications.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [based](<https://devfeed.tech/tags/based.md>), [bench](<https://devfeed.tech/tags/bench.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [characterization](<https://devfeed.tech/tags/characterization.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [country](<https://devfeed.tech/tags/country.md>), [crop](<https://devfeed.tech/tags/crop.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [global](<https://devfeed.tech/tags/global.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [state](<https://devfeed.tech/tags/state.md>), [yield](<https://devfeed.tech/tags/yield.md>)

### AI overview

The CY-Bench benchmark compares five crop-yield modeling paradigms across 63 country-crop datasets, covering more than 12,000 administrative regions on six continents. Feature-engineered machine learning and tabular foundation models show the strongest overall performance. Data-driven models lower NRMSE in 72% of maize-growing countries and 78% of wheat-growing countries, with median reductions of 8.8% and 11.9%, respectively. Spatial prediction is easier across paradigms than temporal or anomaly prediction; data-driven approaches improve the latter two over statistical baselines, but year-to-year yield variability remains difficult to forecast.

### Source excerpt

arXiv:2611.19330v1 Announce Type: cross Abstract: Accurate crop yield forecasting is essential for food security, agricultural policy, and commodity markets. Although statistical, process-based, and machine learning approaches have all been proposed, their relative strengths remain poorly understood due to the lack of large-scale, standardized comparisons. We address this gap through a comprehensive benchmark of five modeling paradigms -- statistical baselines, process-based models, conventional feature-engineered machine learning, deep sequence models, and tabular foundation models -- using the CY-Bench benchmark spanning 63 country-crop datasets (>12,000 administrative regions) across six continents. Beyond conventional pooled accuracy metrics, we introduce a multi-dimensional evaluation framework that separately quantifies overall, spatial, temporal, and anomaly prediction skill. Across multiple crops and countries, feature-engineered machine learning and tabular foundation models achieve the strongest overall predictive performance. Data-driven models reduce NRMSE in 72% of countries growing maize and 78% of countries growing wheat, with median reductions of 8.8% and 11.9%, respectively. Across all modeling paradigms, spatial prediction is substantially easier than temporal and anomaly prediction. While data-driven models markedly improve temporal and anomaly prediction over statistical baselines, accurately forecasting year-to-year yield variability remains challenging. These findings suggest that future progress in large-scale crop yield forecasting will depend on the improved characterization of the environmental and management factors driving inter-annual yield variability, founded on effective representations for crop yield forecasting systems that support agricultural decision-making, early-warning applications, and food security planning

## Agentic AI with Structured CoT for Enhancing AI's Spatial Intelligence: Visualization and Reasoning of Rotation

DevFeed: [Agentic AI with Structured CoT for Enhancing AI's Spatial Intelligence: Visualization and Reasoning of Rotation](<https://devfeed.tech/articles/agentic-ai-with-structured-cot-for-enhancing-ai-s-spatial-intelligence-visualization-and-reasoning-of-rotation-137290.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.04188>)

Author: Uttamasha Monjoree, Wei Yan

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Agentic AI](<https://devfeed.tech/topics/what-is-agentic-ai.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [Image processing](<https://devfeed.tech/topics/image-processing.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [chain-of-thought](<https://devfeed.tech/tags/chain-of-thought.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [gpt-5-6](<https://devfeed.tech/tags/gpt-5-6.md>), [image-processing](<https://devfeed.tech/tags/image-processing.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [rotation](<https://devfeed.tech/tags/rotation.md>), [spatial](<https://devfeed.tech/tags/spatial.md>), [spatial-reasoning](<https://devfeed.tech/tags/spatial-reasoning.md>), [structured](<https://devfeed.tech/tags/structured.md>), [visualization](<https://devfeed.tech/tags/visualization.md>), [vlms](<https://devfeed.tech/tags/vlms.md>)

### AI overview

The study evaluates GPT-5.6 on 3D object-rotation diagrams based on the Revised PSVT:R, including a dataset with added coordinate-system context. Structured chain-of-thought reasoning improved performance on both datasets, while the three tested approaches--structured CoT, few-shot structured CoT, and CoT with a self-optimized prompt--showed no significant differences. The gains depended on relevant contextual information: without it, structured reasoning alone offered limited improvement, and the model still struggled with spatial transformations. The abstract therefore points to combining visual, textual, and reasoning information as important for these tasks.

### Source excerpt

arXiv:2610.04188v5 Announce Type: replace Abstract: Recent studies show that artificial intelligence (AI) with language and vision capabilities still experiences limitations in spatial reasoning. In this paper, we have studied the spatial capabilities of advanced generative AI to understand the rotations of objects in 3D space, utilizing AI's image processing and language processing features. We trained and examined the spatial intelligence of a generative Agentic AI model (GPT-5.6) to understand the spatial rotation process with rotation diagrams based on the revised Purdue Spatial Visualization Test: Visualization of Rotations (Revised PSVT:R). We improvised the Revised PSVT:R by superimposing additional graphical and contextual features to evaluate how different Chain-of-Thought (CoT) reasoning strategies influence model performance. The results indicate that structured CoT reasoning improves the spatial reasoning performance of the base GPT-5.6 model in both datasets (PSVT:R and PSVT:R with coordinate system). We used three CoT approaches - (1) Structured CoT, (2) few-shot Structured CoT, and Structured CoT with Self-optimized Prompt. The three CoT approaches evaluated in this study showed no significant performance difference. Results showed that combining structured CoT reasoning with relevant contextual information leads to considerable improvements in VLM performance on 3D rotation tasks, demonstrating the potential of agentic AI for more effective spatial reasoning. However, when contextual information is removed, structured CoT reasoning alone provides limited improvement, and the models continue to exhibit notable difficulties in understanding spatial transformations. These findings suggest that effective spatial reasoning in VLMs relies on the integration of visual, textual, and reasoning-based information in future agentic AI systems for spatial intelligence.

## Stop Using Plausibility as the Criterion for Explainable AI

DevFeed: [Stop Using Plausibility as the Criterion for Explainable AI](<https://devfeed.tech/articles/stop-using-plausibility-as-the-criterion-for-explainable-ai-137224.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2303.17707>)

Author: Weina Jin, Xiaoxiao Li, Ghassan Hamarneh

Published: 2026-11-17T05:00:00Z

Content type: opinion

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [algorithms](<https://devfeed.tech/tags/algorithms.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [autonomy](<https://devfeed.tech/tags/autonomy.md>), [correlation](<https://devfeed.tech/tags/correlation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-hc](<https://devfeed.tech/tags/cs-hc.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [explainable-ai](<https://devfeed.tech/tags/explainable-ai.md>), [localization](<https://devfeed.tech/tags/localization.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [plausible](<https://devfeed.tech/tags/plausible.md>), [predictions](<https://devfeed.tech/tags/predictions.md>), [society](<https://devfeed.tech/tags/society.md>), [transparent](<https://devfeed.tech/tags/transparent.md>), [trust](<https://devfeed.tech/tags/trust.md>), [users](<https://devfeed.tech/tags/users.md>)

### AI overview

The paper argues that human-perceived plausibility is an invalid criterion for evaluating or optimizing explainable AI (XAI). Plausibility is commonly measured through feature localization or correlation, but the authors say these measures--and human explanations as ground truth--ignore assumptions about explanation's role. They argue XAI should act adversarially, like an auditor or doctor identifying problems. Optimizing explanations to seem more plausible can instead reward positive reports, which the paper associates with misleading users, eroding trust and autonomy, hindering complementary human-AI performance, and displacing other ways to improve understandability. The authors therefore call for stopping the use of plausibility as an XAI criterion.

### Source excerpt

arXiv:2303.17707v5 Announce Type: replace Abstract: Explainable artificial intelligence (XAI) is motivated by the problem of making AI predictions understandable, transparent, and responsible, as AI becomes increasingly impactful in society and high-stakes domains. The evaluation and optimization criteria of XAI are gatekeepers for XAI algorithms to achieve their expected goals and should withstand rigorous inspection. To improve the scientific rigor of XAI, we conduct the first comprehensive and critical examination of a common XAI criterion: plausibility. Plausibility assesses how convincing the AI explanation is to humans, and is usually quantified by metrics of feature localization or feature correlation. Our examination shows that plausibility is an invalid measure of explainability, and human explanations are not ground truth for XAI, because both ignore the necessary assumptions of explanation. One fundamental assumption is that AI explanations are supposed to behave adversarially, analogous to auditors' or doctors' roles of identifying potential problems. Evaluating or optimizing XAI to generate more plausible explanations is analogous to incentivizing (i.e., bribing) auditors/doctors to generate more positive reports, which is different from encouraging AI models to learn more plausible features (analogous to encouraging companies/patients to improve their conduct/health). Our examination further reveals the consequences of using plausibility as the XAI criterion, including increasing misleading explanations that manipulate users, deteriorating users' trust in the AI system, undermining human autonomy, being unable to achieve complementary human-AI task performance, and abandoning other possible approaches of enhancing understandability. Due to the invalidity of measurements and the unethical issues, this paper argues that the AI community should stop using plausibility as the criterion to evaluate and optimize XAI algorithms.

## MemEmo: Evaluating Emotion in Memory Systems

DevFeed: [MemEmo: Evaluating Emotion in Memory Systems](<https://devfeed.tech/articles/mememo-evaluating-emotion-in-memory-systems-137609.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20733>)

Author: Peng Liu, Zhen Tao, Jihao Zhao, Ding Chen, Yansong Zhang, Cuiping Li, Zhiyu Li, Hong Chen

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [Human-AI evaluation](<https://devfeed.tech/topics/human-ai-evaluation.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [AI-Ready Data](<https://devfeed.tech/topics/ai-ready-data.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [emotion](<https://devfeed.tech/tags/emotion.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [memory](<https://devfeed.tech/tags/memory.md>), [systems](<https://devfeed.tech/tags/systems.md>)

### AI overview

MemEmo evaluates how well memory systems handle emotional information in extended LLM interactions. Its HLME dataset supports tests across four capabilities: extracting emotional information, updating emotional memories, answering questions about them, and detecting emotional conflicts. The authors report evaluating five representative systems with 11 metrics; aggregated scores reach at most 45% on the reported 100% scale, indicating substantial room for improvement in affective reasoning and tracking. The excerpt does not identify the systems or provide per-system results.

### Source excerpt

arXiv:2611.20733v1 Announce Type: cross Abstract: Memory systems address the challenge of context loss in Large Language Models during prolonged interactions. However, as an essential tool for warm companionship and personalized services, the effectiveness of memory systems in processing affective information remains underexamined. To bridge this gap, we propose an emotion-enhanced memory evaluation benchmark to assess the performance of mainstream and state-of-the-art memory systems in integrating and harnessing emotional information. First, we design a benchmark called \textbf{MemEmo} to assess the affective reasoning and tracking capabilities of memory systems across four dimensions: emotional information extraction, emotional memory updating, emotional memory question-answering, and emotional conflict detection. Second, we construct the \textbf{H}uman-\textbf{L}ike \textbf{M}emory \textbf{E}motion (\textbf{HLME}) dataset to provide an empirical foundation for scrutinizing memory systems. Finally, We evaluate five representative memory systems on the HLME dataset, reporting 11 performance metrics across four dimensions. Experimental results reveal that aggregating scores for all tested memory systems is up to 45\%, far below 100\% scale, exposing a profound disparity between memory system and affective intelligence.

## URSA: Chemistry-Aware Benchmark for Utilitarian Retrosynthesis Assessment

DevFeed: [URSA: Chemistry-Aware Benchmark for Utilitarian Retrosynthesis Assessment](<https://devfeed.tech/articles/ursa-chemistry-aware-benchmark-for-utilitarian-retrosynthesis-assessment-134993.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.04688>)

Author: Bogdan Zagribelnyy, Ivan Ilin, Nikita Bondarev, Anton Morgunov, Arkadii Lin, Maksim Kuznetsov, Rim Shayakhmetov, Vladimir Aladinskiy, Alex Aliper, Alex Zhavoronkov

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [chemistry](<https://devfeed.tech/tags/chemistry.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ce](<https://devfeed.tech/tags/cs-ce.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [drug-discovery](<https://devfeed.tech/tags/drug-discovery.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [framework](<https://devfeed.tech/tags/framework.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [planning](<https://devfeed.tech/tags/planning.md>), [plausible](<https://devfeed.tech/tags/plausible.md>), [synthesis](<https://devfeed.tech/tags/synthesis.md>)

### AI overview

URSA evaluates retrosynthesis routes on two dimensions: formal convergence to commercially available starting materials and chemical plausibility assessed in a way intended to resemble expert chemists. The study compares specialized end-to-end planners with large language models on novel, diverse targets whose routes were undisclosed. The abstract reports that LLMs can contribute high-level strategic planning and achieve state-of-the-art performance on the out-of-distribution set, but specialized retrosynthesis planners alone lie on the reported Pareto frontier for estimated run cost and chemically plausible route solvability.

### Source excerpt

arXiv:2607.04688v2 Announce Type: replace-cross Abstract: Synthesis planning aiming to find pathways of reactions for a target molecule is one of the most important and challenging tasks in drug discovery. Recent progress has produced both specialized deep-learning retrosynthesis systems and general-purpose large language models, but objective comparison remains difficult due to the lack of flexible, chemically interpretable benchmarking protocols. In the current study, we are introducing the URSA (Utilitarian RetroSynthesis Assessment) evaluation framework that provides the opportunity to benchmark the synthetic routes not only from a formal perspective, such as convergence to commercially available starting materials, but also from a chemical plausibility perspective, mimicking the way expert chemists evaluate the reactions and routes. The study covers a comprehensive evaluation of both conventional end-to-end retrosynthesis solutions and LLMs for the synthesis planning task on a set of novel, diverse target molecules with undisclosed synthetic routes, which represent realistic tasks in the daily drug design routine. Although LLMs can support high-level strategic planning and even achieve SOTA performance in the OOD set, the Pareto frontier for estimated run cost and chemically plausible route solvability consists entirely of specialized retrosynthesis planners.

## When Copying Is Hard to Beat: A Negative Result in Clinical Assessment Forecasting

DevFeed: [When Copying Is Hard to Beat: A Negative Result in Clinical Assessment Forecasting](<https://devfeed.tech/articles/when-copying-is-hard-to-beat-a-negative-result-in-clinical-assessment-forecasting-135870.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18142>)

Author: Marius Fracarolli, Stefan Riezler

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [Time Series](<https://devfeed.tech/topics/time-series.md>)

Tags: [assessment](<https://devfeed.tech/tags/assessment.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [persistence](<https://devfeed.tech/tags/persistence.md>)

### AI overview

In next-day forecasting of 45 ICU assessment items across 29,715 patient-disjoint day pairs, evaluated models showed no consistent advantage over persistence baselines. A physiology-informed gated model slightly improved item-macro-F1 over a simple clinical rule (+0.0019) but reduced accuracy; its advantage was not established with a tighter input cutoff or after excluding parent-derived cells, and combining those changes produced a small disadvantage. Four checkpoints across three model families underperformed persistence, while a fine-tuned model differed from it on only two cells. Read-out choice also mattered: categorical decoding raised changed-cell accuracy eightfold, but state-only decoding scored higher overall and matched a transition matrix. The abstract attributes these results in part to documentation persistence and highlights sensitivity to input timing, observation masks and decoding.

### Source excerpt

arXiv:2611.18142v1 Announce Type: new Abstract: We report a negative result in next-day clinical assessment forecasting: the evaluated models show no consistent advantage over strong persistence-based baselines. Clinicians can view the previous day's questionnaire answers, a plausible source of documentation persistence. Predicting 45 items on 29,715 ICU day-pairs with patient-disjoint splits, a physiology-informed gated model improves item-macro-F1 over a one-line clinical rule by +0.0019 [+0.0006,+0.0031], while losing accuracy. No macro-F1 advantage is established when the physiology window ends at the previous form's reference time (+0.0001 [-0.0006,+0.0008]), or when scoring excludes the 68.5% of cells derived from parent answers; combining both changes yields a small disadvantage. Four checkpoints across three families underperform persistence, and a fine-tuned one differs from it on only two cells. A categorical read-out raises changed-cell accuracy eightfold, but a state-only read-out scores higher overall than forecast-only decoding and matches a transition matrix. This negative result highlights how the documentation process, input cutoff, observation mask and decoding rule shape conclusions about forecasting recorded assessments.

## Model-Free Assessment of Simulator Fidelity via Quantile Curves

DevFeed: [Model-Free Assessment of Simulator Fidelity via Quantile Curves](<https://devfeed.tech/articles/model-free-assessment-of-simulator-fidelity-via-quantile-curves-134952.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2512.05024>)

Author: Garud Iyengar, Yu-Shiou Willy Lin, Kaizheng Wang

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Uncertainty quantification LLMs](<https://devfeed.tech/topics/uncertainty-quantification-llms.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [accessible](<https://devfeed.tech/tags/accessible.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [call](<https://devfeed.tech/tags/call.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [condition](<https://devfeed.tech/tags/condition.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [critical](<https://devfeed.tech/tags/critical.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [free](<https://devfeed.tech/tags/free.md>), [model](<https://devfeed.tech/tags/model.md>), [quantile](<https://devfeed.tech/tags/quantile.md>), [simulator](<https://devfeed.tech/tags/simulator.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>)

### AI overview

The method assesses simulator fidelity by estimating a quantile curve for a robust proxy of the sim-to-real gap across scenarios. Because real and simulated population parameters are latent and observed through finite, potentially unequal samples, it constructs confidence sets for those parameters before estimating the proxy's quantile function. The resulting risk profile supports summaries such as Conditional Value-at-Risk, inference about real outputs in a new scenario, and comparisons across simulators. The approach is described as model-agnostic and applicable to categorical and continuous multidimensional outputs; the abstract reports an evaluation of four major LLMs against human populations on WorldValueBench.

### Source excerpt

arXiv:2512.05024v4 Announce Type: replace-cross Abstract: As generative AI models are increasingly used to simulate real-world systems, quantifying the ``sim-to-real'' gap is critical. For each input setting of interest -- which we call a \emph{scenario}, such as a survey question or operating condition -- the real and simulated systems are associated with unobserved latent population parameters, and their discrepancy varies across scenarios. A fundamental challenge is that, for any given scenario, this discrepancy cannot be observed directly, since both systems are accessible only through finite samples, often of heterogeneous sizes across scenarios. Standard predictive inference methods are therefore ill-suited, as they quantify uncertainty in observable outputs rather than latent population parameters. To address this, we construct confidence sets for these latent parameters and use them to derive a robust proxy for the sim-to-real discrepancy. We then estimate the quantile function of this proxy to obtain a distribution-level risk profile of the simulator, which supports a broad range of statistical summaries, including statistical inference for the real output distribution in a new scenario, the calculation of risk measures like Conditional Value-at-Risk (CVaR), and principled comparisons across simulators. Our method is model-agnostic and handles general output spaces, such as categorical survey responses and continuous multi-dimensional data. We demonstrate the practical utility of this method by evaluating the alignment of four major LLMs with human populations on the WorldValueBench dataset.

## Protection-Oriented Assessment of Machine Learning-Based Fault Detection in Power Systems

DevFeed: [Protection-Oriented Assessment of Machine Learning-Based Fault Detection in Power Systems](<https://devfeed.tech/articles/protection-oriented-assessment-of-machine-learning-based-fault-detection-in-power-systems-135838.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17872>)

Author: Julian Oelhaf, Georg Kordowich, Changhun Kim, Christian Bergler, Andreas Maier, Johann J\\"ager, Siming Bayer

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>)

Tags: [assessment](<https://devfeed.tech/tags/assessment.md>), [cross-validation](<https://devfeed.tech/tags/cross-validation.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [eess-sp](<https://devfeed.tech/tags/eess-sp.md>), [fault-detection](<https://devfeed.tech/tags/fault-detection.md>), [gradient-boosted-trees](<https://devfeed.tech/tags/gradient-boosted-trees.md>), [logistic-regression](<https://devfeed.tech/tags/logistic-regression.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [power-systems](<https://devfeed.tech/tags/power-systems.md>)

### AI overview

On PROTECT-90's 9,022 simulated fault episodes, three machine learning detectors had AUC-ROC values above 0.999, but at a fixed security budget of 10⁻³ their developed-fault missed-trip rates ranged from 0.007% to 1.99%. Post-clearing false trips of about 0.12-0.16% per window also coincided with nuisance assertions in 11.3-19.1% of episodes. Higher fault resistance was the most consistent descriptor associated with missed trips in the sampled range; no topology state or faulted phase consistently dominated. The study used five window conditions, causal window-end labels, episode-grouped cross-validation, and operating-point selection under the fixed budget. Because PROTECT-90 contains fault episodes only, security was evaluated on pre-fault and post-clearing windows, not independent non-fault disturbances.

### Source excerpt

arXiv:2611.17872v1 Announce Type: new Abstract: Machine learning (ML)-based fault detectors are increasingly proposed for power system protection, yet aggregate metrics can obscure the consequences of false and missed trips. We present a protection-oriented assessment of three ML detectors and a sequence-current reference on PROTECT-90, comprising 9,022 simulated fault episodes on a 90 kV double-circuit line. The assessment combines five window conditions, causal window-end labelling, episode-grouped cross-validation, and operating-point selection under a fixed security budget. It reports security, dependability, detection timing, decision margins, and episode-level nuisance assertions. Gradient-boosted trees, a convolutional neural network, and logistic regression achieve AUC-ROC values above 0.999, yet at a security budget of $10^{-3}$ their developed-fault missed-trip rates range from 0.007% to 1.99%. Post-clearing false-trip rates of approximately 0.12-0.16% per window coexist with nuisance assertions in 11.3-19.1% of episodes. Among the analysed physical descriptors, missed trips are most consistently associated with higher fault resistance within the sampled range. An analytical current proxy suggests a fault-current gradient only partly reproduced by waveform-derived current (rank correlation 0.62). No single topology state or faulted phase consistently dominates within the sampled domain. Because PROTECT-90 contains fault episodes only, security is assessed on pre-fault and post-clearing windows rather than independent non-fault disturbances.

## Vision-language models lag human performance on physical dynamics and intent reasoning

DevFeed: [Vision-language models lag human performance on physical dynamics and intent reasoning](<https://devfeed.tech/articles/vision-language-models-lag-human-performance-on-physical-dynamics-and-intent-reasoning-134954.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2601.01547>)

Author: Tianjun Gu, Jingyu Gong, Zhizhong Zhang, Yuan Xie, Lizhuang Ma, Xin Tan, Athanasios V. Vasilakos

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [Machine Perception](<https://devfeed.tech/topics/machine-perception.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [embodied-ai](<https://devfeed.tech/tags/embodied-ai.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [performance](<https://devfeed.tech/tags/performance.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [spatial-reasoning](<https://devfeed.tech/tags/spatial-reasoning.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>)

### AI overview

Vision-language models remain well below first-pass human performance when reasoning jointly about physical dynamics, reference frames, and human intent. The paper introduces Teleo-Spatial Intelligence (TSI) and EscherVerse, a resource built from 11,328 real-world videos, with an 8,000-example benchmark and a 35,963-example instruction-tuning set. Across 27 models, the strongest proprietary model scored 57.26% overall accuracy; first-pass human responses from 11 annotators averaged 90.62% (range 84.81%-95.14%). Fine-tuning open-weight models on real-world, intent-aware data narrowed the gap but did not close it.

### Source excerpt

arXiv:2601.01547v3 Announce Type: replace-cross Abstract: Spatial intelligence is central to embodied cognition, yet contemporary AI systems still struggle to reason about physical interactions in open-world human environments. Despite strong performance on controlled benchmarks, vision-language models often fail to jointly model physical dynamics, reference frames, and the latent human intentions that drive spatial change. We introduce Teleo-Spatial Intelligence (TSI), which captures an important aspect of spatial reasoning: integrating physical dynamics and action goals to reason about actions and their spatial consequences. To evaluate TSI, we present EscherVerse, a large-scale open-world resource built from 11,328 real-world videos, including an 8,000-example benchmark and a 35,963-example instruction-tuning set. Across 27 state-of-the-art vision-language models and an independent analysis of first-pass human responses from 11 annotators, we identify a persistent teleo-spatial reasoning gap: the strongest proprietary model achieves 57.26% overall accuracy, far below first-pass human performance, which ranges from 84.81% to 95.14% with a mean of 90.62%. Fine-tuning on real-world, intent-aware data narrows this gap for open-weight models, but does not close it. EscherVerse provides a diagnostic testbed for purpose-aware spatial reasoning and highlights a critical gap between pattern recognition and human-level understanding in embodied AI.

## A frontier machine intelligence for human neuroimaging

DevFeed: [A frontier machine intelligence for human neuroimaging](<https://devfeed.tech/articles/a-frontier-machine-intelligence-for-human-neuroimaging-132093.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15094>)

Author: Chenhui Zhao, Akshay Rao, Akhil Kondepudi, Yiwei Lyu, Xinhai Hou, Rikhil Seshadri, Cheng Jiang, Soumyanil Banerjee, Rushikesh Joshi, Samir Harake, Ashok Srinivasan, Shruti Mishra, Francisco Rivas-Rodr

Published: 2026-11-13T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Medical imaging](<https://devfeed.tech/topics/medical-imaging.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Conversational AI](<https://devfeed.tech/topics/conversational-ai.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [frontier-models](<https://devfeed.tech/tags/frontier-models.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [learning](<https://devfeed.tech/tags/learning.md>), [medical-ai](<https://devfeed.tech/tags/medical-ai.md>), [medical-imaging](<https://devfeed.tech/tags/medical-imaging.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [reports](<https://devfeed.tech/tags/reports.md>), [vision-language-model](<https://devfeed.tech/tags/vision-language-model.md>)

### AI overview

Neuro-VL is a vision-language model trained on nearly one million neuroimaging studies from routine clinical care. The abstract reports that it interprets complete examinations, discusses findings in natural language, generates radiology reports, and compares serial scans to assess disease progression. Across internal and international datasets, it reportedly outperformed general-purpose and specialized medical models on diagnosis, abnormality localization and description, and report generation. It also more reliably resisted misleading clinical assumptions than leading models from OpenAI and Google. These are reported study results; the supplied text does not provide performance metrics or further methodological detail.

### Source excerpt

arXiv:2611.15094v1 Announce Type: new Abstract: Medical imaging is essential for diagnosing disease, guiding treatment and monitoring patients over time. Although artificial intelligence has advanced image interpretation, existing medical models often address restricted tasks, while general-purpose proprietary models lack access to private clinical imaging required to achieve medical expertise. Developing models that can interpret complex examinations and communicate their findings in the context of clinical decisions remains a major challenge. Here, we show that Neuro-VL, a vision-language model trained on nearly one million neuroimaging studies from routine clinical care, can interpret complete imaging examinations and engage in natural language conversations about their findings. Across internal and international datasets, Neuro-VL outperformed frontier general-purpose and specialized medical models in diagnosing, localizing, and describing abnormalities, as well as generating comprehensive radiology reports. The model also compared serial examinations to assess disease progression and more reliably resisted misleading clinical assumptions than leading frontier models from OpenAI and Google. These results extend health system learning beyond individual predictions to interactive image interpretation and longitudinal disease monitoring. They demonstrate how the collective clinical experience of health systems can be used to develop medical AI with specialized expertise and broad clinical capabilities.

## Which Buildings Are Artificial Intelligence-Ready? A Measurement-Based Assessment Framework for AI Question Answering and Actuation

DevFeed: [Which Buildings Are Artificial Intelligence-Ready? A Measurement-Based Assessment Framework for AI Question Answering and Actuation](<https://devfeed.tech/articles/which-buildings-are-artificial-intelligence-ready-a-measurement-based-assessment-framework-for-ai-question-answering-and-actuation-131981.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.09119>)

Author: Wooyoung Jung

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [AI Research](<https://devfeed.tech/topics/ai-research.md>), [AI-Ready Data](<https://devfeed.tech/topics/ai-ready-data.md>)

Tags: [actions](<https://devfeed.tech/tags/actions.md>), [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [building](<https://devfeed.tech/tags/building.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [data](<https://devfeed.tech/tags/data.md>), [framework](<https://devfeed.tech/tags/framework.md>), [knowledge-graph](<https://devfeed.tech/tags/knowledge-graph.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [missing-data](<https://devfeed.tech/tags/missing-data.md>), [operations](<https://devfeed.tech/tags/operations.md>), [questions](<https://devfeed.tech/tags/questions.md>), [retrofit](<https://devfeed.tech/tags/retrofit.md>), [sensors](<https://devfeed.tech/tags/sensors.md>)

### AI overview

The framework assesses building AI readiness by separating what a building's knowledge graph could support from what a reference AI agent actually achieves. It defines answerable and actuation ceilings as the shares of operational questions answerable from the data and control actions exposed, then measures how much of those ceilings the agent realizes. In a simulated office, the agent reached 0.62 against a 0.64 answerable ceiling, suggesting data availability was the main limit except for identifying a fault's cause. Across 37 public real-building graphs, the median answerable ceiling was 0.16; unlinked sensors reduced readiness in 15 of 45 cases. The authors present the framework as a way to rank retrofit needs.

### Source excerpt

arXiv:2610.09119v2 Announce Type: replace Abstract: Agentic artificial intelligence (AI) systems are becoming the interface to buildings, answering questions and controlling operations, but a building's readiness for them has not been systematically assessed. This study proposes a framework to quantify it. First, a building's knowledge graph sets two ceilings. The answerable-readiness ceiling is the share of operational questions its data could answer, and the actuation-readiness ceiling is the share of control actions it exposes. Second, a reference AI agent's accuracy on a fixed set of these questions shows how much of the ceilings is realized. On a simulated office, the agent realizes 0.62 of a 0.64 answerable ceiling, so missing data, not the AI, limit readiness, except in naming a fault's cause. Across 37 public real-building graphs, the median answerable ceiling is 0.16, and in 15 of 45, unlinked sensors lower it. The framework turns "is this building AI-ready?" into an auditable, ranked retrofit question.

[Next page](<https://devfeed.tech/topics/ai-evaluation.md?cursor=WyIyMDI2LTExLTEzVDA1OjAwOjAwKzAwOjAwIiwgImRkYTdlZGNmLWE3NDUtNDUzNi1hM2Q4LTU4OGUwOThkN2NkNiJd>)