# Interactive AI Systems

Interactive AI systems are human-centered AI systems whose outputs depend on user actions and that require systematic evaluation during development.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## TuringSpace: Grounding Spatial Intelligence in Action with Interactive CAPTCHAs

DevFeed: [TuringSpace: Grounding Spatial Intelligence in Action with Interactive CAPTCHAs](<https://devfeed.tech/articles/turingspace-grounding-spatial-intelligence-in-action-with-interactive-captchas-145049.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25387>)

Author: Xuwei Ding, Skylar Zhai, Linxin Song, Jiarui Feng, Dongqi Fu, Huajun Xi, Junnan Li, Jingcheng Liang, Jieyu Zhao, Yinglong Xia, Hong Li, Jian Kang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Hallucination detection](<https://devfeed.tech/topics/hallucination-detection.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [actions](<https://devfeed.tech/tags/actions.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [captchas](<https://devfeed.tech/tags/captchas.md>), [clicks](<https://devfeed.tech/tags/clicks.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [it](<https://devfeed.tech/tags/it.md>), [k3](<https://devfeed.tech/tags/k3.md>), [kimi-k3](<https://devfeed.tech/tags/kimi-k3.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [results](<https://devfeed.tech/tags/results.md>), [spatial](<https://devfeed.tech/tags/spatial.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [test](<https://devfeed.tech/tags/test.md>)

### AI overview

TuringSpace tests spatial reasoning through action: its 125 synthetic, interactive CAPTCHAs require clicks and drags across five spatial abilities, with each task generated by a program that includes a checker and oracle. Humans solve 94.1%; GPT-6 Astra fails 25.9%, and open-weight Kimi K3 fails 68.0%. The authors attribute most model failures to misreading screenshots and report that reasoning often defends incorrect beliefs with hallucinated evidence. Fine-tuning Qwen3.5-9B on the tasks raises success from 1.6% to 33.1%, above Kimi K3, and improves results on all seven out-of-distribution benchmarks, including a 9.9-point gain on VSI-Bench indoor videos. The abstract reports transfer to real-world scenes; it does not establish broader scope beyond these stated evaluations.

### Source excerpt

arXiv:2611.25387v1 Announce Type: new Abstract: Humans act in the visual world by inferring the 3D layout behind a 2D view, predicting how an action changes it, and correcting themselves after seeing the result. Can multimodal large language models (MLLMs) do the same? CAPTCHAs offer a natural test, since humans solve them through this very loop, clicking and dragging on the screen and adjusting after each result. Yet existing CAPTCHA benchmarks are largely solved and rarely spatial, while spatial benchmarks rarely require action. We therefore present TuringSpace, a synthetic web environment of 125 interactive CAPTCHAs that span five spatial abilities and must be solved through clicks and drags on the screen. Every task is a program that generates new instances with a checker and an oracle, so the tasks also form a scalable training gym. Humans solve 94.1% of the tasks, whereas the best model, GPT-6 Astra, still fails 25.9% of them, and the best open-weight model, Kimi K3, fails 68.0%. Our further analysis shows that most failures come from misreading the screenshot, and that model reasoning rarely corrects a wrong belief but hallucinates evidence to defend it. To test whether such spatial abilities can be learned, we fine-tune Qwen3.5-9B on this gym. Its success rate rises from 1.6% to 33.1%, above Kimi K3, and it improves on all seven out-of-distribution benchmarks, including a 9.9-point gain on the real-world indoor videos of VSI-Bench. These results show that grounding spatial intelligence in screen actions exposes what MLLMs lack and offers a scalable way to teach it, with gains that transfer to real-world scenes. We will release TuringSpace to support further study.

## AGAMOTTO: An AI-Enabled Framework for Time-Critical Tactical Course-of-Action Evaluation and Generation

DevFeed: [AGAMOTTO: An AI-Enabled Framework for Time-Critical Tactical Course-of-Action Evaluation and Generation](<https://devfeed.tech/articles/agamotto-an-ai-enabled-framework-for-time-critical-tactical-course-of-action-evaluation-and-generation-142809.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24568>)

Author: Matthew L. Corbett

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [c2](<https://devfeed.tech/tags/c2.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decision-making](<https://devfeed.tech/tags/decision-making.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generation](<https://devfeed.tech/tags/generation.md>), [research](<https://devfeed.tech/tags/research.md>), [simulation](<https://devfeed.tech/tags/simulation.md>), [time](<https://devfeed.tech/tags/time.md>)

### AI overview

AGAMOTTO is an in-development framework intended to add simulation-based course-of-action (COA) evaluation and generation to command-and-control platforms for time-critical tactical decisions. Developed within the Army's NGC2 modernization effort, which includes systems such as the Tactical Assault Kit, it aims to help commanders explore options during live operations. Initial development and experiments are described as demonstrating feasibility, while also revealing practical constraints of operational AI deployment. The work remains preliminary; the paper outlines remaining research and a training strategy for making AI-generated COAs feasible in tactical scenarios.

### Source excerpt

arXiv:2611.24568v1 Announce Type: new Abstract: Current command and control (C2) platforms lack integrated simulation capabilities that would allow commanders to dynamically explore courses of action (COAs) during live operations. This paper presents ongoing research and contributions to the development of AGAMOTTO (Advanced Guidance and Analytics for Military Operations, Tactical Threats, and Outcomes) as part of the overarching NGC2 program, which advances the Army's efforts to modernize C2 systems, such as the Tactical Assault Kit (TAK). The paper outlines the current design, preliminary findings, and the remaining research tasks, with an emphasis on how the completed work will support tactical decision-making in future time-constrained environments. While the full system is still under development, initial development and results demonstrate the feasibility of AGAMOTTO. Early experiments highlight both the potential benefits and practical constraints of deploying AI-enabled tools in operational settings, and introduce lessons learned and a viable training strategy to make AI-generated COAs feasible in tactical scenarios.

## Movie2Game-Bench: Holistic Evaluation of Film-to-Game Adaptation Beyond Playability

DevFeed: [Movie2Game-Bench: Holistic Evaluation of Film-to-Game Adaptation Beyond Playability](<https://devfeed.tech/articles/movie2game-bench-holistic-evaluation-of-film-to-game-adaptation-beyond-playability-142821.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24669>)

Author: Jialing Xie, Yifan Zhou, Yiwen Zhang, Junjun He

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [interactive storytelling](<https://devfeed.tech/topics/interactive-storytelling.md>), [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>)

Tags: [bench](<https://devfeed.tech/tags/bench.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-mm](<https://devfeed.tech/tags/cs-mm.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [film](<https://devfeed.tech/tags/film.md>), [game](<https://devfeed.tech/tags/game.md>)

### AI overview

Movie2Game-Bench evaluates film-to-game adaptation through general and movie-specific assessments plus playtesting by three GUI model players. Its protocol links source content to gameplay steps and implementation evidence; it covers 22 films and 658 reference items. Across 14 models, playable sequences had the lowest content-presence mean for every model, averaging 37.9/100 versus 54.8 for iconic elements. In judgments of retained emotional-development content, narrative choices accounted for 60.0% and scenario operation 7.2%. The findings point to preserving film situations that can support gameplay and linking retained content to player actions and consequences.

### Source excerpt

arXiv:2611.24669v1 Announce Type: new Abstract: Adapting a film into a game requires selecting source elements and translating them into player actions, rules, and feedback. We introduce Movie2Game-Bench (M2G-bench), a benchmark for evaluating film-to-game adaptation beyond playability. M2G-bench combines general evaluation, movie-specific evaluation, and playtesting by three GUI model players. The protocol connects source content to gameplay steps and implementation evidence, capturing both content preservation and interactive realization. The benchmark comprises 22 films and 658 source-specific reference items. Evaluating 14 models on the same films with their selected primary harnesses reveals complementary strengths in game quality, adaptation, and player experience. A shared weakness concerns playable sequences: this category has the lowest content-presence mean in all 14 models, averaging 37.9 out of 100 compared with 54.8 for iconic elements. Among judgments of retained emotionaldevelopment content, narrative choices account for 60.0% and scenario operation for 7.2%. These findings identify two linked priorities for film-to-game adaptation: preserving source situations that can support gameplay and connecting retained content to player actions and consequences.

## On Evaluating User Simulators for Interactive Agents

DevFeed: [On Evaluating User Simulators for Interactive Agents](<https://devfeed.tech/articles/on-evaluating-user-simulators-for-interactive-agents-143578.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24892>)

Author: Hengguang Zhou, I-Hung Hsu, Li Chen, Priyanshi Shrivastava, Anastasios Angelopoulos, Wei-Lin Chiang, Ion Stoica, Cho-Jui Hsieh

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [harvestbench](<https://devfeed.tech/topics/harvestbench.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>)

Tags: [acceptance](<https://devfeed.tech/tags/acceptance.md>), [agents](<https://devfeed.tech/tags/agents.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [assets](<https://devfeed.tech/tags/assets.md>), [automated](<https://devfeed.tech/tags/automated.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [combines](<https://devfeed.tech/tags/combines.md>), [constraint](<https://devfeed.tech/tags/constraint.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [detection](<https://devfeed.tech/tags/detection.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [failures](<https://devfeed.tech/tags/failures.md>), [frameworks](<https://devfeed.tech/tags/frameworks.md>), [full-stack](<https://devfeed.tech/tags/full-stack.md>), [games](<https://devfeed.tech/tags/games.md>), [goals](<https://devfeed.tech/tags/goals.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [languages](<https://devfeed.tech/tags/languages.md>), [user](<https://devfeed.tech/tags/user.md>)

### AI overview

Existing user-simulator evaluation methods may give misleadingly positive results: metrics from three prior studies detected only 5-6 of 10 deliberate replay failures, and their combined scores favored simulations instructed to accept broken work over unmodified controls. The proposed suite combines agent-evaluation utility with population-level fidelity to collective interaction patterns and trace-level fidelity to individual users' goals, constraints, and persona; it detected all 10 perturbations. In tests of six simulators on 351 real multi-turn sessions involving 348 users across 23 languages, population-level measures sometimes missed wrong goals, ignored constraints, or out-of-character behavior, underscoring the value of assessing fidelity at both levels.

### Source excerpt

arXiv:2611.24892v1 Announce Type: new Abstract: User simulators offer a scalable way to evaluate interactive agents, but how to assess their quality remains underexplored. In particular, it is unclear whether existing user simulator evaluation frameworks reliably detect meaningful failures in simulated user behavior. This uncertainty makes it difficult to interpret simulator evaluation. To understand what a high benchmark score tells us about simulator quality, we (meta-)evaluate the metrics themselves. To instantiate this meta-evaluation, we deliberately perturb simulated replays to induce failures such as altered goals, omitted constraints, premature stopping, and acceptance of broken work, then test whether the metrics penalize these failures. Evaluation methods from three prior studies detect only 5-6 of the 10 perturbations; their combined scores even favor populations instructed to accept broken work over unmodified controls. Motivated by these gaps, we propose a new evaluation suite that combines utility of user simulators for evaluating agents and two measures of fidelity to real users: population-level fidelity assesses collective interaction patterns, while trace-level fidelity assesses the preservation of individual users' goals, constraints, and persona. Under the same detection criterion, our evaluation suite detects all 10 perturbations. Evaluating six user simulators on 351 real multi-turn sessions in which 348 distinct users across 23 languages and agents work together to create and refine artifacts such as full-stack web apps, games, design assets, and automated workflows, we find that population-level measures can miss when a simulated user pursues the wrong goal, ignores a constraint, or behaves out of character, highlighting the need to assess fidelity at comprehensive granularities.

## FVScore: Specification-Driven Neuro-Symbolic Video Evaluation

DevFeed: [FVScore: Specification-Driven Neuro-Symbolic Video Evaluation](<https://devfeed.tech/articles/fvscore-specification-driven-neuro-symbolic-video-evaluation-140065.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22673>)

Author: Do Xuan Long, Long T. Le, Yiwen Song, Min-Yen Kan, Tomas Pfister, Yale Song

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [real-time face tracking](<https://devfeed.tech/topics/real-time-face-tracking.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [open-llm-leaderboard](<https://devfeed.tech/topics/open-llm-leaderboard.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [correlation](<https://devfeed.tech/tags/correlation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generation](<https://devfeed.tech/tags/generation.md>), [grounded](<https://devfeed.tech/tags/grounded.md>), [logic](<https://devfeed.tech/tags/logic.md>), [operators](<https://devfeed.tech/tags/operators.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [report](<https://devfeed.tech/tags/report.md>), [requirements](<https://devfeed.tech/tags/requirements.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

FVSCORE evaluates generated videos by translating prompt requirements and implicit physical, temporal, and cross-modal constraints into propositions, grounding them with specialized perceptual tools, and assessing confidence traces with soft temporal operators inspired by Linear Temporal Logic. Significance-Aware Calibration filters for perceptually significant violations before producing reports tied to timestamps and evidence. The abstract reports stronger correlation with human judgments than holistic and agent-based evaluators across public and complex multi-event benchmarks, alongside interpretable feedback; it provides no specific performance figures.

### Source excerpt

arXiv:2611.22673v1 Announce Type: cross Abstract: We introduce Faithfulness Video Score (FVSCORE), a neuro-symbolic framework for specification-driven video generation evaluation. Existing evaluators rely on holistic MLLM judgments or vision-language metrics, yet aligning their scores with human judgments and tracing them to specific temporal violations remain open challenges. FVSCORE addresses these limitations by compiling each video into explicit propositions capturing prompt requirements and implicit propositions capturing unstated physical, temporal, and cross-modal constraints. It grounds each proposition using specialized perceptual tools and evaluates the resulting confidence traces with soft temporal operators inspired by Linear Temporal Logic. A Significance-Aware Calibration step then aligns verification with human perception by retaining only perceptually significant violations before producing a timestamp- and evidence-grounded report. We conduct extensive experiments across public and complex multi-event benchmarks with the strongest video models to date, demonstrating that FVSCORE outperforms holistic and agent-based evaluators in human correlation while providing highly accurate and interpretable feedback.

## An AI-Driven Assistant for Human-Centric Building Digital Twin Systems: Design, Implementation, and Comparative User Evaluation

DevFeed: [An AI-Driven Assistant for Human-Centric Building Digital Twin Systems: Design, Implementation, and Comparative User Evaluation](<https://devfeed.tech/articles/an-ai-driven-assistant-for-human-centric-building-digital-twin-systems-design-implementation-and-comparative-user-evaluation-140016.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22350>)

Author: Leonardo Messi, Erika Anneli P\\"arn, Juan Diego Casta\\~no Molina, Oleksandr Anyshchenko, Dijar Selami, Peihang Luo, Borja Garc\\'ia de Soto

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [human computer interaction](<https://devfeed.tech/topics/human-computer-interaction.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [User Experience](<https://devfeed.tech/topics/user-experience.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [human centric design](<https://devfeed.tech/topics/human-centric-design.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [ai](<https://devfeed.tech/tags/ai.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [building](<https://devfeed.tech/tags/building.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-hc](<https://devfeed.tech/tags/cs-hc.md>), [digital-twin](<https://devfeed.tech/tags/digital-twin.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [interface](<https://devfeed.tech/tags/interface.md>), [interfaces](<https://devfeed.tech/tags/interfaces.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [new-york](<https://devfeed.tech/tags/new-york.md>), [new-york-university](<https://devfeed.tech/tags/new-york-university.md>), [presentation](<https://devfeed.tech/tags/presentation.md>), [prototype](<https://devfeed.tech/tags/prototype.md>), [study](<https://devfeed.tech/tags/study.md>), [systems](<https://devfeed.tech/tags/systems.md>), [systems-design](<https://devfeed.tech/tags/systems-design.md>), [user](<https://devfeed.tech/tags/user.md>)

### AI overview

The prototype combines a conversational assistant, adaptive information presentation, and auto-suggestions for Indoor Environmental Quality monitoring in a human-centric building digital twin. Its architecture separates generative interaction from deterministic environmental assessment and coordinates answers with visual presentation. In a wave-assigned, non-randomized quasi-experimental comparison at New York University Abu Dhabi, the experimental interface significantly reduced task-completion time and perceived workload. Accuracy and usability improved, but not enough to meet the adjusted significance criterion.

### Source excerpt

arXiv:2611.22350v1 Announce Type: cross Abstract: Smart buildings generate streams of operational data, yet this richness rarely translates into accessible, decision-ready information for stakeholders who inhabit and manage them. Current Building Digital Twin Systems (BDTS) often remain oriented toward technical specialists and rely on static, one-size-fits-all interfaces. This paper presents the design, implementation, and comparative user evaluation of an AI-driven conversational assistant integrated into a human-centric BDTS for Indoor Environmental Quality monitoring, combined with adaptive information presentation and auto-suggest functionality. The prototype was deployed at New York University Abu Dhabi. A controlled, two-arm, between-subjects, quasi-experimental study with wave-based, non-randomized assignment compared the Baseline interface with the Experimental interface. Results showed significant reductions in task-completion time and perceived workload, while accuracy and usability improved but did not meet the adjusted significance criterion. The study contributes an implemented and empirically evaluated architecture that separates generative interaction from deterministic environmental assessment and coordinates conversational responses with adaptive visual presentation.

## EgoLink: Egocentric Language-Vision Interactive Network Knowledge Challenge Summary

DevFeed: [EgoLink: Egocentric Language-Vision Interactive Network Knowledge Challenge Summary](<https://devfeed.tech/articles/egolink-egocentric-language-vision-interactive-network-knowledge-challenge-summary-139896.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21690>)

Author: Renhe Sun, Yueying Feng, Bohan Yu, Tong Niu, Yunqi Liu, Zitong Wang, Haolin He, Kexin Shi, Hualei Wang, Qiuqiang Kong, Chang Yao, Yuge Huang, Weiqiang Wang, Jingyuan Chen, Jian Liu

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Chain-of-thought](<https://devfeed.tech/topics/chain-of-thought.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [actions](<https://devfeed.tech/tags/actions.md>), [agents](<https://devfeed.tech/tags/agents.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [automatic](<https://devfeed.tech/tags/automatic.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [canonicalization](<https://devfeed.tech/tags/canonicalization.md>), [chain-of-thought](<https://devfeed.tech/tags/chain-of-thought.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [choice](<https://devfeed.tech/tags/choice.md>), [combines](<https://devfeed.tech/tags/combines.md>), [complete](<https://devfeed.tech/tags/complete.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [language](<https://devfeed.tech/tags/language.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

EgoLink benchmarks whether egocentric multimodal systems can connect first-person perception and social reasoning with interactive action. Its two tracks assess social reasoning in video and service tasks completed through dialogue, tools, and closed-loop execution. The benchmark combines answer scoring with process and result evaluation; the reported best scores were 0.9263 accuracy in Track 1 and 0.844 joint success with 0.916 micro accuracy in Track 2. Participant reports emphasize making intermediate evidence explicit, using candidate verification and option-robust reasoning for answers, and scenario-aware planning, tool canonicalization, and state-aware correction during actions. The abstract also describes a multi-stage Track 1 pipeline for question generation, reasoning generation, automatic quality grading, and filtering videos by contribution.

### Source excerpt

arXiv:2611.21690v1 Announce Type: cross Abstract: The EgoLink Challenge at ACM Multimedia 2026 benchmarks egocentric multimodal intelligence linking first-person perception, social reasoning, and interactive action. Unlike benchmarks centered on visible objects and activities, EgoLink evaluates whether models can interpret emotional and social cues, resolve visually grounded references, and convert user requests into verified tool-mediated actions. Its two tracks cover social reasoning in egocentric video and interactive agents that complete service tasks through dialogue, tool use, and closed-loop execution. Together, they test whether perception, reasoning, and decision-making remain coupled under both offline question answering and state-changing interaction. We highlight a multi-stage Track 1 data pipeline for multiple-choice question (MCQ) generation, chain-of-thought generation, automatic quality grading, and video-contribution filtering. We summarize the task design, evaluation metrics, rules, leaderboards, and representative solutions. The benchmark combines objective answer scoring with process- and result-based evaluation, enabling analysis of both final correctness and execution quality. The best Track 1 system achieved 0.9263 accuracy, while the best Track 2 system achieved 0.844 joint success and 0.916 micro accuracy. Participant reports reveal a shared principle: strong systems make intermediate evidence explicit through candidate verification, option-robust reasoning, scenario-aware planning, tool canonicalization, and state-aware correction before answering or acting. These findings highlight the importance of process verification and evidence-grounded reasoning for reliable egocentric multimodal intelligence.

## Ask Precisely, Assume Wisely: Interactive Evaluation of Ambiguity Resolution in Large Language Models

DevFeed: [Ask Precisely, Assume Wisely: Interactive Evaluation of Ambiguity Resolution in Large Language Models](<https://devfeed.tech/articles/ask-precisely-assume-wisely-interactive-evaluation-of-ambiguity-resolution-in-large-language-models-137361.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18905>)

Author: Avik Dutta, Aayush Kumar, Zahra Ashktorab, Hosein Hasanbeig, Austin Z. Henley, Gustavo Soares

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [Usability](<https://devfeed.tech/topics/usability.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [framework](<https://devfeed.tech/tags/framework.md>), [human-ai-interaction](<https://devfeed.tech/tags/human-ai-interaction.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [models](<https://devfeed.tech/tags/models.md>), [system-prompts](<https://devfeed.tech/tags/system-prompts.md>), [user-intent](<https://devfeed.tech/tags/user-intent.md>)

### AI overview

The paper introduces a preference-grounded simulated-user framework for evaluating how language models resolve ambiguity through multi-turn interaction. It measures three dissatisfaction signals beyond task accuracy: incorrect assumptions, repeated clarifications, and irrelevant clarifications. The reported results show a trade-off: clarification-heavy models tend to over-clarify and prolong interactions, while assumption-driven models more often miss user intent. System prompts optimized against these signals reportedly balance all three metrics and reduce overall error rates by up to 7.65%. The evidence provided is an abstract, so it does not detail the evaluation setup or results beyond these claims.

### Source excerpt

arXiv:2611.18905v1 Announce Type: new Abstract: Ambiguity is a core challenge in human--AI interaction, where models must decide when to make assumptions and when to seek clarification. Existing approaches to ambiguity evaluation largely focus on coverage over multiple interpretations using static metrics, offering limited insight into how efficiently a model converges toward a single user-intended interpretation through interaction. We introduce a preference-grounded simulated user framework for controlled, multi-turn evaluation of ambiguity resolution, grounding simulator responses strictly in user disambiguation preferences to avoid hallucination and over-disclosure. Alongside this, we define three observable dissatisfaction signals --- incorrect assumptions, repeating clarifications, and irrelevant clarifications --- that characterize interaction quality beyond task accuracy. Our analysis reveals a clear trade-off: models that frequently seek clarification tend to over-clarify and prolong interactions, while assumption-driven models are more prone to misaligning with user intent. We demonstrate that system prompts optimized on these dissatisfaction signals achieve a Pareto-optimal balance across all three metrics, reducing overall error rates by up to 7.65%, underscoring the value of our framework in designing agents that ask precisely and assume wisely.

## AgentFaultBench: State-Verified Evaluation of Tool-Fault Recovery

DevFeed: [AgentFaultBench: State-Verified Evaluation of Tool-Fault Recovery](<https://devfeed.tech/articles/agentfaultbench-state-verified-evaluation-of-tool-fault-recovery-137606.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20711>)

Author: Sachin Gupta

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [ai-experiments](<https://devfeed.tech/topics/ai-experiments.md>)

Tags: [advice](<https://devfeed.tech/tags/advice.md>), [agent](<https://devfeed.tech/tags/agent.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [errors](<https://devfeed.tech/tags/errors.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [execution](<https://devfeed.tech/tags/execution.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [features](<https://devfeed.tech/tags/features.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [files](<https://devfeed.tech/tags/files.md>), [generation](<https://devfeed.tech/tags/generation.md>), [gitlab](<https://devfeed.tech/tags/gitlab.md>), [installation](<https://devfeed.tech/tags/installation.md>), [issue](<https://devfeed.tech/tags/issue.md>), [it](<https://devfeed.tech/tags/it.md>), [langgraph](<https://devfeed.tech/tags/langgraph.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [run](<https://devfeed.tech/tags/run.md>), [sol](<https://devfeed.tech/tags/sol.md>), [state](<https://devfeed.tech/tags/state.md>), [tool](<https://devfeed.tech/tags/tool.md>)

### AI overview

AgentFaultBench evaluates whether recovery advice helps LangGraph and CrewAI agents complete specified tasks after tool errors in an isolated GitLab installation. It verifies resulting files and issues, required reads and checks, and the final response; expected issue and report text is supplied, so the benchmark measures execution rather than content generation. In 1,944 planned primary runs, advice changed success by −8.33, 0.00, and +1.06 percentage points versus a neutral reminder; missing outcomes make the latter two comparisons inconclusive. Under a revised policy, GPT-4o mini improved, while Sol and Claude already completed recoverable cases. On six related tasks, guidance yielded 161 correct final states and one unknown in 192 planned runs, versus 142/192 with neutral feedback, a reported gain of +9.90 to +10.42 points including missing outcomes. Removing issue IDs produced 47/48 successes versus 46/48 with IDs, with other execution information retained. A second checker agreed with 1,879 observed primary outcomes and 241 constructed cases. No human validation was collected.

### Source excerpt

arXiv:2611.20711v1 Announce Type: new Abstract: Does recovery advice help an agent finish a task after a tool fails? AgentFaultBench evaluates LangGraph and CrewAI agents performing specified tasks in an isolated GitLab installation. We inject tool errors and check both the resulting files and issues and the required reads, verification and final response. The prompt supplies the expected issue and report text, so the task tests execution rather than content generation. In the primary comparison of 1,944 planned runs, advice changes success by -8.33, 0.00 and +1.06 percentage points relative to a neutral reminder; missing outcomes leave the latter two comparisons inconclusive. Later experiments test a revised recovery policy. GPT-4o mini improves, while Sol and Claude already complete the recoverable cases. In a separate 624-run study on six related tasks not previously used by that policy, the guided condition records 161 correct final states and one unknown outcome in 192 planned runs, versus 142/192 with neutral feedback. The gain lies between +9.90 and +10.42 pp when missing outcomes are included. Removing issue IDs gives 47/48 successes versus 46/48 with IDs; other execution information remains. A second checker agrees on 1,879 observed primary outcomes and 241 constructed test cases. No human validation was collected. The findings support testing advice against explicit recovery controls rather than assuming it helps.

## DetourBench: Trajectory-Level Evaluation of Interactive Agents with Exact Minimum-Action Bounds

DevFeed: [DetourBench: Trajectory-Level Evaluation of Interactive Agents with Exact Minimum-Action Bounds](<https://devfeed.tech/articles/detourbench-trajectory-level-evaluation-of-interactive-agents-with-exact-minimum-action-bounds-137621.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20844>)

Author: Hanlin Tian, Sihan Zhu, Zhao Yang, Yuxiang Wang, Zhiquan Hu, Yu Mi, Hongquan Zhu, Qiufei Hu

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [after](<https://devfeed.tech/tags/after.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [also](<https://devfeed.tech/tags/also.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [complete](<https://devfeed.tech/tags/complete.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [decision](<https://devfeed.tech/tags/decision.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [environment](<https://devfeed.tech/tags/environment.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evaluation-harness](<https://devfeed.tech/tags/evaluation-harness.md>), [exhaustive](<https://devfeed.tech/tags/exhaustive.md>), [exists](<https://devfeed.tech/tags/exists.md>), [failures](<https://devfeed.tech/tags/failures.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

DetourBench evaluates how efficiently interactive agents gather evidence, not just whether they reach the correct final state. Its finite, partially observable tasks use unit-cost accesses, with resources becoming globally addressable once discovered. Exhaustive search computes the full-information minimum action count and a replayable witness; for graph-recoverable terminal states, it also computes exact residual work. This distinguishes near-complete failures from wandering and records irreversible wrong commitments as having no finite residual distance. The benchmark reports success, attainment, residual progress, and normalized detour separately. Across 375 episodes on 25 tasks, five model endpoints had success rates of 58.7%-96.0% and conditional attainment of 0.803-0.890; attainment ranked models differently. The excerpt says instances, oracle witnesses, and the evaluation harness are open-sourced.

### Source excerpt

arXiv:2611.20844v1 Announce Type: new Abstract: Benchmarks judge language-model agents mainly by whether they reach the correct final state, not by how they got there. Endpoint scoring cannot separate an agent that retrieved exactly the evidence a decision required from one that read most of the environment, nor an agent that never located the evidence from one that stopped a single action short. Settling this needs a reference cost, and authored solutions establish only that a task is solvable, not that no shorter route exists. We present DetourBench, which constrains the environment until that minimum is computable. Each task is a finite, partially observable transition system with unit-cost accesses whose resources become globally addressable after discovery. Exhaustive search yields the full-information minimum action count $L_0$ with a replayable witness; from graph-recoverable terminal states it also yields exact residual work $R_T$, which separates an episode that stopped one access short from one that wandered, while an irreversible wrong commitment is recorded as having no finite residual distance rather than assigned one. Success, attainment, residual progress, and normalized detour are reported separately. Across 375 scored episodes --- five model endpoints, 25 integrated tasks, three repeats --- success ranges from 58.7% to 96.0% while conditional attainment ranges from 0.803 to 0.890 and orders the models differently. Among recoverable failures, residual work separates near-complete trajectories from costly incompletion and from irreversible commitment. We open-source the instances, oracle witnesses, and evaluation harness at https://github.com/DatagridsAI/DetourBench.

## Advancing Cognitive Pairwise Comparisons for Human-Centric Artificial Intelligence and Data Science: A Human-in-the-Loop Framework

DevFeed: [Advancing Cognitive Pairwise Comparisons for Human-Centric Artificial Intelligence and Data Science: A Human-in-the-Loop Framework](<https://devfeed.tech/articles/advancing-cognitive-pairwise-comparisons-for-human-centric-artificial-intelligence-and-data-science-a-human-in-the-loop-framework-137632.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20926>)

Author: Kevin Kam Fung Yuen

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Data Science](<https://devfeed.tech/topics/data-science.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [applications](<https://devfeed.tech/tags/applications.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [automation-bias](<https://devfeed.tech/tags/automation-bias.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-hc](<https://devfeed.tech/tags/cs-hc.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [feature-selection](<https://devfeed.tech/tags/feature-selection.md>), [framework](<https://devfeed.tech/tags/framework.md>), [human-in-the-loop](<https://devfeed.tech/tags/human-in-the-loop.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [learning](<https://devfeed.tech/tags/learning.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [math-oc](<https://devfeed.tech/tags/math-oc.md>), [model-selection](<https://devfeed.tech/tags/model-selection.md>), [recommender-systems](<https://devfeed.tech/tags/recommender-systems.md>), [reduction](<https://devfeed.tech/tags/reduction.md>), [unsupervised-learning](<https://devfeed.tech/tags/unsupervised-learning.md>), [usability](<https://devfeed.tech/tags/usability.md>)

### AI overview

The extended Cognitive Pairwise Comparison (CPC) framework is presented as a structured way to elicit and quantify human preferences in AI and data science tasks that often require repeated human input, including data transformation, feature selection and weighting, and model selection. The abstract says it aims to address limits of fully automated machine learning, such as automation bias and failure to incorporate tacit domain knowledge. Applications using the MovieLens dataset illustrate interactions between human judgments and data across supervised learning, dimensionality reduction, and unsupervised learning. The supplied evidence does not provide details on implementation or comparative performance.

### Source excerpt

arXiv:2611.20926v1 Announce Type: new Abstract: Human-centric artificial intelligence and data science (HCAD) systems increasingly involve human-in-the-loop (HITL) interventions to overcome the limitations of fully automated machine learning algorithms, such as automation bias and the inability to incorporate tacit domain knowledge, and continuously improve predictive performance. Data transformations, feature selection, feature weighting, and model selection usually require laborious human involvement across iterations of artificial intelligence and data science (AI&DS) pipelines, making development a challenging and time-consuming process. The Cognitive Pairwise Comparison (CPC) framework is proposed as a human-centric methodology to elicit preferences and priorities in a structured format. To establish essential facets of human-centricity, this research advances the CPC methodology so that user preference elicitation can be systematically quantified in these challenging AI&DS tasks. To demonstrate the usability and validity of these facets, this study presents several applications within movie recommender systems using the MovieLens dataset. It exemplifies how the extended CPC framework can facilitate interactions between human judgments and data across supervised learning, dimensionality reduction, and unsupervised learning algorithms.

## Interactive Program Induction with Language Model Priors

DevFeed: [Interactive Program Induction with Language Model Priors](<https://devfeed.tech/articles/interactive-program-induction-with-language-model-priors-135205.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18485>)

Author: Robin Shing Moon Chan, Ryan Cotterell

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [few-shot](<https://devfeed.tech/topics/few-shot.md>), [few-shot-learning](<https://devfeed.tech/topics/few-shot-learning.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>)

Tags: [22](<https://devfeed.tech/tags/22.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [fewer](<https://devfeed.tech/tags/fewer.md>), [induction](<https://devfeed.tech/tags/induction.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [learner](<https://devfeed.tech/tags/learner.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [program](<https://devfeed.tech/tags/program.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [user](<https://devfeed.tech/tags/user.md>)

### AI overview

The paper treats program induction as Bayesian inference: a language model supplies a prior over candidate programs, and a greedy learner queries an oracle for outputs on inputs chosen to be maximally informative. Because finite examples can fit many programs, the method uses a PAC-style relaxation of equivalence and importance sampling to estimate query value. The authors state that it terminates after O((1/γ) log(1/p⁰(f⋆))) informative queries and recovers the target up to approximate equivalence when no other class outweighs it under the prior. On CodeARC, it improves accuracy over an LM-only agent by up to 22 percentage points while using up to 10.5 fewer input-output examples.

### Source excerpt

arXiv:2611.18485v1 Announce Type: cross Abstract: Program induction refers to the process in which a learner aims to recover a program from a few given input-output examples. This paradigm broadens programming access by letting users demonstrate the rule-based behavior they want to apply to unseen data, instead of having to implement it themselves. However, recovering a program from finite input-output examples is fundamentally ambiguous, as a finite set of input-output examples is theoretically consistent with infinitely many programs whose output diverges on unseen inputs. We view this problem from a Bayesian perspective, where, given a prior $p^0$ over programs, identifying the target program means concentrating posterior probability mass on it by observing additional input-output examples. In the interactive program induction setting, the learner chooses maximally informative inputs and queries an oracle, such as the user, for their outputs. We derive a greedy query-selection algorithm and show that it terminates after $\mathcal{O}(\frac{1}{\gamma}\log 1/p^0(f^\star))$ informative oracle queries, and returns the target program $f^\star$ up to approximate equivalence when no other class outweighs it under the prior. The algorithm uses a language model as the prior under a PAC-style relaxation of program equivalence, where the query-selection objective is estimated using importance sampling. On the CodeARC benchmark, the algorithm improves accuracy over the LM only agent by up to 22 percentage points while observing up to 10.5 fewer input-output examples.

## Interactive deep-learning based interface for brain tumour segmentation

DevFeed: [Interactive deep-learning based interface for brain tumour segmentation](<https://devfeed.tech/articles/interactive-deep-learning-based-interface-for-brain-tumour-segmentation-135158.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17983>)

Author: Enora Giffard (LTSI), Pierre Jannin (LTSI), John S H Baxter (LTSI)

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Deep learning](<https://devfeed.tech/topics/deep-learning.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [User interface design](<https://devfeed.tech/topics/ui-design.md>), [Usability](<https://devfeed.tech/topics/usability.md>)

Tags: [automatic](<https://devfeed.tech/tags/automatic.md>), [brain](<https://devfeed.tech/tags/brain.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [interface](<https://devfeed.tech/tags/interface.md>), [precision](<https://devfeed.tech/tags/precision.md>), [study](<https://devfeed.tech/tags/study.md>), [usability](<https://devfeed.tech/tags/usability.md>)

### AI overview

Brain tumour segmentation from MR images can fail when tumour characteristics vary or imaging sequences differ across vendors. The paper presents a system that segments automatically by default and lets human users correct its results to support accurate segmentation across patients. It evaluates the automatic approach's accuracy and the interface's usability in a preliminary observational study with experts; the supplied abstract gives no results or performance figures.

### Source excerpt

arXiv:2611.17983v1 Announce Type: new Abstract: Brain tumour segmentation from MR images is an integral part of their treatment, either through surgical resection or radiotherapy. Both applications require a high degree of precision that can be difficult to ensure for all patients due to the high degree of variability in the size, shape, number, and location of these tumours. This leads to difficulties with their automatic segmentation via deep learning, which can display catastrophic segmentation errors when presented with images that differ, even subtly, in terms of the specific MR sequence used, which varies from vendor to vendor. This article presents a deep-learning-based interactive segmentation system that, while being fully automatic by default, has the capability of being easily corrected by human users, allowing them ensure sufficiently high brain tumour segmentation accuracy for all patients. This interface is evaluated both in terms of the accuracy of its automatic approach as well as its usability through a preliminary observational expert user study.

## Generated Evaluation Suites as Diversity-Driven Search for Real Agent Regressions

DevFeed: [Generated Evaluation Suites as Diversity-Driven Search for Real Agent Regressions](<https://devfeed.tech/articles/generated-evaluation-suites-as-diversity-driven-search-for-real-agent-regressions-135090.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17632>)

Author: Pulkit Yadav

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [gaia](<https://devfeed.tech/topics/gaia.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [Dataset development](<https://devfeed.tech/topics/dataset-development.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [archive](<https://devfeed.tech/tags/archive.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [defect](<https://devfeed.tech/tags/defect.md>), [defects](<https://devfeed.tech/tags/defects.md>), [detection](<https://devfeed.tech/tags/detection.md>), [diversity](<https://devfeed.tech/tags/diversity.md>), [domains](<https://devfeed.tech/tags/domains.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [generator](<https://devfeed.tech/tags/generator.md>), [improvement](<https://devfeed.tech/tags/improvement.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [regressions](<https://devfeed.tech/tags/regressions.md>), [requirements](<https://devfeed.tech/tags/requirements.md>), [retail](<https://devfeed.tech/tags/retail.md>), [search](<https://devfeed.tech/tags/search.md>), [specification](<https://devfeed.tech/tags/specification.md>), [tool](<https://devfeed.tech/tags/tool.md>), [trace](<https://devfeed.tech/tags/trace.md>)

### AI overview

The study finds that average pass rates can conceal real agent regressions because a defect can affect only the scenarios within its coverage footprint. In tests of seven controlled defects across two domains, four of five traceable defects had a maximum possible suite-level effect at or below the 0.10 detection threshold. Tool removal destroyed all covered scenarios in both domains, yet the aggregate pass rate moved in the opposite direction. Per-cell scoring recovered that defect and revealed another as a significant improvement; it also detected forced escalation in retail despite zero traced coverage, indicating a behavior-based footprint. Diversity helped when it aligned with the defect: at fixed suite size, a happy-path-only suite detected nothing, while a kind-diverse suite detected an orchestration defect in 5 of 20 draws. The authors recommend scoring evaluation archives per cell and publishing coverage.

### Source excerpt

arXiv:2611.17632v1 Announce Type: new Abstract: Generating an evaluation suite from an agent's specification is diversity-driven search: a generator fills a descriptor space (requirements $\times$ personas $\times$ scenario kinds), and the suite it emits is an archive, then scored like a benchmark by its mean pass rate, which throws away the search's product. We inject 7 controlled defects into a $\tau^2$-bench agent across two domains. Coverage yields a ceiling: if a defect's traced footprint holds $b$ scenarios that pass at baseline, it can itself contribute at most $b/N$ to a suite of $N$. For 4 of 5 traceable defects that ceiling lies at or below the detection threshold $\theta = 0.10$, so no within-footprint effect can move the suite past it; in both domains the tool-removal defect destroys every covered scenario it can (exactly its ceiling) while the suite-level pass rate moves the other way. Defects split into two regimes: trace-aligned ones, where per-cell scoring recovers the tool-removal defect in full and reveals a second as a significant improvement, and trace-orthogonal ones, whose footprint is behavioural: forced escalation is detected in retail at a traced coverage of exactly zero. Diversity along the generator's own axes helps only where it aligns: at fixed suite size a happy-path-only suite detects nothing, while a kind-diverse one detects the orchestration defect in 5 of 20 draws. Evaluation archives should be scored per cell and should publish coverage as a first-class property.

## MOON-RL: Multi-Objective Optimization via Objective Space Navigation with Reinforcement Learning

DevFeed: [MOON-RL: Multi-Objective Optimization via Objective Space Navigation with Reinforcement Learning](<https://devfeed.tech/articles/moon-rl-multi-objective-optimization-via-objective-space-navigation-with-reinforcement-learning-132969.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15227>)

Author: Andrii Artomov, Soheyl Massoudi, J\\"urg Schiffmann

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Operations research and optimization](<https://devfeed.tech/topics/operations-research-and-optimization.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [actuator](<https://devfeed.tech/tags/actuator.md>), [constraint](<https://devfeed.tech/tags/constraint.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [design-tools](<https://devfeed.tech/tags/design-tools.md>), [navigation](<https://devfeed.tech/tags/navigation.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [ppo](<https://devfeed.tech/tags/ppo.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>)

### AI overview

MOON-RL turns constrained multi-objective optimization into preference-directed movement through objective space. A lightweight PPO policy observes objective values and constraints, then updates a design toward a user-specified trade-off. A meta-algorithm periodically resets the policy, draws new starting designs from an evolving archive of non-dominated solutions, and adjusts step sizes and penalties. On actuator-design problems and convergence-hard constrained benchmarks, it was competitive with established evolutionary algorithms in focused search regions under matched evaluation budgets. Ablations indicate both adaptive RL sampling and the meta-algorithm contribute to this behavior. The framework also supports adaptive non-RL samplers, while PPO performed strongly for fixed-target navigation. The supplied abstract does not report specific metrics or establish broad superiority.

### Source excerpt

arXiv:2611.15227v1 Announce Type: new Abstract: Many engineering design problems involve multiple conflicting objectives and strict feasibility constraints. They are typically solved with population-based optimizers that require many evaluations and offer limited online control over where the search is focused. We present MOON-RL, a reinforcement-learning-based optimization framework that formulates constrained multi-objective search as preference-directed navigation in the objective space. In MOON-RL, a lightweight PPO policy receives objective-space and constraint-related observations and acts as an instance-specific adaptive search process. It learns how to update the design so that the current solution advances along a user-specified trade-off direction, while a simple meta-algorithm periodically resets the policy, resamples starting designs from a continuously updated archive of non-dominated solutions, and adjusts step-size and penalty schedules. We evaluate MOON-RL on real-world actuator-design problems and additional convergence-hard constrained benchmarks, comparing against established evolutionary algorithms under matched evaluation budgets. In focused search scenarios, MOON-RL achieves competitive performance in the region of interest, and ablation studies show that both the RL-driven adaptive sampling process and the meta-algorithm contribute to this behaviour, suggesting that repurposing RL training as objective-space navigation is a viable building block for interactive, preference-guided design tools. Additional experiments show that the framework also accommodates adaptive non-RL samplers, while PPO remains a strong choice for fixed-target, preference-directed objective-space navigation.

## Verify by Doing: Co-Evolving Agent Execution and Evaluation without External Feedback

DevFeed: [Verify by Doing: Co-Evolving Agent Execution and Evaluation without External Feedback](<https://devfeed.tech/articles/verify-by-doing-co-evolving-agent-execution-and-evaluation-without-external-feedback-132119.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15276>)

Author: Yitong Lin, Jianda Chen, Yihao Hu, Pan Wang, Hang Wang, Zhihao Wen, Xiaoxue Ren, Wei Wu

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>)

Tags: [actor](<https://devfeed.tech/tags/actor.md>), [agent](<https://devfeed.tech/tags/agent.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [automationbench](<https://devfeed.tech/tags/automationbench.md>), [banking](<https://devfeed.tech/tags/banking.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [doing](<https://devfeed.tech/tags/doing.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [execution](<https://devfeed.tech/tags/execution.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [model-weights](<https://devfeed.tech/tags/model-weights.md>), [verification](<https://devfeed.tech/tags/verification.md>), [verify](<https://devfeed.tech/tags/verify.md>)

### AI overview

Verify by Doing co-evolves an agent's execution and evaluation procedures while keeping model weights fixed. Its Evaluator uses tools to inspect artifacts and environment state, grounding an internal success signal that guides output selection and updates to both procedures. Evaluators are compared on common candidates, while Actor updates must be supported by fresh outputs. On AutomationBench, τ³ Banking, and selected GDPval tasks, VbD scored 3.3, 4.1, and 2.3 percentage points above the strongest compared methods without external feedback, respectively. Ablations support active verification and Evaluator evolution; fixed-candidate selection suggests stronger verification can recover successful outputs without generating new ones.

### Source excerpt

arXiv:2611.15276v1 Announce Type: new Abstract: Agent self-improvement requires feedback about task success, yet external feedback may be unavailable at test time. Agents must then infer success from observable evidence, and incomplete verification can misguide subsequent updates. We introduce Verify by Doing (VbD), a method that co-evolves Actor and Evaluator harnesses within a single task while keeping model weights fixed. The Evaluator actively inspects artifacts and environment state with tools; its findings ground an internal signal for output selection and guide updates to both execution and verification procedures. Proposed Evaluators undergo comparative review on common candidates, while Actor updates require support from their own fresh outputs. Across AutomationBench, $\tau^3$ Banking, and selected GDPval tasks, VbD achieves the highest overall scores among the compared methods operating without external feedback, with gains of 3.3, 4.1, and 2.3 percentage points over the strongest baselines on the main backbone. Ablations support the contributions of active verification and Evaluator evolution, and selection from fixed candidate pools shows that improved verification can recover successful outputs without generating new ones.

## When Both Sides Adapt: The Challenges of Evaluating Interactive LLM Systems

DevFeed: [When Both Sides Adapt: The Challenges of Evaluating Interactive LLM Systems](<https://devfeed.tech/articles/when-both-sides-adapt-the-challenges-of-evaluating-interactive-llm-systems-132324.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17239>)

Author: Hannah Li

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>)

Tags: [a-b-testing](<https://devfeed.tech/tags/a-b-testing.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [effect](<https://devfeed.tech/tags/effect.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [hacking](<https://devfeed.tech/tags/hacking.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [linear](<https://devfeed.tech/tags/linear.md>), [live](<https://devfeed.tech/tags/live.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llms](<https://devfeed.tech/tags/llms.md>), [measuring](<https://devfeed.tech/tags/measuring.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [offline](<https://devfeed.tech/tags/offline.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>)

### AI overview

Users adapt their prompts and tasks in response to model outputs, so evaluation can change the prompt distribution as well as measure model performance. The paper distinguishes the common-distribution effect (comparing models on the same prompt distribution) from the induced-distribution effect (comparing each on prompts generated through interaction). Standard offline benchmark difference-in-means estimates target the former; live A/B test estimates target the latter. These can rank models differently and change deployment recommendations. The authors characterize when rankings reverse and give a sufficient condition for agreement as a heuristic for judging whether offline evaluation can reproduce a live decision. In a stylized linear interaction model, user adaptivity can also increase estimation variance. The paper connects adaptivity to challenges including reward hacking and novelty effects, and argues that evaluation should measure both model performance and user behavior induced by the model.

### Source excerpt

arXiv:2611.17239v1 Announce Type: cross Abstract: In interactive systems where users engage with LLMs, users are adaptive and, after observing LLM responses, can change their prompting strategy, as well as the type and difficulty of subsequent tasks. Thus model responses can influence the set of prompts on which the model is subsequently evaluated. We study the implications of this user adaptivity and prompt endogeneity for the evaluation of LLMs. We define two classes of estimands for comparing models: the common-distribution effect, which compares model performance on the same distribution of prompts, and the induced-distribution effect, which compares each model on the prompt distribution generated through user interaction. We show that offline benchmark evaluation and live A/B testing target different quantities, with the standard difference-in-means estimators for the two methods converging to the common-distribution effect and the induced-distribution effect, respectively. This distinction can affect decision-making: the two comparisons can rank models differently and lead to discrepancies in deployment recommendations. We characterize when such ranking reversals occur and provide a sufficient condition under which the rankings are guaranteed to agree. This result provides a practical heuristic for assessing whether offline evaluation is sufficient to reproduce the decision suggested by a live experiment. Third, we show, in a stylized linear model of interactions, that user adaptivity can increase variance in estimation. Finally, we discuss implications of user adaptivity for the interpretation of experiments and show that adaptivity creates a common mechanism leading to common evaluation challenges, such as reward hacking and novelty effects. Together, this work highlights the necessity of measuring both model performance as well as changes in the user behavior that the model induces.

## REVELIO: Universal Multimodal Task Load Estimation for Cross-Domain Generalization

DevFeed: [REVELIO: Universal Multimodal Task Load Estimation for Cross-Domain Generalization](<https://devfeed.tech/articles/revelio-universal-multimodal-task-load-estimation-for-cross-domain-generalization-128483.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2509.01642>)

Author: Maximilian P. Oppelt, Andreas Foltyn, Nadine R. Lang-Richter, Bjoern M. Eskofier

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Transformers](<https://devfeed.tech/topics/transformers.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>)

Tags: [cross](<https://devfeed.tech/tags/cross.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [domain](<https://devfeed.tech/tags/domain.md>), [load](<https://devfeed.tech/tags/load.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>)

### AI overview

REVELIO evaluates task-load estimation across four settings: a working-memory task, driving simulation, and two games. Its dataset includes 96 sessions of cardiac, electrodermal, muscle, respiration, skin-temperature, eye-tracking, and facial signals collected with consistent hardware and annotation. Multimodal fusion reached up to 0.87 AUROC, compared with 0.83 for eye tracking, 0.74 for facial action units, and 0.62 for biosignals. Multi-domain training remained calibrated on domains represented in training, but excluding the target domain caused classifier collapse in some applications, underscoring a limit to transfer across settings.

### Source excerpt

arXiv:2509.01642v2 Announce Type: replace Abstract: Adaptive interactive systems that respond to user mental state require task load estimation that stays reliable outside its training context. Models built in one setting rarely transfer, limiting real-world use. This work presents Robust Estimation Via End-to-End Learning of Multimodal Observations (REVELIO), a unified framework, dataset and evaluation protocol for cross-domain task load estimation. Cardiac, electrodermal, muscle, respiration, skin temperature, eye tracking and facial signals were recorded in 96 sessions, spanning a laboratory working memory task, a driving simulator and two games under consistent hardware and annotation. Multimodal fusion reached up to 0.87 area under the receiver operating characteristic curve, above single input eye tracking (0.83), facial action units (0.74) and biosignals (0.62). Multi-domain training stayed calibrated on domains seen in training, whereas removing the target domain caused the classifier to collapse in some applications. REVELIO provides a standardized foundation for measuring modality contributions and cross-domain generalization toward deployable workload-aware systems.

## Ninety-Six Ways to Measure a Scribe: Fragmented Evaluation of Ambient Scribes in Healthcare

DevFeed: [Ninety-Six Ways to Measure a Scribe: Fragmented Evaluation of Ambient Scribes in Healthcare](<https://devfeed.tech/articles/ninety-six-ways-to-measure-a-scribe-fragmented-evaluation-of-ambient-scribes-in-healthcare-128416.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.14284>)

Author: Amama Mahmood, Shiye Cao, Laura Fleig, Junan Jiang, Theodore Jensen, Yee-Yin Choong, Kristen Greene, Chien-Ming Huang

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>)

Tags: [ambient](<https://devfeed.tech/tags/ambient.md>), [ambient-ai](<https://devfeed.tech/tags/ambient-ai.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [billing](<https://devfeed.tech/tags/billing.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-hc](<https://devfeed.tech/tags/cs-hc.md>), [dimensions](<https://devfeed.tech/tags/dimensions.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [experience](<https://devfeed.tech/tags/experience.md>), [healthcare](<https://devfeed.tech/tags/healthcare.md>), [impact](<https://devfeed.tech/tags/impact.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [notes](<https://devfeed.tech/tags/notes.md>), [performance](<https://devfeed.tech/tags/performance.md>), [protocols](<https://devfeed.tech/tags/protocols.md>), [research](<https://devfeed.tech/tags/research.md>), [review](<https://devfeed.tech/tags/review.md>), [safety](<https://devfeed.tech/tags/safety.md>), [sources](<https://devfeed.tech/tags/sources.md>)

### AI overview

Across 54 studies evaluating ambient AI scribes, the review finds 40 distinct metrics used in 96 ways, with 35 approaches appearing in only one study. Evaluation centers on documentation efficiency and time (44 studies); safety and clinical outcomes are assessed least consistently. Studies also rarely examine wider care-system effects, including patient outcomes, work redistribution, billing and coding, or other clinicians' use of notes. Heavy reliance on self-report and inconsistent metric definitions make results difficult to compare or synthesize. The authors review study designs, data sources, and measures, and call for a more holistic evaluation using standardized, multidimensional metrics.

### Source excerpt

arXiv:2611.14284v1 Announce Type: cross Abstract: Ambient artificial intelligence (AI) scribes have moved rapidly from experimental pilots to enterprise-wide deployment across health systems, promising reduced documentation burden and improved clinician experience. While research evaluating these tools has expanded quickly, there are no agreed-upon protocols or standardized metrics for assessing their performance, making it difficult to compare findings or cumulatively synthesize evidence across studies. Across 54 recent evaluation studies, we identify ten evaluation dimensions and 40 distinct metrics, operationalized in 96 different ways-35 of them (36%) used by only a single study. Assessment is dominated by documentation efficiency and time, measured in 44 of 54 studies. The other dimensions appear at broadly similar rates (16-24 studies each), and safety and clinical outcome least consistently of all. Ripple effects on the broader care system, such as impact on patients' care outcomes, redistribution of work, changes in billing and coding intensity, and downstream use of notes by other clinicians are rarely assessed. Moreover, the metrics used rely heavily on self-report and are fragmented and inconsistently operationalized, which limits comparability and hinders synthesis across studies. As an effort to support more rigorous and cumulative evaluation, in this paper we review the study designs, data sources, and metrics commonly used to evaluate ambient AI scribes in healthcare settings, discuss the limitations of current evaluation practice, and argue for a more holistic approach grounded in a standardized, multi-dimensional set of metrics.

## Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents

DevFeed: [Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents](<https://devfeed.tech/articles/comparative-analysis-of-agent-evaluation-frameworks-stability-detection-and-discovery-in-enterprise-analytics-agents-128437.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.14493>)

Author: Howard Zhang, Chuck Arvin, Logan Brown, Mihir Bhatia, Neelu Choudhary, Lily Miao

Published: 2026-11-11T05:00:00Z

Content type: comparison

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [Detection engineering](<https://devfeed.tech/topics/detection-engineering.md>), [threat detection](<https://devfeed.tech/topics/threat-detection.md>), [Cryptography](<https://devfeed.tech/topics/cryptography.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [adversarial](<https://devfeed.tech/tags/adversarial.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-evaluation](<https://devfeed.tech/tags/agent-evaluation.md>), [agent-evaluation-frameworks](<https://devfeed.tech/tags/agent-evaluation-frameworks.md>), [agents](<https://devfeed.tech/tags/agents.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [balance](<https://devfeed.tech/tags/balance.md>), [based](<https://devfeed.tech/tags/based.md>), [best](<https://devfeed.tech/tags/best.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [detection](<https://devfeed.tech/tags/detection.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [frameworks](<https://devfeed.tech/tags/frameworks.md>)

### AI overview

In a controlled study of two enterprise analytics agents using frozen execution traces, trace-based LLM judging achieved the strongest balance on 72 known-risk cases (F1=0.90). Text-only LLM judging matched its recall but produced five times more false alarms. Across repeated measurements, 97% of configurations met the stability threshold, with patterns independent of the agent. Functional testing and security red-teaming uncovered 24 novel failures with no overlap, suggesting they find complementary risks. The authors recommend trace-based evaluation for functional assessment and adversarial red-teaming for security validation. These findings come from the study's specific controlled setting.

### Source excerpt

arXiv:2611.14493v1 Announce Type: new Abstract: Deploying LLM-based analytics agents in enterprise settings requires evaluation frameworks that can reliably detect failures across complex, multi-tool workflows. We present a three-phase comparative study of three evaluation frameworks, each representing a distinct evaluation paradigm (trace-based LLM judging, text-only LLM judging with red-teaming, and deterministic heuristics), applied to two analytics agents in a controlled research setting using frozen execution traces. Phase 1 quantifies evaluation harness stability through repeated measurements, finding that 97% of configurations meet the stability threshold and that stability patterns are agent-independent. Phase 2 measures known risk detection on 72 ground-truth test cases, where the trace-based framework achieves the best precision-recall balance (F1=0.90) while text-only evaluation matches on recall but generates 5x more false alarms. Phase 3 evaluates unknown risk discovery, uncovering 24 novel failures with zero overlap between functional testing and security red-teaming. Our results demonstrate that (1) the information available to the evaluation harness determines evaluation quality, (2) functional and security evaluation are complementary, and (3) LLM-based evaluation harnesses surface subtle errors that human reviewers miss. We recommend a two-framework strategy: trace-based evaluation for functional assessment and adversarial red-teaming for security validation.

## JioStar's chief architect on AI discovery at 500M-user scale

DevFeed: [JioStar's chief architect on AI discovery at 500M-user scale](<https://devfeed.tech/articles/jiostar-s-chief-architect-on-ai-discovery-at-500m-user-scale-129535.md>)

Original publisher: [Read original article](<https://openai.com/index/jiostar-vijay-seshadri>)

Published: 2026-11-11T00:00:00Z

Content type: news

Language: en

Sources: [OpenAI News](<https://devfeed.tech/sources/openai-news.md>)

Topics: [AI for interactive applications](<https://devfeed.tech/topics/ai-for-interactive-applications.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [AI interaction best practices](<https://devfeed.tech/topics/ai-interaction-best-practices.md>), [User interface design](<https://devfeed.tech/topics/ui-design.md>), [digital experience monitoring](<https://devfeed.tech/topics/digital-experience-monitoring.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [accelerate](<https://devfeed.tech/tags/accelerate.md>), [ai](<https://devfeed.tech/tags/ai.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [announcement](<https://devfeed.tech/tags/announcement.md>), [api](<https://devfeed.tech/tags/api.md>), [apply](<https://devfeed.tech/tags/apply.md>), [architect](<https://devfeed.tech/tags/architect.md>), [asean](<https://devfeed.tech/tags/asean.md>), [balance](<https://devfeed.tech/tags/balance.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [building](<https://devfeed.tech/tags/building.md>), [but](<https://devfeed.tech/tags/but.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [evals](<https://devfeed.tech/tags/evals.md>), [guardrails](<https://devfeed.tech/tags/guardrails.md>), [production](<https://devfeed.tech/tags/production.md>), [scale](<https://devfeed.tech/tags/scale.md>), [user](<https://devfeed.tech/tags/user.md>)

### AI overview

JioStar built a conversational video and cricket-stat discovery assistant for JioHotstar using the OpenAI API, supporting more than 15 languages in about eight weeks. The team balanced accuracy, latency and cost, with voice used by more than 60% of its users. Its evaluation approach combined a seed dataset expanded with models, domain-expert input and a maker-checker setup in which a second model validated answers; the validator itself also needed evaluation. The team prioritized defining what the system should not answer and putting guardrails in place. For production, Vijay Seshadri says to monitor model efficacy alongside infrastructure measures such as P99 latency, and to rely on skilled people to validate systems: tools such as Codex can accelerate eval generation and updates, but do not replace judgment.

### Source excerpt

JioStar's Vijay Seshadri explains AI discovery at 500M-user scale. Read his lessons on evals, guardrails, and taking AI into production.

## Actions Should EARN Their Credit: Test-Time Adaptation and Route Consolidation for Frozen LLM Agents

DevFeed: [Actions Should EARN Their Credit: Test-Time Adaptation and Route Consolidation for Frozen LLM Agents](<https://devfeed.tech/articles/actions-should-earn-their-credit-test-time-adaptation-and-route-consolidation-for-frozen-llm-agents-125414.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11565>)

Author: Bian Sun, Kevin Zhai, Mubarak Shah, Song Wang

Published: 2026-11-10T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [actions](<https://devfeed.tech/tags/actions.md>), [agents](<https://devfeed.tech/tags/agents.md>), [also](<https://devfeed.tech/tags/also.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [best](<https://devfeed.tech/tags/best.md>), [budget](<https://devfeed.tech/tags/budget.md>), [can](<https://devfeed.tech/tags/can.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [effective](<https://devfeed.tech/tags/effective.md>), [experience](<https://devfeed.tech/tags/experience.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [route](<https://devfeed.tech/tags/route.md>), [test](<https://devfeed.tech/tags/test.md>), [time](<https://devfeed.tech/tags/time.md>)

### AI overview

EARN helps frozen LLM agents improve through repeated attempts without changing model parameters. It reduces credit for actions that keep reaching previously attained scores and increases credit for action segments that achieve higher scores, adjusting the probabilities of LLM-proposed actions. In final episodes, it replays stored actions directly when the current context matches and those actions remain available, bypassing candidate generation. On average score across the final four episodes, EARN had the best result, including ties, in 31 of 33 task-budget settings across Jericho, ScienceWorld, and Crafter. In evaluated Jericho budgets, replay bypassed 78.9-86.3% of candidate-generation calls in those final episodes, averaged across games.

### Source excerpt

arXiv:2611.11565v1 Announce Type: new Abstract: LLM agents can use feedback from repeated attempts to improve their decisions without updating model parameters. However, positive action credits can continue to favor actions that repeatedly achieve the same score, even when higher scores are achievable. Under a fixed interaction budget, trying new actions in the final attempts can also lower performance. We introduce EARN, a method that uses past experience to guide further progress and retain earlier performance through conditional reuse of effective action sequences. EARN reduces action credits for reaching scores repeatedly attained in earlier episodes and assigns larger credits to action segments that reach higher scores. These credits adjust the probabilities of LLM-proposed actions. During the final episodes, EARN directly replays stored actions when the current context matches the recorded context and the actions remain available, bypassing LLM candidate generation. We evaluate performance at the end of the interaction budget using the average score over the final four episodes. Across Jericho, ScienceWorld, and Crafter, EARN achieves the best score on this metric in 31 of 33 task--budget settings among the evaluated methods, including ties. Across the evaluated Jericho budgets, replay bypasses 78.9--86.3\% of candidate-generation calls in the final four episodes, averaged across games.

## Interactive Person Retrieval with On-the-Fly Visual Feedback

DevFeed: [Interactive Person Retrieval with On-the-Fly Visual Feedback](<https://devfeed.tech/articles/interactive-person-retrieval-with-on-the-fly-visual-feedback-126035.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10869>)

Author: Lyucheng Qian, John Yuehan Zhang, Pingyu Wang

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [information retrieval](<https://devfeed.tech/topics/information-retrieval.md>), [AI-driven information retrieval](<https://devfeed.tech/topics/ai-driven-information-retrieval.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>)

Tags: [candidate](<https://devfeed.tech/tags/candidate.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generation](<https://devfeed.tech/tags/generation.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>)

### AI overview

The paper proposes visual-feedback interactive person retrieval (VFI-PR), where retrieved candidates provide evidence for subsequent questions and re-ranking. This addresses a limitation of existing interactive retrieval methods, which mainly learn question orderings offline and rely on answers from witness-side memory. The authors introduce the VFI-PEDES dataset and evaluation protocol, and VFI-ReID, a framework combining a Dynamic Accumulated-evidence Retriever (DAR) with retrieval-aware question optimization to adapt retrieval and question generation as evidence changes. The abstract reports that VFI-ReID improves retrieval across interaction stages and outperforms existing interactive person retrieval baselines.

### Source excerpt

arXiv:2611.10869v1 Announce Type: cross Abstract: Traditional text-based person retrieval (TPR) assumes that a witness provides a complete description at once, whereas interactive person retrieval (Inter-PR) refines a partial description through question-answer exchanges. In this setting, the questioner inspects retrieved candidates to decide what evidence to request next, while the answerer sees only the witness-side memory and cannot report candidate-specific visual differences. Existing methods mainly learn offline question orderings, leaving this visual-feedback loop underexplored. To address this limitation, we introduce visual-feedback interactive person retrieval (VFI-PR), a new interactive retrieval paradigm in which retrieved candidates produce visual feedback that becomes evidence for subsequent questioning and re-ranking. To support training and evaluation in this setting, we construct VFI-PEDES, a dataset and evaluation protocol for visual-feedback interaction, and propose VFI-ReID, a framework that learns to use visual feedback during interactive person retrieval. Through the Dynamic Accumulated-evidence Retriever (DAR) and retrieval-aware question optimization, VFI-ReID adapts retrieval and question generation to evolving evidence states, turning visual feedback into complementary evidence for subsequent retrieval. Experiments show that VFI-ReID consistently improves retrieval across VFI-PR interaction stages, demonstrating the utility of visual feedback for interactive person retrieval and outperforming existing Inter-PR baselines.

## Joi-Bench-Task: Benchmarking Task Coordination in Full-duplex Voice Agents through Interactive Simulation

DevFeed: [Joi-Bench-Task: Benchmarking Task Coordination in Full-duplex Voice Agents through Interactive Simulation](<https://devfeed.tech/articles/joi-bench-task-benchmarking-task-coordination-in-full-duplex-voice-agents-through-interactive-simulation-126232.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.12118>)

Author: Xiangheng He, Mingjun Zhao, Zhehao Fan, Zhengjia Li, Xiaopeng Sun, Qingqing Xiao, Haolan Chen, Yu Xu

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Conversational AI](<https://devfeed.tech/topics/conversational-ai.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [task management](<https://devfeed.tech/topics/task-management.md>), [Simulation](<https://devfeed.tech/topics/simulation.md>)

Tags: [adapting](<https://devfeed.tech/tags/adapting.md>), [agents](<https://devfeed.tech/tags/agents.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [changes](<https://devfeed.tech/tags/changes.md>), [completion](<https://devfeed.tech/tags/completion.md>), [complex](<https://devfeed.tech/tags/complex.md>), [conversations](<https://devfeed.tech/tags/conversations.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [dimensions](<https://devfeed.tech/tags/dimensions.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [execution](<https://devfeed.tech/tags/execution.md>), [full](<https://devfeed.tech/tags/full.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [simulation](<https://devfeed.tech/tags/simulation.md>), [voice](<https://devfeed.tech/tags/voice.md>), [voice-agents](<https://devfeed.tech/tags/voice-agents.md>)

### AI overview

Joi-Bench-Task evaluates how full-duplex voice agents coordinate ongoing tasks with conversation as users change requests or interrupt. Its interactive simulator schedules configurable interventions during tool execution and spoken result delivery, enabling reproducible tests of execution responsiveness and coordination under changes and interruptions. Across five commercial agents, the highest result completion rate after request changes was 55%; four agents had average information omission rates of at least 23% after interruption. The results point to persistent difficulty adapting to evolving conversations while tracking what remains to be completed or reported.

### Source excerpt

arXiv:2611.12118v1 Announce Type: new Abstract: Full-duplex voice agents enable natural, hands-free interaction, allowing users to refine requests, redirect ongoing work, and seek clarification while the agent is speaking. However, existing voice-agent benchmarks insufficiently isolate coordination between task execution and conversation. We introduce a benchmark that assesses this coordination by examining whether agents track what is being executed, what has been completed, what has been communicated, and what remains to be reported as the conversation evolves. We propose an interactive user simulator that introduces configurable interventions at designated timing in tool execution and spoken result delivery, supporting controllable and reproducible evaluation conditions. We evaluate five leading commercial voice agents along three dimensions: execution responsiveness, coordination under request changes, and coordination under interruptions. Agent evaluations reveal persistent coordination failures. Under request changes, the highest result completion rate is only 55%, indicating difficulty for current voice agents adapting to new requests while preserving earlier obligations. After user interruption, four of the five agents exhibit average information omission rates of at least 23%. These findings highlight task coordination as a central challenge for current voice agents in complex, evolving conversations.

[Next page](<https://devfeed.tech/topics/interactive-ai-systems.md?cursor=WyIyMDI2LTExLTEwVDA1OjAwOjAwKzAwOjAwIiwgIjIwMzRjMDBmLTU1MzEtNDllYi05NTBkLTI0MmQ5NjhmY2QxZiJd>)