# NIBench: benchmarking autonomous AI agents for execution, skill evolution and scientific judgment in neuroimaging

DevFeed: [NIBench: benchmarking autonomous AI agents for execution, skill evolution and scientific judgment in neuroimaging](<https://devfeed.tech/articles/nibench-benchmarking-autonomous-ai-agents-for-execution-skill-evolution-and-scientific-judgment-in-neuroimaging-132176.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15831>)

Author: Lujia Zhong, Yihao Xia, Jianwei Zhang, Shuo Huang, Xinyu Nie, Mingyang Xia, Zhiwei Deng, Jiaxin Yue, Xinkai Wang, Junhan Zhao, Yonggang Shi

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [Agent Skills](<https://devfeed.tech/topics/agent-skills.md>), [gaia](<https://devfeed.tech/topics/gaia.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [execution](<https://devfeed.tech/tags/execution.md>), [imaging](<https://devfeed.tech/tags/imaging.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [performance](<https://devfeed.tech/tags/performance.md>)

## AI overview

NIBench evaluates autonomous agents on neuroimaging workflow execution, procedural skill evolution, and scientific judgment. It tests 19 language models in a shared framework, using 56 tasks across seven tool ecosystems and 36 claims from Alzheimer's disease studies. Leading agents nearly perfectly execute supplied protocols, but performance differences widen when procedural guidance is withheld. Distilled skills help on tasks agents have encountered and can transfer between models, while gains on unseen tasks are inconsistent. Agents can reproduce their analytical results yet still reach conclusions that disagree with expert-derived answers; access to cohort data does not always improve judgment.

## Source excerpt

arXiv:2611.15831v1 Announce Type: new Abstract: Autonomous AI agents are increasingly used to automate biomedical research, creating a growing need to benchmark their capabilities on real scientific data and workflows. Their reliability in neuroimaging remains insufficiently characterized, where plausible analytical outputs can conceal errors that affect scientific conclusions. Existing studies have advanced scientific question answering and tool-based automation, but provide limited evidence on workflow reconstruction, acquisition of procedural expertise and evidence-based scientific judgment. Here we present NIBench, a neuroimaging benchmark comprising three complementary evaluations of autonomous agents. First, we benchmark execution and workflow reconstruction using 56 tasks across seven tool ecosystems, covering structural, diffusion and functional magnetic resonance imaging, positron emission tomography and multimodal analyses. Second, we examine whether agents can evolve their procedural skills through distillation of execution trajectories, evaluating the resulting skills on previously encountered tasks, unseen tasks and other models. Third, we assess scientific judgment using 36 claims from published Alzheimer's disease studies, requiring agents to analyse raw cohort data and provide reproducible evidence. We evaluate 19 language models within a shared agent framework, with selected models undergoing skill distillation and claim evaluation. Leading agents execute supplied analysis protocols nearly perfectly, whereas withholding procedural guidance widens performance differences. Distilled skills improve performance on acquisition tasks and transfer between models, but benefits on unseen tasks are inconsistent. Agents reproduce their reported results yet sometimes reach conclusions inconsistent with expert-derived answers, and access to data does not always improve judgment.