# How to "train" a surgeon? Benchmarking generalist agents in surgical scene understanding

DevFeed: [How to "train" a surgeon? Benchmarking generalist agents in surgical scene understanding](<https://devfeed.tech/articles/how-to-train-a-surgeon-benchmarking-generalist-agents-in-surgical-scene-understanding-126114.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11437>)

Author: Gengyuan Zhang, Xiao Han, Shijie Zhou, Abdullah Koukash, Jingpei Wu, Jindong Gu, Yuqicheng Zhu, Nassir Navab, Volker Tresp, Ghazal Ghazaei

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [Machine Perception](<https://devfeed.tech/topics/machine-perception.md>), [gaia](<https://devfeed.tech/topics/gaia.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [bottlenecks](<https://devfeed.tech/tags/bottlenecks.md>), [boundary](<https://devfeed.tech/tags/boundary.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [data](<https://devfeed.tech/tags/data.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [diagnostic](<https://devfeed.tech/tags/diagnostic.md>), [environment](<https://devfeed.tech/tags/environment.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [failure](<https://devfeed.tech/tags/failure.md>), [fine](<https://devfeed.tech/tags/fine.md>), [generalist](<https://devfeed.tech/tags/generalist.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [train](<https://devfeed.tech/tags/train.md>)

## AI overview

The study finds that general-purpose multimodal language models have partial visual priors for surgical scenes but struggle with relationships, procedural understanding, and workflow assessment. It introduces SurgKnowBench to evaluate pretrained models across public datasets and SurgAgentGym to test whether bounded SurgWiki search and visual grounding can improve performance at inference time, without additional surgical training. Tool use narrows the gap mainly on perception-heavy tasks; fine-grained localization and expert-style assessment remain bottlenecks. The supplied text is an abstract, so it does not provide detailed methods or results.

## Source excerpt

arXiv:2611.11437v1 Announce Type: cross Abstract: Evaluating general-purpose multimodal large language models (MLLMs) for surgical scene understanding remains limited due to scarce surgical data and fragmented evaluation protocols. We address these challenges through a two-stage evaluation framework for surgical scene understanding. In Stage I, we introduce SurgKnowBench, a diagnostic benchmark that harmonizes public surgical datasets across procedures and tasks and evaluates proprietary and open-source MLLMs to map the boundary of pretrained surgical competence. SurgKnowBench reveals that current models retain partial visual priors but remain weak on relational and procedural understanding and workflow assessment. Building on this failure frontier and motivated by the scaffolded nature of surgical training, we introduce SurgAgentGym, an agentic environment where general-purpose MLLM agents can use bounded knowledge search over SurgWiki and general-purpose visual grounding for surgical tasks. SurgAgentGym evaluates whether generalist agents can improve surgical understanding at inference time through tool-mediated evidence use, without additional surgical training. Our results show that agentic inference narrows the gap mainly for perception-heavy tasks, while fine-grained localization and expert-style assessment remain persistent bottlenecks. This opens a path toward future studies of agentic surgical understanding.