# GRAC: Graph-Based Retrieval-Augmented Captioning for Surgical Scene Description

DevFeed: [GRAC: Graph-Based Retrieval-Augmented Captioning for Surgical Scene Description](<https://devfeed.tech/articles/grac-graph-based-retrieval-augmented-captioning-for-surgical-scene-description-124737.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09866>)

Author: Rochak Dhakal, Chen Zhao, Soup Cobb, Xin Qian, Xiaofeng Yang, Weihua Zhou

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [Machine Perception](<https://devfeed.tech/topics/machine-perception.md>), [Graphs](<https://devfeed.tech/topics/graphs.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>)

Tags: [actions](<https://devfeed.tech/tags/actions.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [features](<https://devfeed.tech/tags/features.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generation](<https://devfeed.tech/tags/generation.md>), [graph](<https://devfeed.tech/tags/graph.md>), [manual-review](<https://devfeed.tech/tags/manual-review.md>), [pairs](<https://devfeed.tech/tags/pairs.md>), [performance](<https://devfeed.tech/tags/performance.md>), [procedural](<https://devfeed.tech/tags/procedural.md>), [quality](<https://devfeed.tech/tags/quality.md>), [relationships](<https://devfeed.tech/tags/relationships.md>), [reports](<https://devfeed.tech/tags/reports.md>), [results](<https://devfeed.tech/tags/results.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [structured](<https://devfeed.tech/tags/structured.md>), [support](<https://devfeed.tech/tags/support.md>), [target](<https://devfeed.tech/tags/target.md>), [training](<https://devfeed.tech/tags/training.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [using](<https://devfeed.tech/tags/using.md>)

## AI overview

GRAC improves surgical scene captioning by representing instruments, actions, and anatomy as a graph, retrieving related graph-caption examples, and using differences in instrument-tissue interactions to condition caption generation. On EndoVis-18, it scored 5.88 CIDEr-D versus 4.23 for SGT and 3.36 for the strongest conventional Transformer baseline; exact instrument-action-target triplet-set accuracy was 52.6%, versus 32.7% and 26.2%. On TORS, its CIDEr-D was 5.06 versus 3.80 for the strongest baseline. These reported results indicate stronger caption quality and factual agreement on the evaluated benchmarks.

## Source excerpt

arXiv:2611.09866v1 Announce Type: cross Abstract: Purpose: Surgical operative reports support procedural documentation, surgical training, performance assessment, and workflow review. However, extracting meaningful information often requires extensive manual review and clinical expertise. Automated surgical scene descriptions could reduce this burden by leveraging recurring surgical activities and prior knowledge. Methods: We propose GRAC (Graph-Based Retrieval-Augmented Captioning), a framework that represents surgical scenes as graphs capturing relationships among instruments, actions, and anatomical structures. Given an input scene graph, GRAC learns to retrieve relevant graph-caption pairs from training data. It compares instrument-tissue interactions between retrieved examples and the input scene, using comparison features to condition retrieved-caption representations during generation. Results: On the EndoVis-18 benchmark, GRAC achieved a CIDEr-D of 5.88, compared with 4.23 for SGT and 3.36 for the strongest conventional Transformer baseline. Exact instrument-action-target triplet-set accuracy reached 52.6%, compared with 32.7% for SGT and 26.2% for the strongest conventional baseline. These results demonstrate improvements in language quality and factual agreement. On TORS, GRAC achieved a CIDEr-D of 5.06, compared with 3.80 for the strongest baseline. Conclusion: GRAC demonstrates that combining structured surgical scene representations with retrieved examples improves surgical scene captioning and produces more contextually meaningful descriptions.