# VagueBench: Benchmarking the Robustness of Embodied Agents on Ambient Context Illusions

DevFeed: [VagueBench: Benchmarking the Robustness of Embodied Agents on Ambient Context Illusions](<https://devfeed.tech/articles/vaguebench-benchmarking-the-robustness-of-embodied-agents-on-ambient-context-illusions-117376.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.03765>)

Author: Yang Qu, Qiang Fu, Zonghao Ying, Jiayi Zhao, Yaoyuan Zhang, Jinyang Guo, Jian Yang, Aishan Liu, Yingxun Wang, Xianglong Liu

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [context](<https://devfeed.tech/topics/context.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ambient-context](<https://devfeed.tech/tags/ambient-context.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [dependency](<https://devfeed.tech/tags/dependency.md>), [environment](<https://devfeed.tech/tags/environment.md>), [environments](<https://devfeed.tech/tags/environments.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [instructions](<https://devfeed.tech/tags/instructions.md>), [invariance](<https://devfeed.tech/tags/invariance.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [pairs](<https://devfeed.tech/tags/pairs.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [risks](<https://devfeed.tech/tags/risks.md>), [robustness](<https://devfeed.tech/tags/robustness.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [simulation](<https://devfeed.tech/tags/simulation.md>)

## AI overview

VagueBench evaluates whether embodied agents can carry out tasks from semantically sufficient instructions when non-essential contextual cues are removed. Built in AI2-THOR, it contains 1,303 instruction pairs organized around four semantic components--entities, procedures, constraints, and prior knowledge--with four categories and nine subtypes for reducing auxiliary information. Across six embodied agents, the reduced-context instructions were associated with a relative 19.65% drop in task success and a relative 54.10% increase in empty plans. The authors also report that the problem persists in real-world robotic systems, where it can contribute to incorrect decisions and potential safety risks.

## Source excerpt

arXiv:2611.03765v1 Announce Type: new Abstract: Embodied agents are increasingly deployed in open-world environments, yet current robustness evaluations remain biased toward hazardous instructions and jailbreak vulnerabilities. A more fundamental issue is largely overlooked: agents are rarely tested under semantically sufficient but contextually minimal instructions (i.e., common in real-world human communication), where auxiliary contextual cues are absent. For the first, we identify a novel phenomenon termed Ambient Context Illusion, where agents fail to execute tasks correctly when non-essential contextual details are omitted, despite unchanged task semantics. To systematically study this, we establish VagueBench, a benchmark built upon the AI2-THOR interactive simulation environment for the evaluation of context dependency in embodied agents. We systematically consider key semantic components of instructions (entities, procedures, constraints, and prior knowledge) and derive a unified taxonomy with four categories and nine subtypes to reduce auxiliary information. VagueBench contains 1,303 instruction pairs across diverse indoor environments, constructed via a human-agent co-generation framework that integrates semantic redundancy estimation, taxonomy-guided transformation, and rigorous semantic-invariance verification. Through extensive evaluation on 6 state-of-the-art embodied agents, we observe significant degradation on VagueBench in both task performance and executability (relatively -19.65% in task success rate and relatively +54.10% in empty plan rate) revealing a strong reliance on auxiliary linguistic signals rather than core task semantics. Furthermore, we demonstrate that this phenomenon persists in real-world robotic systems, where it can lead to incorrect decisions and potential safety risks.