# Ask Precisely, Assume Wisely: Interactive Evaluation of Ambiguity Resolution in Large Language Models

DevFeed: [Ask Precisely, Assume Wisely: Interactive Evaluation of Ambiguity Resolution in Large Language Models](<https://devfeed.tech/articles/ask-precisely-assume-wisely-interactive-evaluation-of-ambiguity-resolution-in-large-language-models-137361.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18905>)

Author: Avik Dutta, Aayush Kumar, Zahra Ashktorab, Hosein Hasanbeig, Austin Z. Henley, Gustavo Soares

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [Usability](<https://devfeed.tech/topics/usability.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [framework](<https://devfeed.tech/tags/framework.md>), [human-ai-interaction](<https://devfeed.tech/tags/human-ai-interaction.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [models](<https://devfeed.tech/tags/models.md>), [system-prompts](<https://devfeed.tech/tags/system-prompts.md>), [user-intent](<https://devfeed.tech/tags/user-intent.md>)

## AI overview

The paper introduces a preference-grounded simulated-user framework for evaluating how language models resolve ambiguity through multi-turn interaction. It measures three dissatisfaction signals beyond task accuracy: incorrect assumptions, repeated clarifications, and irrelevant clarifications. The reported results show a trade-off: clarification-heavy models tend to over-clarify and prolong interactions, while assumption-driven models more often miss user intent. System prompts optimized against these signals reportedly balance all three metrics and reduce overall error rates by up to 7.65%. The evidence provided is an abstract, so it does not detail the evaluation setup or results beyond these claims.

## Source excerpt

arXiv:2611.18905v1 Announce Type: new Abstract: Ambiguity is a core challenge in human--AI interaction, where models must decide when to make assumptions and when to seek clarification. Existing approaches to ambiguity evaluation largely focus on coverage over multiple interpretations using static metrics, offering limited insight into how efficiently a model converges toward a single user-intended interpretation through interaction. We introduce a preference-grounded simulated user framework for controlled, multi-turn evaluation of ambiguity resolution, grounding simulator responses strictly in user disambiguation preferences to avoid hallucination and over-disclosure. Alongside this, we define three observable dissatisfaction signals --- incorrect assumptions, repeating clarifications, and irrelevant clarifications --- that characterize interaction quality beyond task accuracy. Our analysis reveals a clear trade-off: models that frequently seek clarification tend to over-clarify and prolong interactions, while assumption-driven models are more prone to misaligning with user intent. We demonstrate that system prompts optimized on these dissatisfaction signals achieve a Pareto-optimal balance across all three metrics, reducing overall error rates by up to 7.65%, underscoring the value of our framework in designing agents that ask precisely and assume wisely.