# When Should AI Chime In? Benchmarking Timely Contribution of Proactive Agents in Multi-Party Chat

DevFeed: [When Should AI Chime In? Benchmarking Timely Contribution of Proactive Agents in Multi-Party Chat](<https://devfeed.tech/articles/when-should-ai-chime-in-benchmarking-timely-contribution-of-proactive-agents-in-multi-party-chat-98001.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.25896>)

Author: Ruotong Wang, Brian S. Yu, Xinyi Zhou, Joseph Chee Chang, Pao Siangliulue, Jonathan Bragg, Liwei Jiang, Amy X Zhang

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI interaction best practices](<https://devfeed.tech/topics/ai-interaction-best-practices.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [human review](<https://devfeed.tech/topics/human-review.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [chat](<https://devfeed.tech/tags/chat.md>), [collaboration](<https://devfeed.tech/tags/collaboration.md>), [conversations](<https://devfeed.tech/tags/conversations.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [live](<https://devfeed.tech/tags/live.md>), [models](<https://devfeed.tech/tags/models.md>), [proactive](<https://devfeed.tech/tags/proactive.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [relevance](<https://devfeed.tech/tags/relevance.md>), [timing](<https://devfeed.tech/tags/timing.md>), [waiting](<https://devfeed.tech/tags/waiting.md>)

## AI overview

PITCH benchmarks when an AI should contribute a fixed message in unscripted group chats, using 1,091 human judgments from 89 participants across 35 conversations. In live decisions, models achieved 19.0% best-turn accuracy, down from 57.8% when given the full conversation history; lay humans deciding live reached 23.7%. Model placements were also premature more often than human placements (46.4% versus 39.7%). The authors report that 93% of model justifications cite relevance, while almost none consider waiting, pointing to a need for agents to reason explicitly about timing. These findings are based on 15 models across 28 configurations.

## Source excerpt

arXiv:2610.25896v2 Announce Type: replace Abstract: Proactive agents can support group collaboration by surfacing useful information without waiting to be asked, but even relevant contributions can be disruptive if poorly timed. In this work, we isolate the timing question: given a fixed candidate contribution such as a clarification or relevant fact, when should an AI agent chime in? We introduce PITCH (Proactive Intervention Timing in CHat), to our knowledge the first benchmark with large-scale human judgments of proactive agent timing collected from participants in the same unscripted multi-party conversations they took part in. To collect high-quality judgments, we design a novel interface in which 2-4 people chat synchronously, author candidate AI contribution messages, place their own and others' contributions into the chat history, and then deliberate as a group about the best placement for each message. PITCH contains 384 candidate contributions and 1,091 judgments, each marking an ideal turn, an acceptable window, or that a contribution should not appear, from 89 participants engaged in 35 conversations. Evaluating 15 models across 28 configurations, we find that while models can place contributions well when given the entire conversation history, they systematically chime in too early when asked to decide live as the conversation unfolds: best-turn accuracy falls from 57.8% to 19.0%, with nearly half of placements occurring before every participant's acceptable window. In comparison, lay humans deciding live are more accurate (23.7% vs 19.0%), and fewer of their placements are premature (39.7% vs 46.4%). Model reasoning traces show that 93% of intervention justifications cite relevance, while almost none consider waiting. By providing a human-grounded benchmark, our work reveals gaps in agents' ability to integrate into group settings and points towards proactive agents that can reason explicitly about appropriate timing.