# Correct Answers from Insufficient Histories: A Controlled Study of Agent Memory Evaluation

DevFeed: [Correct Answers from Insufficient Histories: A Controlled Study of Agent Memory Evaluation](<https://devfeed.tech/articles/correct-answers-from-insufficient-histories-a-controlled-study-of-agent-memory-evaluation-126189.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11840>)

Author: Quan Xue, Zerui Yang, Chaorui Zhang, Haolin Ruan, Xueyan Niu, Xuanyu Zheng, Xiyuan Wang, Weixi Zhang, Wei Han, Bo Bai

Published: 2026-11-10T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-memory](<https://devfeed.tech/tags/agent-memory.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [csv](<https://devfeed.tech/tags/csv.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [deepseek-v4-flash](<https://devfeed.tech/tags/deepseek-v4-flash.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [logs](<https://devfeed.tech/tags/logs.md>), [memory](<https://devfeed.tech/tags/memory.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [study](<https://devfeed.tech/tags/study.md>)

## AI overview

The study separates reference agreement from whether an agent's visible history actually supports an answer. It certifies insufficiency by constructing two legal histories with identical visible contexts but different answers, and uses public chronology and query cutoffs to identify sufficient prefixes. Across 203 insufficient raw-prefix cases, Qwen3-32B matched 87 hidden references and appropriately abstained on seven; a fixed-input DeepSeek-V4-Flash replication matched 85 and abstained on 21. Estimated appropriateness effects for compact CSV were inconclusive for both readers. A held-out judge was 99.38% correct but still made support errors and suffered truncations, while a deterministic baseline solved all constructed cases. The authors frame these results as an evaluation diagnosis for two hosted configurations, not evidence of general memory-system gains or natural-language evidence assessment.

## Source excerpt

arXiv:2611.11840v1 Announce Type: new Abstract: An answer can match the reference for a complete agent history even when the reader's visible history does not determine it. We study this distinction using executable device logs. Our protocol separates reference agreement, certified evidence sufficiency, and context-appropriate behavior. Insufficiency requires two legal histories yielding byte-identical visible contexts but different answers; public chronology and query cutoffs certify sufficient prefixes. On 80 confirmation histories, Qwen3-32B matches 87 hidden references among 203 insufficient raw-prefix instances, with only seven appropriate abstentions. A fixed-input DeepSeek-V4-Flash replication matches 85 and abstains on 21 of the same 203 instances. Each reader makes 1,600 requests. Compact CSV's estimated appropriateness effects remain inconclusive: +3.16 percentage points, clustered 95% interval [0.00,6.57], for Qwen and +2.46, [-1.84,6.94], for DeepSeek. A separate held-out judge study achieves 99.38% correctness but retains support errors and truncations; a deterministic baseline solves all constructed cases. These results support an evaluation diagnosis across two hosted reader configurations, not gains for a general memory system or natural-language evidence assessment.