# Benchmarks Are Surprisingly Compaction-Friendly

DevFeed: [Benchmarks Are Surprisingly Compaction-Friendly](<https://devfeed.tech/articles/benchmarks-are-surprisingly-compaction-friendly-99508.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.31778>)

Author: Jeanne Salle, Hardik Bhatnagar, Matthias Bethge, Sahar Abdelnabi, Maksym Andriushchenko

Published: 2026-10-26T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [context window](<https://devfeed.tech/topics/context-window.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [long-running](<https://devfeed.tech/topics/long-running.md>), [Frontier AI](<https://devfeed.tech/topics/frontier-ai.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-tasks](<https://devfeed.tech/tags/agentic-tasks.md>), [agents](<https://devfeed.tech/tags/agents.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agent](<https://devfeed.tech/tags/coding-agent.md>), [compaction](<https://devfeed.tech/tags/compaction.md>), [context](<https://devfeed.tech/tags/context.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [friendly](<https://devfeed.tech/tags/friendly.md>), [frontier-models](<https://devfeed.tech/tags/frontier-models.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [gpt-5](<https://devfeed.tech/tags/gpt-5.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [open-weight-models](<https://devfeed.tech/tags/open-weight-models.md>), [opencode](<https://devfeed.tech/tags/opencode.md>), [performance](<https://devfeed.tech/tags/performance.md>)

## AI overview

In tests using OpenCode, five context-compaction strategies, three frontier models and three long-horizon benchmarks, compaction usually had little measurable effect on the two models with 1M-token context windows. GPT-5.3-Codex, with the smallest window, performed worse at lower compaction thresholds on PaperBench and less clearly on PostTrainBench. No tested strategy beat default self-summarization; discarding more content cost some models up to 20 points while leaving others unchanged. Yet summaries could silently drop constraints: the first compaction lost about a third of ProgramBench's anti-cheating rules on average, with losses growing as compactions accumulated. In SWE-chat sessions, instructions about how the agent should work or what to avoid were dropped about six times more often than build specifications. The authors argue these benchmarks are compaction-friendly because objectives stay fixed and progress remains on disk, making most discarded information recoverable--except constraints recorded nowhere else.

## Source excerpt

arXiv:2610.31778v1 Announce Type: new Abstract: Long horizon agentic tasks are now pushing LLM agents past their context window. To continue the task, the agent scaffold has to free up the context window, a process known as context compaction. Compaction is generally lossy by design: agents can silently lose task constraints, instructions, or their reasoning behind earlier decisions. Although compaction is a standard component of coding agent scaffolds, it has not been studied systematically with frontier models on realistic, long horizon agentic tasks. Previous work evaluate smaller open weight models on toy or synthetic trajectories. We fix the agent scaffold (OpenCode), implement five compaction strategies inside it, and evaluate three frontier models on three long horizon benchmarks (PaperBench, ProgramBench, PostTrainBench). For the two 1M context models, the compaction threshold does not measurably affect the task performance, whereas GPT-5.3-Codex, the model with the smallest context window, performs worse at lower thresholds on PaperBench and, less clearly, on PostTrainBench. No compaction method outperforms default self-summarization, and methods that discard more content cost some models up to 20 points while leaving other models the same. Compaction can, however, silently remove task constraints: on ProgramBench, the first compaction summary already drops about one third of the task's anticheating rules on average, and this rises monotonically with the number of compactions. On real user-agent sessions from SWE-chat, instructions about how the agent should work and what it should not do are dropped around 6 times more than specifications of what to build. We argue that current benchmarks are compaction-friendly: i.e., the objective is fixed and the agent's progress stays on disk, so most of what compaction discards can be recovered, except constraints that don't exist anywhere else.