# CTE-Bench: Counterfactual Trace Evaluation for Stateful Software Simulators

DevFeed: [CTE-Bench: Counterfactual Trace Evaluation for Stateful Software Simulators](<https://devfeed.tech/articles/cte-bench-counterfactual-trace-evaluation-for-stateful-software-simulators-107355.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.36647>)

Author: Xinran Zhang

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Stateful](<https://devfeed.tech/topics/stateful.md>), [Software](<https://devfeed.tech/topics/software.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [bench](<https://devfeed.tech/tags/bench.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [claude](<https://devfeed.tech/tags/claude.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [kimi](<https://devfeed.tech/tags/kimi.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [predictions](<https://devfeed.tech/tags/predictions.md>), [python](<https://devfeed.tech/tags/python.md>), [qwen3-6](<https://devfeed.tech/tags/qwen3-6.md>), [score](<https://devfeed.tech/tags/score.md>), [scripts](<https://devfeed.tech/tags/scripts.md>), [service](<https://devfeed.tech/tags/service.md>), [software](<https://devfeed.tech/tags/software.md>), [state](<https://devfeed.tech/tags/state.md>), [stateful](<https://devfeed.tech/tags/stateful.md>), [trace](<https://devfeed.tech/tags/trace.md>), [v1](<https://devfeed.tech/tags/v1.md>)

## AI overview

CTE-Bench evaluates whether models can predict how code edits or state overwrites change a deterministic service's future responses. In each scenario, a model receives Python service code, the pre-intervention call history, an intervention, and 40 fixed future calls; an executable oracle checks its predictions. Three protocols vary whether correct prior responses, no prior responses, or the model's own predictions are available. Across 255 scenarios and six services, four API-hosted models achieved 54.3%-61.7% exact-match accuracy on intervention-affected calls when given correct feedback. Accuracy fell to 23.2%-29.0% without that feedback and was 24.8%-33.2% when models relied on their own predictions; at most 1.2% of scenarios were predicted exactly end to end. The results suggest errors compound across a rollout. The release includes the benchmark, executable oracle, evaluation scripts, and an evaluation card.

## Source excerpt

arXiv:2609.36647v2 Announce Type: replace Abstract: Coding agents change running software: they patch a service's code or overwrite its stored state, and then act on their own expectation of how the service will respond afterwards. A wrong expectation may surface only several calls later. Function-level code-execution benchmarks omit persistent service state, and agent benchmarks score the actions an agent takes or the final state it reaches. We introduce CTE-Bench, which measures whether a model can predict how an intervention changes a stateful service's future behavior, without asking it to choose actions. Each scenario gives the model Python service code, the calls and responses observed before the intervention, the intervention itself (a source edit or a state overwrite), and 40 fixed future calls; the model predicts every future response, and predictions are checked by executing the service. Three memory protocols control whether the model sees the correct earlier responses, none of them, or its own earlier predictions. CTE-Bench-Core-v1 contains 255 scenarios over six deterministic Python services, giving 10,200 predictions per model. The main score is effect-step value match (VM): exact response equality on the 2,476 future calls whose response the intervention changes. With correct earlier responses revealed, four API-hosted models (DeepSeek V4-Flash, Kimi K2.5, Qwen3.6-35B-A3B, and Claude Sonnet 4.6) reach 54.3%-61.7% effect-step VM. Hiding those responses lowers effect-step VM to 23.2%-29.0%; conditioning on self-generated predictions gives 24.8%-33.2%, and at most 1.2% of scenarios are predicted exactly end to end. Current models thus track intervention effects mainly when correct feedback is supplied, and their errors compound over a rollout. We release CTE-Bench-Core-v1 with its executable oracle, evaluation scripts, and an evaluation card mapping each claim to its protocol.