# Programmable Distributions of Semantics-Preserving Variants for Evaluating Code Agents

DevFeed: [Programmable Distributions of Semantics-Preserving Variants for Evaluating Code Agents](<https://devfeed.tech/articles/programmable-distributions-of-semantics-preserving-variants-for-evaluating-code-agents-135001.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.18389>)

Author: Hasan Najib Mahmud, Shreya Gupta, Isha Chaudhary, Nathaniel Enis, Ravi Mangal, Gagandeep Singh, Corina Pasareanu

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [Refactoring](<https://devfeed.tech/topics/refactoring.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [claude](<https://devfeed.tech/tags/claude.md>), [claude-opus](<https://devfeed.tech/tags/claude-opus.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [distributions](<https://devfeed.tech/tags/distributions.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [frontier-models](<https://devfeed.tech/tags/frontier-models.md>), [kimi](<https://devfeed.tech/tags/kimi.md>), [minimax](<https://devfeed.tech/tags/minimax.md>), [opencode](<https://devfeed.tech/tags/opencode.md>), [probabilistic](<https://devfeed.tech/tags/probabilistic.md>), [programming-language](<https://devfeed.tech/tags/programming-language.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [refactoring](<https://devfeed.tech/tags/refactoring.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [robustness](<https://devfeed.tech/tags/robustness.md>), [semantics](<https://devfeed.tech/tags/semantics.md>), [swe-bench](<https://devfeed.tech/tags/swe-bench.md>), [swe-bench-pro](<https://devfeed.tech/tags/swe-bench-pro.md>)

## AI overview

The study tests whether code agents can resolve repository issues after surrounding code is rewritten into semantically equivalent variants. It introduces VDL, a probabilistic programming language for defining distributions of variants through randomly applied semantics-preserving transformations. Using a generic distribution, the authors evaluate mini-SWE agent and OpenCode with four underlying models on SWE-bench Verified and SWE-bench Pro. Resolve rates fall by as much as 6.7 percentage points, with significant drops in 6 of 16 configurations. Model robustness rankings do not hold across the two scaffolds, suggesting robustness depends on the scaffold-model pairing. The supplied abstract does not specify individual transformation types or per-configuration results.

## Source excerpt

arXiv:2608.18389v2 Announce Type: replace Abstract: Code agents are increasingly deployed to resolve real software issues, yet their reliability under superficial code variations remains poorly understood. We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form. We sample semantically equivalent variants of a repository from an explicitly defined distribution and compare the agent's resolve rate on them with that on the original. Different distributions reveal different aspects of an agent, such as its sensitivity to routine refactoring or to misleading code near the location of a fix. We develop VDL, a probabilistic programming language for expressing such distributions, whose programs randomly apply semantics-preserving transformations (SPTs) to a repository. For our evaluation, we use it to define a generic distribution that is not targeted at any particular failure mode. We evaluate two agentic scaffolds (mini-SWE agent and OpenCode), each backed by one of four frontier models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B), on instances from SWE-bench Verified and SWE-bench Pro. Resolve rates drop by up to 6.7 percentage points, significantly in 6 of 16 configurations. No ranking of models by robustness holds across scaffolds, revealing a jagged robustness frontier.