# mutation-testing

A software-testing technique that evaluates unit-test quality by making deliberate code changes and checking whether tests detect them.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## What AI-First Engineering Means for the Future of Software Quality

DevFeed: [What AI-First Engineering Means for the Future of Software Quality](<https://devfeed.tech/articles/what-ai-first-engineering-means-for-the-future-of-software-quality-145967.md>)

Original publisher: [Read original article](<https://testgrid.io/blog/ai-first-engineering/>)

Author: Ankita Nandi

Published: 2026-11-19T10:55:22Z

Content type: article

Language: en

Sources: [TestGrid](<https://devfeed.tech/sources/testgrid-blog.md>)

Topics: [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>), [business logic](<https://devfeed.tech/topics/business-logic.md>), [engineering-leadership](<https://devfeed.tech/topics/engineering-leadership.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-generated-tests](<https://devfeed.tech/tags/ai-generated-tests.md>), [ai-testing](<https://devfeed.tech/tags/ai-testing.md>), [ai-workflow](<https://devfeed.tech/tags/ai-workflow.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [automation](<https://devfeed.tech/tags/automation.md>), [bank](<https://devfeed.tech/tags/bank.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [billing](<https://devfeed.tech/tags/billing.md>), [block](<https://devfeed.tech/tags/block.md>), [browser](<https://devfeed.tech/tags/browser.md>), [browsers](<https://devfeed.tech/tags/browsers.md>), [credit-card](<https://devfeed.tech/tags/credit-card.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [quality](<https://devfeed.tech/tags/quality.md>), [rules](<https://devfeed.tech/tags/rules.md>), [software](<https://devfeed.tech/tags/software.md>), [system](<https://devfeed.tech/tags/system.md>), [tests](<https://devfeed.tech/tags/tests.md>)

### AI overview

AI-first workflows can accelerate software delivery, but when an agent derives code, tests and documentation from the same incomplete requirement, they can all reinforce one mistaken assumption. The excerpt distinguishes tests that confirm behavior inferred from buggy code from tests that correctly enforce a flawed or underspecified business rule; either can leave a green build that fails users. It recommends agreeing on expected behavior and edge-case examples before implementation, having an accountable domain expert approve them, and generating business-rule tests from those approved expectations. Validate AI-generated tests by planting known bugs. For high-risk changes, add independent security review, hands-on human testing and sign-off. Use execution evidence such as video, screenshots, logs and network traffic to investigate failures, and measure customer bugs, pre-release risks and diagnosis time rather than test counts alone. The excerpt also calls for junior engineers to practice questioning requirements and explaining conclusions. Its example is hypothetical, and the overview reflects the supplied excerpt.

### Source excerpt

Let's say you oversee a bank's credit-card billing-error system. Your product team gives an AI agent a requirement: "Accept written billing-error notices within 60 days." The agent implements the rule, generates 40 tests, and reports that every test passes. But it has started the clock on the transaction date. Under US federal billing-error rules, the [...]

## When Exact Inference Changes Code Selection: A Decision-Theoretic Analysis of B4

DevFeed: [When Exact Inference Changes Code Selection: A Decision-Theoretic Analysis of B4](<https://devfeed.tech/articles/when-exact-inference-changes-code-selection-a-decision-theoretic-analysis-of-b4-142167.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23306>)

Author: Yanliang Cai (Independent Researcher)

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [Operations research and optimization](<https://devfeed.tech/topics/operations-research-and-optimization.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [decision](<https://devfeed.tech/tags/decision.md>), [inference](<https://devfeed.tech/tags/inference.md>), [model](<https://devfeed.tech/tags/model.md>), [selection](<https://devfeed.tech/tags/selection.md>), [software-testing](<https://devfeed.tech/tags/software-testing.md>), [validation](<https://devfeed.tech/tags/validation.md>)

### AI overview

In the B4 code-test model, exact candidate marginalization can be computed efficiently by grouping programs with identical execution signatures, while a prior-budget condition identifies when B4's whole-group rule already achieves joint MAP. A finite correction bound can certify when marginalization will preserve the selected group. In default settings, no candidate changes appeared in 2,421 development comparisons or 4,035 comparisons on a separate archive. Across 80 literature-derived priors, 85 of 107,600 comparisons changed top sets; only six coincided with failure of whole-group MAP. Finite-test proxy labels showed four changes increasing and five decreasing expected success, across seven snapshots, with per-prior mean differences from −0.074 to +0.045 percentage points. The reported findings are descriptive and do not recommend a tuned prior or new performance method.

### Source excerpt

arXiv:2611.23306v1 Announce Type: new Abstract: When generated programs are ranked using generated tests, a more exact posterior computation need not improve the selected program. We characterize this distinction in the published B4 code--test model. Identical execution signatures reduce exact candidate marginalization to N summands and nonempty joint maximum a posteriori (MAP) inference to two weights per group. A prior--budget condition guarantees that B4's whole-group rule already attains MAP; a sharp finite correction bound certifies when candidate marginals must select the same group. All 2,421 default-setting comparisons on 807 development matrices select the same candidates. A prospectively specified validation on 1,345 execution snapshots from a separate public archive also finds no changes in 4,035 default-setting comparisons. Across 80 literature-derived priors, this new archive yields 85 changed top sets in 107,600 comparisons; only six coincide with failure of whole-group MAP. Under finite-test proxy labels, four changes increase and five decrease expected success. These descriptive changes span seven snapshots; per-prior mean differences range from -0.074 to +0.045 percentage points. The results do not recommend a tuned prior or a new performance method. Exact enumeration and independent high-precision checks support the formulas and decision diagnostics. A supplementary fixed-test analysis separates asymptotic failure from observational ambiguity. The contribution is a model-specific account of when an inference intervention can change a software-testing decision, and when its effect must be sought in the priors or evidence instead.

## Can Coding Agents Write and Transpile Unit Tests

DevFeed: [Can Coding Agents Write and Transpile Unit Tests](<https://devfeed.tech/articles/can-coding-agents-write-and-transpile-unit-tests-139595.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21797>)

Author: Gil Pasternak, AJ Langley, Tom Yotam, Ellie Kitanidis

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>)

Tags: [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [tests](<https://devfeed.tech/tags/tests.md>), [unit-tests](<https://devfeed.tech/tags/unit-tests.md>)

### AI overview

The study benchmarks three coding agents using native frontier models at high reasoning effort, assessing their ability to write tests and translate tests between programming languages. Agent-written suites outperform human-written suites on traditional coverage measures, but the advantage is uneven: test quality declines sharply as project size grows, and much of the coverage gain comes from writing more tests. Test translation appears strong in aggregate but remains inconsistent and unreliable.

### Source excerpt

arXiv:2611.21797v1 Announce Type: new Abstract: Coding assistants built on Large Language Models (LLMs) have become a staple of modern software development, increasingly trusted to execute workflows and modernize code end-to-end. Much of this trust rests on passing tests, which are often written by the coding agents themselves. In this work, we examine the ability of these agents to both write reliable tests and translate them between programming languages. We benchmark three popular coding agents, each evaluated with a native frontier model and high reasoning effort. Verifying across a broad suite of metrics, we find that coding agents far exceed human written suites on traditional measures of coverage. This advantage, however, is uneven: agent test quality degrades sharply with project size, and much of the advantage lies in a greater number of written tests. We also show that agentic test translation---while seemingly strong in aggregate---remains inconsistent and unreliable.

## Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop

DevFeed: [Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop](<https://devfeed.tech/articles/done-but-still-working-anytime-evaluation-of-when-coding-agents-should-stop-140093.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22901>)

Author: Zerui Yang, Quan Xue, Miaomiao Dong, Chaorui Zhang, Xuanyu Zheng, Xiyuan Wang, Haolin Ruan, Xueyan Niu, Guangcheng Song, Weixi Zhang, Wei Han, Bo Bai

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [coding assistant](<https://devfeed.tech/topics/coding-assistant.md>)

Tags: [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [quality](<https://devfeed.tech/tags/quality.md>), [replication](<https://devfeed.tech/tags/replication.md>), [saving](<https://devfeed.tech/tags/saving.md>), [stop](<https://devfeed.tech/tags/stop.md>)

### AI overview

The study evaluates when coding agents can stop by replaying recorded runs, scoring intermediate patches with hidden tests, and tracking cost. Across about 6,000 SWE-bench Verified runs involving eight open models and two scaffolds, the potential savings at the point when self-tests pass vary widely: under 1% to 57%. A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but its quality test does not establish non-inferiority. Fine-tuning agents to imitate the rule saves 30% across six seeds, while whether quality is preserved remains uncertain. The analysis links stopping outcomes to how often a rule triggers, whether patches change afterward, and whether those changes repair or break them.

### Source excerpt

arXiv:2611.22901v1 Announce Type: new Abstract: Coding agents often continue working after their own tests pass. Does this additional work improve the patch, and how much could be saved by stopping earlier? We study these questions through anytime evaluation: replaying recorded runs, scoring intermediate patches with hidden tests, and measuring the cost incurred at each step. For policies that only stop a run or select an earlier patch, this allows paired comparisons without generating new trajectories, provided replay preserves the relevant states. Across about 6,000 runs of eight open models in two scaffolds on SWE-bench Verified, we find substantial differences in the value of continued work. The hindsight saving available at passing self-tests ranges from under 1% to 57%, depending on the model and scaffold. A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but the preregistered quality test does not establish non-inferiority. We explain these differences by measuring how often a rule fires, how often the patch changes afterwards, and whether those changes repair or break it. Fine-tuning agents to imitate the rule saves 30% over six seeds, while quality preservation remains uncertain. Our results help identify both the opportunities and the risks of early stopping.

## Teralizer: A Tool for Generalizing JUnit Tests into Property-Based jqwik Tests

DevFeed: [Teralizer: A Tool for Generalizing JUnit Tests into Property-Based jqwik Tests](<https://devfeed.tech/articles/teralizer-a-tool-for-generalizing-junit-tests-into-property-based-jqwik-tests-137985.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19242>)

Author: Johann Glock, Clemens Bauer, Martin Pinzger

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Test automation](<https://devfeed.tech/topics/test-automation.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [generate](<https://devfeed.tech/tags/generate.md>), [junit](<https://devfeed.tech/tags/junit.md>), [mutation](<https://devfeed.tech/tags/mutation.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [property](<https://devfeed.tech/tags/property.md>), [property-based-testing](<https://devfeed.tech/tags/property-based-testing.md>), [regression](<https://devfeed.tech/tags/regression.md>), [runtime](<https://devfeed.tech/tags/runtime.md>), [tests](<https://devfeed.tech/tags/tests.md>)

### AI overview

Teralizer converts JUnit tests into property-based jqwik tests using single-path symbolic analysis. For each test execution, it derives a path condition describing inputs that follow the same execution path and a symbolic output expression that serves as the expected-result oracle; generated inputs must satisfy that condition. In the reported evaluation, this generalization raised EvoSuite suite mutation scores by 0.8-3.9 percentage points. On EqBench, generalizing EvoSuite suites generated with 1 second per class matched the mutation score of suites generated with 60 seconds per class, with 31.9% less total runtime.

### Source excerpt

arXiv:2611.19242v1 Announce Type: new Abstract: Background: Property-based testing validates properties across many generated inputs and can thereby detect regressions that conventional unit tests miss. However, writing property-based tests is challenging for practitioners, which limits their adoption. Tool: Teralizer automatically generalizes JUnit tests into property-based jqwik tests for regression detection. Through single-path symbolic analysis, it extracts from each test execution a path condition, which characterizes the inputs that follow the same execution path, and a symbolic output expression, which computes the expected output for these inputs. The generalized tests generate inputs that satisfy the path condition and use the symbolic output expression as the oracle. Results: In our evaluation, generalization increased the mutation scores of EvoSuite-generated test suites by 0.8 to 3.9 percentage points. On the EqBench benchmark, generalizing the test suites that EvoSuite generated with 1 s per class reached the mutation score of EvoSuite's 60 s test suites with 31.9% less total runtime. Availability: Teralizer is available as open source at https://github.com/glockyco/Teralizer.

## Noise-Aware Testing of NISQ Quantum Programs: Self-Baseline Oracle with a Controlled False-Positive Rate

DevFeed: [Noise-Aware Testing of NISQ Quantum Programs: Self-Baseline Oracle with a Controlled False-Positive Rate](<https://devfeed.tech/articles/noise-aware-testing-of-nisq-quantum-programs-self-baseline-oracle-with-a-controlled-false-positive-rate-138003.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20613>)

Author: Mir Mohammad Yousuf, Shabir Ahmad Sofi

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Test-driven development](<https://devfeed.tech/topics/tdd.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [Phoronix Test Suite](<https://devfeed.tech/topics/phoronix-test-suite.md>)

Tags: [cs-se](<https://devfeed.tech/tags/cs-se.md>), [false-positive-rate](<https://devfeed.tech/tags/false-positive-rate.md>), [noise](<https://devfeed.tech/tags/noise.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [quant-ph](<https://devfeed.tech/tags/quant-ph.md>), [quantum](<https://devfeed.tech/tags/quantum.md>), [testing](<https://devfeed.tech/tags/testing.md>), [testing-framework](<https://devfeed.tech/tags/testing-framework.md>)

### AI overview

The framework controls false positives in NISQ program testing by comparing each test run with repeated runs of a trusted reference on the same backend. It sets the Jensen-Shannon divergence threshold from the finite-shot null distribution at a chosen level α, and uses a noncentral chi-square model to predict detection power. Homogeneity testing and a two-run sentinel address drift; zero-noise extrapolation helps distinguish fault-related rejections from noise. In simulations, at α = 10⁻³ it produced a 0.0011 false-positive rate across 9,000 correct runs, versus 36-74% for fixed thresholds, 8.9% for a QOIN-style filter, and 10.8% for a global threshold; its F1 on non-equivalent faults was 0.941. On IBM Heron, it rejected none of 82 correct runs and detected 120 of 121 faults and semantic mutants using about 14 minutes of QPU time. The reported results are limited to the abstract's simulation and device evaluations.

### Source excerpt

arXiv:2611.20613v1 Announce Type: cross Abstract: Testing quantum programs on Noisy Intermediate-Scale Quantum (NISQ) devices faces a test-oracle problem: hardware noise and finite-shot sampling make the runs of a correct program vary as much as those of a faulty one, so oracles with a fixed acceptance threshold produce false positives that depend on the program and the backend. We present a noise-aware testing framework built on self-baseline calibration: each run of the program under test is compared with repeated executions of a trusted reference program on the same backend, and the acceptance threshold is the $(1-\alpha)$ quantile of the finite-shot null distribution of the Jensen-Shannon divergence, so the oracle's false-positive rate is a level set in advance. A noncentral chi-square model predicts detection power, a homogeneity test and a two-run sentinel detect drift, a mutation engine labels every mutant exactly within a three-class noise-aware taxonomy, and zero-noise extrapolation attributes rejections to faults or noise. In simulation on nine core benchmarks, 32 QASMBench programs, ten real bugs, seven noise regimes and three device snapshots, the oracle's false-positive rate is 0.0011 at $\alpha = 10^{-3}$ over 9,000 correct runs, against 36-74% for fixed thresholds, 8.9% for a QOIN-style filter and 10.8% for a globally calibrated threshold, while attaining the highest F1 on non-equivalent faults (0.941). On an IBM Heron device, using about 14 minutes of QPU time, it rejected none of 82 correct runs and detected 120 of 121 faults and semantic mutants, and a two-run sentinel reduced drift-induced false positives from 27.4% to 1.8%.

## Finding Side-Effect Loopholes in AppWorld and WorkArena Verifiers

DevFeed: [Finding Side-Effect Loopholes in AppWorld and WorkArena Verifiers](<https://devfeed.tech/articles/finding-side-effect-loopholes-in-appworld-and-workarena-verifiers-137293.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.09142>)

Author: Richard Abrich

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [rlvr](<https://devfeed.tech/topics/rlvr.md>), [AI Bots](<https://devfeed.tech/topics/ai-bots.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [automated](<https://devfeed.tech/tags/automated.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [change](<https://devfeed.tech/tags/change.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [false-positives](<https://devfeed.tech/tags/false-positives.md>), [fields](<https://devfeed.tech/tags/fields.md>), [mutation](<https://devfeed.tech/tags/mutation.md>), [records](<https://devfeed.tech/tags/records.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [tests](<https://devfeed.tech/tags/tests.md>)

### AI overview

Mutation tests found that AppWorld and WorkArena verifiers can reward unrequested side effects even when agents complete the requested task. AppWorld accepted duplicate writes in all six tasks across two templates; adding one count assertion per template rejected the duplicates while preserving all correct runs. Three RL methods training on one template would fully reward the duplicate. WorkArena's verifier, called once after submission, accepted all 36 edited forms with an extra field. Checking after each field caught extras entered first or through drop-downs, but still missed all eight cases where a text or reference field was filled last. The tests found no false positives from planted omissions or incorrect values. The results indicate that checking requested changes alone does not reliably ensure other state remains untouched.

### Source excerpt

arXiv:2610.09142v3 Announce Type: replace Abstract: Language-model agents act for people inside software, sending payments or editing records. Benchmarks such as AppWorld and WorkArena grade them with automated verifiers, whose verdicts also become reinforcement learning (RL) rewards, so a lenient verifier rewards mistakes. Verifier audits mostly hunt for runs that pass without doing the task; we ask about runs that do it and also change something nobody asked for. We mutation-test both verifiers with hand-written edits to correct runs, designed from the verifier source; an end-state check confirms each false positive. All confirmed false positives were unrequested changes; planted omissions and wrong values produced none. AppWorld's tests passed a duplicate write in all six tasks of two templates, and three RL methods that train on one of these templates would fully reward it; one count assertion per template rejected all six duplicates while passing every correct run. WorkArena's form verifier, called once after submission, passed all 36 edited runs that filled an unrequested field. Checked after every field, it caught extra fields filled first or in drop-downs but still passed all 8 runs that filled a text or reference field last. Both verifiers check that the requested change happened but not reliably that nothing else changed.

## Meet Meta's JiTTests: LLM-Generated Tests Designed to Fail Before Risky Code Lands

DevFeed: [Meet Meta's JiTTests: LLM-Generated Tests Designed to Fail Before Risky Code Lands](<https://devfeed.tech/articles/meet-meta-s-jittests-llm-generated-tests-designed-to-fail-before-risky-code-lands-136414.md>)

Original publisher: [Read original article](<https://www.marktechpost.com/2026/11/16/meta-jittests-llm-catching-tests-privacy-bugs/>)

Author: Michal Sutter

Published: 2026-11-16T10:39:41Z

Content type: article

Language: en

Sources: [MarkTechPost](<https://devfeed.tech/sources/marktechpost.md>)

Topics: [Test automation](<https://devfeed.tech/topics/test-automation.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [Pull Request](<https://devfeed.tech/topics/pull-request.md>), [Unit testing](<https://devfeed.tech/topics/unit-testing.md>)

Tags: [1-6](<https://devfeed.tech/tags/1-6.md>), [2](<https://devfeed.tech/tags/2.md>), [2025](<https://devfeed.tech/tags/2025.md>), [2026](<https://devfeed.tech/tags/2026.md>), [3](<https://devfeed.tech/tags/3.md>), [ach](<https://devfeed.tech/tags/ach.md>), [act](<https://devfeed.tech/tags/act.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [ai-shorts](<https://devfeed.tech/tags/ai-shorts.md>), [applications](<https://devfeed.tech/tags/applications.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [code](<https://devfeed.tech/tags/code.md>), [diffs](<https://devfeed.tech/tags/diffs.md>), [editors-pick](<https://devfeed.tech/tags/editors-pick.md>), [engineer](<https://devfeed.tech/tags/engineer.md>), [enterprise-ai](<https://devfeed.tech/tags/enterprise-ai.md>), [fail](<https://devfeed.tech/tags/fail.md>), [first](<https://devfeed.tech/tags/first.md>), [for-devs](<https://devfeed.tech/tags/for-devs.md>), [harness](<https://devfeed.tech/tags/harness.md>), [llm](<https://devfeed.tech/tags/llm.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [meta](<https://devfeed.tech/tags/meta.md>), [post](<https://devfeed.tech/tags/post.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [production](<https://devfeed.tech/tags/production.md>), [security](<https://devfeed.tech/tags/security.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [staff](<https://devfeed.tech/tags/staff.md>), [system](<https://devfeed.tech/tags/system.md>), [tech-news](<https://devfeed.tech/tags/tech-news.md>), [technology](<https://devfeed.tech/tags/technology.md>), [tests](<https://devfeed.tech/tags/tests.md>), [uncategorized](<https://devfeed.tech/tags/uncategorized.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Meta's JiTTests generates executable tests aimed at failing when a risky code diff changes behavior, providing evidence for engineers to assess before code lands. The production pipeline selects risky diffs overnight, generates candidates with either a "dodgy diff" or intent-aware workflow, scores failures with an LLM ensemble, filters false positives with deterministic rules, and ranks whether to contact an engineer. The excerpt reports 74 privacy SEVs caught at the diff stage in H1 2026, a 20% decline in privacy SEV occurrence, and a 60% engineer-reported true-positive rate; an earlier deployment confirmed 8 of 41 reported catches. Research benchmarks found more candidate catches from diff-aware methods than from hardening tests, but weak catches are candidates rather than confirmed bugs. The system runs on Meta's internal infrastructure; no code, models, or API are publicly available.

### Source excerpt

Meta has moved Catching JiTTests into production. The LLM system writes tests designed to fail on risky diffs, and caught 74 privacy SEVs before landing in H1 2026, with a 60% engineer-reported true-positive rate. The post Meet Meta's JiTTests: LLM-Generated Tests Designed to Fail Before Risky Code Lands appeared first on MarkTechPost.

## I Mutation-Tested 50 Python Libraries. Tests in the Young Ones Missed 41% of Planted Faults.

DevFeed: [I Mutation-Tested 50 Python Libraries. Tests in the Young Ones Missed 41% of Planted Faults.](<https://devfeed.tech/articles/i-mutation-tested-50-python-libraries-tests-in-the-young-ones-missed-41-of-planted-faults-130835.md>)

Original publisher: [Read original article](<https://towardsdatascience.com/i-mutation-tested-50-python-libraries-tests-in-the-young-ones-missed-41-of-planted-faults/>)

Author: Gal Arav

Published: 2026-11-12T15:00:00Z

Content type: article

Language: en

Sources: [Towards Data Science](<https://devfeed.tech/sources/towards-data-science.md>)

Topics: [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [Pytest](<https://devfeed.tech/topics/pytest.md>), [Python](<https://devfeed.tech/topics/python.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [acceptance](<https://devfeed.tech/tags/acceptance.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-skill](<https://devfeed.tech/tags/agent-skill.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-assistant](<https://devfeed.tech/tags/ai-assistant.md>), [ai-development](<https://devfeed.tech/tags/ai-development.md>), [ai-testing](<https://devfeed.tech/tags/ai-testing.md>), [bug](<https://devfeed.tech/tags/bug.md>), [data-loss](<https://devfeed.tech/tags/data-loss.md>), [editor-s-picks](<https://devfeed.tech/tags/editor-s-picks.md>), [libraries](<https://devfeed.tech/tags/libraries.md>), [llm-engineering](<https://devfeed.tech/tags/llm-engineering.md>), [mutation](<https://devfeed.tech/tags/mutation.md>), [python](<https://devfeed.tech/tags/python.md>), [python-libraries](<https://devfeed.tech/tags/python-libraries.md>), [quality-assessment](<https://devfeed.tech/tags/quality-assessment.md>), [readme](<https://devfeed.tech/tags/readme.md>), [software-testing](<https://devfeed.tech/tags/software-testing.md>), [study](<https://devfeed.tech/tags/study.md>), [tests](<https://devfeed.tech/tags/tests.md>), [windows](<https://devfeed.tech/tags/windows.md>)

### AI overview

Mutation testing of 50 active Python libraries found that tests missed 41% of planted faults in 30 young libraries and 17% in 20 mature libraries. Across 2,312 planted faults, the study also identified four real library problems, confirmed by hand and reported to maintainers. A missed mutant is only a lead: some changes are unobservable, and the method cannot reveal behavior absent from the code. The article's workflow plants faults, runs the existing suite, and turns misses into questions about intended behavior. Documentation and signatures can help an LLM draft answers, but coverage was limited (about 1 in 4 questions for mature libraries, 1 in 15 for young ones), and checked drafts were often wrong, especially for young libraries. The author recommends human review of answers before generating tests. In qikly, step 1 is free; steps 2 and 3 use Gemini, Anthropic, or OpenAI APIs and require a key. A Skill can convey practices to an assistant, while the local MCP server withholds acceptance criteria from the coding agent.

### Source excerpt

Across 2,312 planted faults the study also surfaced four real library problems, from a Windows data-loss bug to README examples that do not work as written. The post I Mutation-Tested 50 Python Libraries. Tests in the Young Ones Missed 41% of Planted Faults. appeared first on Towards Data Science.

## Tests as data: ensuring AI-generated code does what we want

DevFeed: [Tests as data: ensuring AI-generated code does what we want](<https://devfeed.tech/articles/tests-as-data-ensuring-ai-generated-code-does-what-we-want-129846.md>)

Original publisher: [Read original article](<https://martinfowler.com/articles/tests-as-data.html>)

Author: Martin Fowler (martin@martinfowler.com)

Published: 2026-11-11T15:17:00Z

Content type: article

Language: en

Sources: [Martin Fowler](<https://devfeed.tech/sources/martin-fowler.md>)

Topics: [Test automation](<https://devfeed.tech/topics/test-automation.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [age-of-ai](<https://devfeed.tech/tags/age-of-ai.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-coding](<https://devfeed.tech/tags/ai-coding.md>), [ai-coding-assistants](<https://devfeed.tech/tags/ai-coding-assistants.md>), [ai-generated-code](<https://devfeed.tech/tags/ai-generated-code.md>), [all](<https://devfeed.tech/tags/all.md>), [automated](<https://devfeed.tech/tags/automated.md>), [automated-testing](<https://devfeed.tech/tags/automated-testing.md>), [code](<https://devfeed.tech/tags/code.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [coding-assistants](<https://devfeed.tech/tags/coding-assistants.md>), [data](<https://devfeed.tech/tags/data.md>), [ensuring](<https://devfeed.tech/tags/ensuring.md>), [tests](<https://devfeed.tech/tags/tests.md>), [want](<https://devfeed.tech/tags/want.md>)

### AI overview

When AI agents write both production code and tests, tests can become hard to review, tightly coupled to implementation, and vulnerable to changes that weaken their assurance. The article proposes treating developers as customers of AI-written code and making readable customer tests the main behavioral contract. Store cases as data in text files--tables for simple inputs and outputs, or a domain-specific language for richer scenarios--and use a test runner to execute them. Black-box tests can target the system, services, or focused components; tests need not all run end to end. Keeping test data separate can make unexpected changes easier to spot, though runners and DSLs still need validation. The author recommends checking test effectiveness with mutation testing and validating expected behavior against domain experts, authoritative documents, or alternative implementations. These methods add DSL and runner maintenance, do not alone assure correctness, and should be paired with techniques such as property-based testing and human exploratory testing. The article presents them for production systems, while noting prototypes may prioritize rapid learning.

### Source excerpt

Automated testing in the age of AI coding assistants is problematic: when AI agents write both production and test code, developers risk losing the assurance that automated tests protect against functional errors. Matteo Vaccari makes three claims: (1) when AI writes the code, developers become its customers; (2) therefore customer tests, not developer tests, become the primary contract between humans and AI; (3) moving that contract into test data rather than test code makes it easier to protect it from unexpected modifications by coding agents. All of this optimizes for human-readable tests that establish a contract between humans and AI. While in the past DSL parsers and custom test runners were expensive to write and maintain, now AI makes them cheap. more...

## TestPrism: Rethinking Test Evaluation Beyond a Single Reference

DevFeed: [TestPrism: Rethinking Test Evaluation Beyond a Single Reference](<https://devfeed.tech/articles/testprism-rethinking-test-evaluation-beyond-a-single-reference-127203.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.12289>)

Author: Han Li, Lingxiang Hu, Jiacheng Huang, Ziqian Jiang, Jingkai Luo, Wei Gao, Yunfan Tan, Zun Wang, Jiaheng Liu

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [assertions](<https://devfeed.tech/tags/assertions.md>), [behaviors](<https://devfeed.tech/tags/behaviors.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agent](<https://devfeed.tech/tags/coding-agent.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [construction](<https://devfeed.tech/tags/construction.md>), [cross-validation](<https://devfeed.tech/tags/cross-validation.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [function](<https://devfeed.tech/tags/function.md>), [generation](<https://devfeed.tech/tags/generation.md>), [harness](<https://devfeed.tech/tags/harness.md>), [improvement](<https://devfeed.tech/tags/improvement.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [reference](<https://devfeed.tech/tags/reference.md>), [test](<https://devfeed.tech/tags/test.md>)

### AI overview

Single-reference evaluation substantially overstates coding agents' test quality in the reported benchmark: across 14 baseline configurations, single-reference success was 59.67%, while TestPrism's Joint Success Function was 28.00%. The metric requires tests to fail on the initial program state, pass every valid implementation, and fail every invalid one. The 300-task benchmark draws from 17 sources and includes 3,000 candidate implementations, evenly split between valid and invalid. The authors identify missed behaviors, unsupported assertions, and faulty test construction. Their TestHelix approach combines varied synthesis of test and repair pairs, peer cross-validation, and recursive self-improvement; across two models, it improved Joint Success Function by 8.67-9.00 percentage points over native harness comparators in the TestHelix evaluation.

### Source excerpt

arXiv:2610.12289v2 Announce Type: replace Abstract: Large language model (LLM) coding agents have advanced test generation across diverse programming tasks. However, the common practice of evaluating tests against a single reference solution overlooks alternative valid implementations and can overstate test quality. We introduce TestPrism, comprising 300 test tasks from 17 sources and 3000 candidate implementations, evenly split between valid and invalid solutions. Its primary metric, Joint Success Function, requires the generated tests to fail on the initial program state, accept every valid candidate, and reject every invalid candidate. Across fourteen baseline coding agent configurations, Joint Success Function reaches only 28.00%, whereas single reference success reaches 59.67%. Our analysis reveals missed behaviors, unsupported assertions, and faulty test construction. To address these weaknesses, we introduce TestHelix, which combines heterogeneous synthesis of test and repair pairs with peer cross validation and recursive self improvement (RSI). Across two models, TestHelix improves Joint Success Function by 8.67 to 9.00 percentage points over the native harness comparators in the TestHelix evaluation

## Mutation Testing for Multi-Agent Workflows

DevFeed: [Mutation Testing for Multi-Agent Workflows](<https://devfeed.tech/articles/mutation-testing-for-multi-agent-workflows-127213.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09895>)

Author: Nafiseh Kahani, Mojtaba Bagherzadeh

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>)

Tags: [agent-workflows](<https://devfeed.tech/tags/agent-workflows.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [multi-agent](<https://devfeed.tech/tags/multi-agent.md>), [multi-agent-workflows](<https://devfeed.tech/tags/multi-agent-workflows.md>), [mutation](<https://devfeed.tech/tags/mutation.md>), [mutation-testing](<https://devfeed.tech/tags/mutation-testing.md>), [testing](<https://devfeed.tech/tags/testing.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

### AI overview

The study adapts mutation testing to assess whether suites detect faults in multi-agent workflow coordination, not just failures in end-task success. Fourteen operators perturb workflow manifests across delegation, tool access, restrictions, and agents or instructions. Suites are run on original and mutated workflows through the OpenAI Agents SDK, with three kill criteria and three repeats decided by majority vote. Across ten workflows, 1,084 mutants, and 29,601 agent runs, the strongest baseline suite reported reaches 0.717 for structural faults but 0.559 for safety faults. Restriction removal is especially hard for baseline suites to catch (0.19-0.28, compared with 1.000 for the full-obligation reference). The authors conclude that task-success-oriented suites catch only about half of permission violations; the abstract does not provide broader validation beyond this evaluation.

### Source excerpt

arXiv:2611.09895v1 Announce Type: new Abstract: Multi-agent large language model (LLM) systems are defined by explicit workflows, agents, tools, tool-access rules, and delegation paths. Most evaluations, however, report end-to-end task success rather than whether a test suite can detect faults in this coordination structure, such as a removed delegation, a silently removed restriction, a misrouted tool edge, or a faulty tool implementation. We adapt mutation analysis to this setting, fourteen operators in four families (delegation, allowed-tool, restriction, agent/instruction) perturb a normalized workflow manifest, and three kill criteria score whether a suite catches the change, namely K-Struct on the observed coordination, KOutcome on the task verdict or step sequence, and K-Safety on a forbidden tool call. Every criterion is evaluated by executing each suite against the original and the mutant through the OpenAI Agents SDK, with k=3 repeats and a majority vote. On ten OpenAI Agents SDK-derived workflows (49 reachable agents, 47 tools, 403 obligations), the evaluation executes 29,601 agent runs over a pool of 1,084 mutants. We use T-Struct only as a full-obligation reference point (MSstrict=1.000 under all three criteria) and measure how much the baseline suites miss, the bestperforming one (T-Dev*) reaches 0.717 (K-Struct) and only 0.559 (K-Safety). Restriction-removal (RES-REM) is the least detected fault for the baselines (0.19 to 0.28 vs 1.000). Executing the suites also allows the obligation-preserving operators (TOOL-FAULT, AG-SWAP, AG-PROMPTDROP) to be evaluated. A manifest-only oracle would not witness these changes. In this setting, tasksuccess- style suites catch only about half of permission violations.

## TATG: Tracking-Aware Testing Objective for LLM-based Test Generation

DevFeed: [TATG: Tracking-Aware Testing Objective for LLM-based Test Generation](<https://devfeed.tech/articles/tatg-tracking-aware-testing-objective-for-llm-based-test-generation-124008.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.03194>)

Author: Guancheng Wang, Qinghua Xu, Lionel Briand

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Test automation](<https://devfeed.tech/topics/test-automation.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [assertions](<https://devfeed.tech/tags/assertions.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [codex](<https://devfeed.tech/tags/codex.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [generation](<https://devfeed.tech/tags/generation.md>), [java](<https://devfeed.tech/tags/java.md>), [llm](<https://devfeed.tech/tags/llm.md>), [mutation](<https://devfeed.tech/tags/mutation.md>), [objectives](<https://devfeed.tech/tags/objectives.md>), [static-analysis](<https://devfeed.tech/tags/static-analysis.md>), [test](<https://devfeed.tech/tags/test.md>), [testing](<https://devfeed.tech/tags/testing.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [tracking](<https://devfeed.tech/tags/tracking.md>), [workflow](<https://devfeed.tech/tags/workflow.md>)

### AI overview

TATG tracks individual unit-testing objectives across generation iterations, using requirements derived from static analysis and dynamic feedback to reduce redundant work and identify unresolved goals. Its two-stage workflow first targets structural coverage, then hardens assertions using mutation guidance. In an evaluation on 141 complex Java methods, it reported average gains over state-of-the-art baselines of 22.15 percentage points in line coverage, 20.14 in branch coverage, and 37.66 in mutation score. Under the same open-weight model configurations, it also exceeded Codex and Claude Code on average coverage and mutation scores while producing fewer tests and using fewer tokens. These results are limited to the reported evaluation.

### Source excerpt

arXiv:2607.03194v2 Announce Type: replace Abstract: Complex Java methods challenge automated unit test generation, as achieving high coverage and fault detection requires satisfying intricate testing requirements involving program states, execution paths, and assertions. Existing LLM-based approaches leverage program context, static analysis, and dynamic feedback but lack explicit tracking of individual testing requirements across iterations, potentially leading to redundant generation and unresolved testing objectives. We present TATG, a tracking-aware LLM-based unit test generation framework that unifies structural coverage and mutation-based fault detection. TATG introduces a structured objective representation capturing testing requirements derived from static analysis and dynamic feedback, enabling fine-grained tracking of their satisfaction status. It employs a two-stage workflow that first improves structural coverage and subsequently strengthens assertions through mutation-guided hardening. We evaluate TATG on 141 complex Java methods against state-of-the-art baselines, including KTester and PANTA. Results show average improvements of 22.15, 20.14, and 37.66 percentage points in line coverage, branch coverage, and mutation score, respectively. TATG also outperforms general-purpose coding assistants, including Codex and Claude Code, under the same open-weight model configurations, achieving higher average coverage and mutation scores while generating fewer tests and consuming fewer tokens. These findings highlight the value of explicitly tracking testing objectives for effective LLM-based test generation.

## Clear your Kotlin's code mutation report using Jev

DevFeed: [Clear your Kotlin's code mutation report using Jev](<https://devfeed.tech/articles/clear-your-kotlin-s-code-mutation-report-using-jev-122572.md>)

Original publisher: [Read original article](<https://le0nidas.gr/2026/11/08/clear-your-kotlins-code-mutation-report-using-jev/>)

Author: Leonidas Partsas

Published: 2026-11-08T16:21:55Z

Content type: tutorial

Language: en

Sources: [le0nidas](<https://devfeed.tech/sources/le0nidas.md>)

Topics: [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [Kotlin](<https://devfeed.tech/topics/kotlin.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [bytecode](<https://devfeed.tech/tags/bytecode.md>), [code](<https://devfeed.tech/tags/code.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [coroutines](<https://devfeed.tech/tags/coroutines.md>), [csv](<https://devfeed.tech/tags/csv.md>), [java](<https://devfeed.tech/tags/java.md>), [jev](<https://devfeed.tech/tags/jev.md>), [jvm](<https://devfeed.tech/tags/jvm.md>), [kotlin](<https://devfeed.tech/tags/kotlin.md>), [mutation](<https://devfeed.tech/tags/mutation.md>), [mutation-testing](<https://devfeed.tech/tags/mutation-testing.md>), [noul](<https://devfeed.tech/tags/noul.md>), [productivity](<https://devfeed.tech/tags/productivity.md>), [report](<https://devfeed.tech/tags/report.md>), [system](<https://devfeed.tech/tags/system.md>), [test](<https://devfeed.tech/tags/test.md>), [testing](<https://devfeed.tech/tags/testing.md>), [tests](<https://devfeed.tech/tags/tests.md>)

### AI overview

Kotlin's compiler-generated bytecode can make PIT reports noisy, since PIT may report mutations in code that was not present in the original source. The described cleanup filters PIT's report to surviving mutations, optionally simplifies entries, and removes mutations in compiler-inserted methods by checking whether the method appears in the source file. For mutations inside existing methods, scripts prepare the method and mutation description for Jev, which scores whether the change could apply to the original code; scores below 0.5 are discarded. An agent then uses the remaining valid surviving mutations to add or strengthen regression tests, without changing production semantics. The excerpt describes this process but does not provide the scripts or their implementation details.

### Source excerpt

Mutation testing is making a come back and rightfully so. Now that we have agents doing all the testing for us we need to make sure that they've written all necessary tests, making our test suite stronger and our project better. Mutation testing In a nutshell, mutation testing is a way to check if the ... Continue reading Clear your Kotlin's code mutation report using Jev ->

## Recursively Testing and Vibe Debugging a Google AI Studio Generated Vibe-Coded Database Software Application

DevFeed: [Recursively Testing and Vibe Debugging a Google AI Studio Generated Vibe-Coded Database Software Application](<https://devfeed.tech/articles/recursively-testing-and-vibe-debugging-a-google-ai-studio-generated-vibe-coded-database-software-application-120800.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.06952>)

Author: Christian Mancas, Cristina Mancas

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [Application Development](<https://devfeed.tech/topics/application-development.md>), [bugfixes](<https://devfeed.tech/topics/bugfixes.md>), [App](<https://devfeed.tech/topics/app.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-studio](<https://devfeed.tech/tags/ai-studio.md>), [application](<https://devfeed.tech/tags/application.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [database](<https://devfeed.tech/tags/database.md>), [debugging](<https://devfeed.tech/tags/debugging.md>), [google](<https://devfeed.tech/tags/google.md>), [google-ai-studio](<https://devfeed.tech/tags/google-ai-studio.md>), [software](<https://devfeed.tech/tags/software.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

The paper presents a sound, complete, and optimal methodology for recursively testing and "vibe debugging" vibe-coded software, extending the concept to non-developers and languages beyond English. It applies the methodology to a medium-complexity database application generated with Google AI Studio Pro and reports that non-developers could reach a bug-free, functional, satisfactory version. The authors also report that despite the tool's built-in language analysis, software development, and automated testing, its generated code still contains many errors, making further testing and debugging necessary.

### Source excerpt

arXiv:2611.06952v1 Announce Type: cross Abstract: This research paper proposes a broader definition of vibe coding, extending the original one with non-developers and natural languages other than English, and introduces the notion of vibe debugging, as well as a sound, complete, and optimal algorithmic methodology for testing and vibe debugging vibe-coded software applications. This methodology is illustrated by applying it to a vibe-coded real-world database application of medium complexity generated with Google AI Studio Pro. We prove that even non-developers can obtain a final bug-free and functional satisfactory application version, even in languages other than English. We also prove that, although Google AI Studio Pro has sophisticated built-in engines for both language analysis, software development, and automated testing, it is still making lot of mistakes, so testing and vibe debugging of its generated code is a must.

## Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations

DevFeed: [Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations](<https://devfeed.tech/articles/accepted-but-wrong-silent-failures-of-llm-code-repairs-after-api-migrations-120299.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.08269>)

Author: Xiaobo Chen, Genliang Zhu, Ziyuan Wang, Chu Wang

Published: 2026-11-06T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [API](<https://devfeed.tech/topics/api.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [assertions](<https://devfeed.tech/tags/assertions.md>), [audit](<https://devfeed.tech/tags/audit.md>), [code](<https://devfeed.tech/tags/code.md>), [contracts](<https://devfeed.tech/tags/contracts.md>), [cpython](<https://devfeed.tech/tags/cpython.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [edge](<https://devfeed.tech/tags/edge.md>), [failed](<https://devfeed.tech/tags/failed.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [frontier-model](<https://devfeed.tech/tags/frontier-model.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [gpt-5-6](<https://devfeed.tech/tags/gpt-5-6.md>), [legacy](<https://devfeed.tech/tags/legacy.md>), [llm](<https://devfeed.tech/tags/llm.md>), [local](<https://devfeed.tech/tags/local.md>), [local-model](<https://devfeed.tech/tags/local-model.md>), [migration](<https://devfeed.tech/tags/migration.md>), [migrations](<https://devfeed.tech/tags/migrations.md>), [model](<https://devfeed.tech/tags/model.md>)

### AI overview

In a preregistered study of 73 tasks spanning 60 documented changes in CPython, Node.js, and SQLite, typical-input release checks accepted 41.0% of wrong LLM repairs. Adding one edge input based on the release note reduced that share to 6.5%, and to 6.1% on 15 unseen clusters, without lowering correct completion across the tested open-weight model families. In harder repair menus, the edge input and a frontier-model-generated test still let 16.8% and 15.3% of hard wrong programs pass. Self-tests helped less, and two local-model test writers failed their preregistered tests. An audit disputed the strictness of 34 contracts; removing those checks changed no verdicts, though class-label disputes remain unresolved.

### Source excerpt

arXiv:2611.08269v1 Announce Type: cross Abstract: After a runtime or API migration, an LLM-repaired procedure can run cleanly yet return wrong results, and a release check on typical test inputs ships it. We report a preregistered study of 73 tasks covering 60 documented changes in CPython, Node.js and SQLite. Every candidate is executed on the real runtimes against withheld contracts. Over the complete repair space, the all-assertions gate on typical inputs released 0.410 of wrong repairs. One edge input written from the release note cut this to 0.065, and to 0.061 on 15 unseen clusters. Three inputs from a blind GPT-5.6-luna writer that never saw the program reached 0.070. Two further frozen stages enlarged the menu to 1491 fragments and then added 233 hard distractors that match the legacy output on typical inputs. The edge input (0.058) and the frontier-model writer (0.058) passed their preregistered tests. Yet 0.168 and 0.153 of the hard wrong programs still passed. Two local-model writers failed their preregistered tests. Across four model families, the edge input cut wrong releases per attempt in all three open-weight families on all three menus, for example from 0.233 to 0.057, without lowering correct completion. Self-tests helped less. A blind AI audit flagged 34 contracts as stricter than their requirements. Dropping the disputed checks changed 0 verdicts, but class-label disputes remain open. No model-written code was executed. Artifacts are available from the authors on request.

## Data Access Testing With @testcontainers

DevFeed: [Data Access Testing With @testcontainers](<https://devfeed.tech/articles/data-access-testing-with-testcontainers-117746.md>)

Original publisher: [Read original article](<https://blog.master.dev/data-access-testing-with-testcontainers/>)

Author: Adam Rackis

Published: 2026-11-04T12:58:05Z

Content type: tutorial

Language: en

Sources: [Master.dev Blog RSS Feed](<https://devfeed.tech/sources/master-dev-blog-rss-feed.md>)

Topics: [Integration testing](<https://devfeed.tech/topics/integration-testing.md>), [Testcontainers](<https://devfeed.tech/topics/testcontainers.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [Unit testing](<https://devfeed.tech/topics/unit-testing.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [blog-post](<https://devfeed.tech/tags/blog-post.md>), [ci-cd](<https://devfeed.tech/tags/ci-cd.md>), [claude](<https://devfeed.tech/tags/claude.md>), [cli](<https://devfeed.tech/tags/cli.md>), [container](<https://devfeed.tech/tags/container.md>), [containers](<https://devfeed.tech/tags/containers.md>), [crud](<https://devfeed.tech/tags/crud.md>), [database](<https://devfeed.tech/tags/database.md>), [database-schema](<https://devfeed.tech/tags/database-schema.md>), [docker](<https://devfeed.tech/tags/docker.md>), [docker-container](<https://devfeed.tech/tags/docker-container.md>), [drizzle](<https://devfeed.tech/tags/drizzle.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [schema](<https://devfeed.tech/tags/schema.md>), [testcontainers](<https://devfeed.tech/tags/testcontainers.md>), [testing](<https://devfeed.tech/tags/testing.md>), [tests](<https://devfeed.tech/tags/tests.md>), [typescript](<https://devfeed.tech/tags/typescript.md>), [unit-tests](<https://devfeed.tech/tags/unit-tests.md>)

### AI overview

For complex data access logic, an isolated database integration test can verify that real inserts and queries preserve the expected object structure. The article uses `@testcontainers/postgresql` to start a temporary Postgres container in Vitest, applies a Drizzle schema with `drizzle-kit push`, and exercises workout insertion and retrieval through the application's actual code. A round-trip assertion catches errors in joins, writes, or ordering without tying the test to particular table rows, which the author says makes it less fragile to schema changes. The tradeoff is setup time: the suite takes over a second to start, and Docker must be available locally; GitHub Actions Ubuntu runners are said to include it. The author recommends reserving this approach for meaningful, non-trivial database logic rather than basic CRUD tests.

### Source excerpt

Testing data access code has always been awkward: unit tests want to be fast and isolated, and a real database is neither. With Docker and @testcontainers you can spin up a throwaway Postgres inside your Vitest suite, push your Drizzle schema into it, and verify that your real insert and query code round-trips correctly.

## MCPGuard: Finding Conformance Gaps in MCP SDKs via Dual-Track Analysis and Dynamic Adjudication

DevFeed: [MCPGuard: Finding Conformance Gaps in MCP SDKs via Dual-Track Analysis and Dynamic Adjudication](<https://devfeed.tech/articles/mcpguard-finding-conformance-gaps-in-mcp-sdks-via-dual-track-analysis-and-dynamic-adjudication-116178.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.03662>)

Author: Xikai Sun, Kebin Liu, Fan Dang, Yawen Zheng, Zihao Yang, Yunhao Liu

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [MSP MCP](<https://devfeed.tech/topics/msp-mcp.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [automated](<https://devfeed.tech/tags/automated.md>), [conformance-testing](<https://devfeed.tech/tags/conformance-testing.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [framework](<https://devfeed.tech/tags/framework.md>), [json-schema](<https://devfeed.tech/tags/json-schema.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [llms](<https://devfeed.tech/tags/llms.md>), [mcp](<https://devfeed.tech/tags/mcp.md>), [mcp-servers](<https://devfeed.tech/tags/mcp-servers.md>), [python](<https://devfeed.tech/tags/python.md>), [sdks](<https://devfeed.tech/tags/sdks.md>), [specification](<https://devfeed.tech/tags/specification.md>), [testing](<https://devfeed.tech/tags/testing.md>), [typescript](<https://devfeed.tech/tags/typescript.md>)

### AI overview

MCPGuard detects specification requirements that MCP SDKs fail to enforce, gaps that endpoint-focused conformance tools may miss. It combines a structural track that aligns JSON Schema constraints with normalized SDK models and a behavioral track that audits code for omissions of normative prose rules. An evidence-aware adjudicator then reviews full source context and builds proof-of-concept tests to validate candidates. On official Python 1.26.0 and TypeScript 1.29.0 SDKs, it confirmed 13 and 19 findings, respectively; two black-box baselines exposed none of them. Tests on open MCP servers indicate these gaps can propagate downstream. The abstract presents MCPGuard as complementing conventional endpoint-oriented suites.

### Source excerpt

arXiv:2611.03662v1 Announce Type: cross Abstract: The Model Context Protocol (MCP) connects large language model agents to external tools and services through official SDKs that implement message validation and session logic. Specification constraints omitted by these SDKs may propagate to downstream applications, highlighting the importance of SDK-level conformance testing. Yet existing MCP conformance tools mainly exercise running endpoints and reveal little about whether SDK models enforce specification requirements. We present MCPGuard, an automated LLM-based framework for detecting specification-to-SDK conformance gaps. Its structural track aligns JSON Schema constraints with normalized SDK models and produces constraint-specific evidence. Its behavioral track extracts normative prose rules, retrieves bounded function-level slices, and audits them with LLMs for candidate omissions. An evidence-aware dynamic adjudicator then inspects complete source context and builds PoCs to validate candidates. A shared kernel with language-specific adapters enables analysis across SDKs. On the official Python 1.26.0 and TypeScript 1.29.0 SDKs, MCPGuard confirms 13 and 19 findings, respectively, none of which are exposed by two black-box baselines. Tests on open MCP servers further show that these gaps can systematically propagate downstream. By making SDK conformance of specification constraints directly testable, MCPGuard complements conventional endpoint-oriented conformance suites.

## Harvesting SWE Benchmarks from Wild Vibe Coding Sessions

DevFeed: [Harvesting SWE Benchmarks from Wild Vibe Coding Sessions](<https://devfeed.tech/articles/harvesting-swe-benchmarks-from-wild-vibe-coding-sessions-113527.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.02440>)

Author: Ao Lu, Yuanhao Zeng, Shaocheng Lan, Sijie Xu, Jiacheng Bao, Kuang Yan, Heng Lin, Kan Ren

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [commits](<https://devfeed.tech/tags/commits.md>), [container](<https://devfeed.tech/tags/container.md>), [context](<https://devfeed.tech/tags/context.md>), [conversations](<https://devfeed.tech/tags/conversations.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [execution](<https://devfeed.tech/tags/execution.md>), [filter](<https://devfeed.tech/tags/filter.md>), [framework](<https://devfeed.tech/tags/framework.md>), [harness](<https://devfeed.tech/tags/harness.md>), [history](<https://devfeed.tech/tags/history.md>), [languages](<https://devfeed.tech/tags/languages.md>), [logs](<https://devfeed.tech/tags/logs.md>), [model](<https://devfeed.tech/tags/model.md>), [sessions](<https://devfeed.tech/tags/sessions.md>), [vibe-coding](<https://devfeed.tech/tags/vibe-coding.md>)

### AI overview

SessionFarmer turns coding-agent conversations and repository checkpoints into executable software engineering tasks by filtering unrelated sessions, grouping work around engineering goals, reconstructing hermetic containers, and creating context-based task statements, rubrics, and tests. Its closed-loop test refinement uses requirement-breaking mutants and a blind solver to reduce false rejections of valid alternatives and false acceptance of incomplete solutions. The resulting CropChat benchmark contains 242 tasks from 61 public repositories across 14 languages; CropEnterprise is built from proprietary logs. In a CropEnterprise audit, original developers endorsed the vast majority of reconstructed episodes, statements, and rubrics. Full-episode context captured substantially more user requirements than single-turn or single-session inputs. Test refinement improved balanced accuracy against reference patch labels by 7-9 percentage points across solvers. On CropChat, the strongest of 12 model-harness configurations reached 39.3% Pass@1, while harness choice shifted results by up to 8.4 points for the same model.

### Source excerpt

arXiv:2611.02440v1 Announce Type: new Abstract: Development teams need reproducible tasks from their own repositories and workflows to evaluate coding agents. Conversational developer-agent collaboration ("vibe coding") produces abundant coding sessions and repository checkpoints, but requirements evolve across turns, unrelated conversations co-occur with commits, and engineering efforts can span multiple sessions. We present SessionFarmer, an automated framework that turns these records into self-contained, executable software engineering tasks. It analyzes dialogue and commit history to filter unrelated sessions and group multi-turn, potentially multi-session interactions into work episodes centered on one engineering goal. For each episode, it reconstructs a hermetic execution container, synthesizes a task statement from the full context, and authors repository-grounded engineering-quality rubrics and unit tests. Closed-loop test refinement uses requirement-breaking mutants and a blind solver to reduce false rejections of valid alternatives and false acceptances of incomplete solutions. We construct CropChat, an open benchmark of 242 tasks across 61 public repositories and 14 programming languages, and CropEnterprise from proprietary enterprise logs. In a CropEnterprise audit, the original developers endorse the vast majority of reconstructed episode boundaries, task statements, and quality rubrics. Full-episode context covers substantially more user requirements than single-turn or single-session inputs. Test refinement cuts false-rejection rates and raises balanced accuracy against reference patch labels by 7-9 percentage points across solvers. Across 12 model-harness configurations on CropChat, the strongest setup achieves 39.3% Pass@1, while harness choice shifts Pass@1 by up to 8.4 percentage points for the same model.

## Causality-Based Fault Localization

DevFeed: [Causality-Based Fault Localization](<https://devfeed.tech/articles/causality-based-fault-localization-113504.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.01292>)

Author: Melane Navaratnarajah, Aidan Dakhama, Tom Yaacov, David A. Kelly, Karine Even-Mendoza, Hana Chockler

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [causality](<https://devfeed.tech/topics/causality.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>)

Tags: [approach](<https://devfeed.tech/tags/approach.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [causality](<https://devfeed.tech/tags/causality.md>), [computer-science](<https://devfeed.tech/tags/computer-science.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [generation](<https://devfeed.tech/tags/generation.md>), [localization](<https://devfeed.tech/tags/localization.md>), [location](<https://devfeed.tech/tags/location.md>), [metric](<https://devfeed.tech/tags/metric.md>), [mutation](<https://devfeed.tech/tags/mutation.md>), [program](<https://devfeed.tech/tags/program.md>), [rank](<https://devfeed.tech/tags/rank.md>), [required](<https://devfeed.tech/tags/required.md>), [responsibility](<https://devfeed.tech/tags/responsibility.md>), [results](<https://devfeed.tech/tags/results.md>), [test](<https://devfeed.tech/tags/test.md>), [tests](<https://devfeed.tech/tags/tests.md>), [theory](<https://devfeed.tech/tags/theory.md>), [tool](<https://devfeed.tech/tags/tool.md>)

### AI overview

CBFL ranks likely fault locations by estimating each code location's causal responsibility for a failing test, then uses a distance metric to prioritize minimal fixes and penalize fixes that break tests. Its implementation, SoftReX, reportedly improves Mean Reciprocal Rank and Top-1 by 16% over state-of-the-art mutation-based fault localization tools when evaluated with the same mutants. The paper also describes a program-structure-aware mutation generation strategy that needs 2.6 times fewer mutants evaluated than state-of-the-art methods while remaining competitive in performance. These results are reported in the supplied abstract; experimental details are not included.

### Source excerpt

arXiv:2611.01292v1 Announce Type: new Abstract: We introduce Causality-Based Fault Localization (CBFL), a novel approach to fault localization grounded in the theory of actual causality. We use the notion of causal responsibility to formalize and define the contribution of a given location in the source code under the context of a failing test. We also utilize a distance metric to prioritize minimal fixes and penalize broken tests, leading to more interpretable and fixable fault localization. We implemented CBFL in a tool, SoftReX, and our evaluation results demonstrate that SoftReX outperforms state-of-the-art Mutation-Based Fault Localization (MBFL) tools based on Mean Reciprocal Rank (MRR) and Top-1 by 16% on both metrics, when using the same set of mutants. We also propose a sophisticated mutation generation strategy based on the structure and representation of the program to improve the accuracy of our ranked list, which outperforms state-of-the-art methods by 2.6x in terms of required mutants evaluated while remaining competitive in performance.

## CodeLift: Your Test Suite Is Secretly a Dense SWE Curriculum

DevFeed: [CodeLift: Your Test Suite Is Secretly a Dense SWE Curriculum](<https://devfeed.tech/articles/codelift-your-test-suite-is-secretly-a-dense-swe-curriculum-111457.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.17224>)

Author: Junjielong Xu, Huachao Zhu, Alyssa Wu, Zhichao Hu, Pinjia He

Published: 2026-11-02T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [rlvr](<https://devfeed.tech/topics/rlvr.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [actions](<https://devfeed.tech/tags/actions.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [patch](<https://devfeed.tech/tags/patch.md>), [regressions](<https://devfeed.tech/tags/regressions.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [repository](<https://devfeed.tech/tags/repository.md>), [reward](<https://devfeed.tech/tags/reward.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [tests](<https://devfeed.tech/tags/tests.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

CodeLift calibrates test pass-rate rewards so software engineering agents learn from partially correct patches without letting numerous easy or already satisfied tests dominate. Each iteration groups tests that fail the same patches and patches that pass the same tests, then weights test groups according to how few patch groups pass them. It applies this scoring rule to repository snapshots within a rollout, giving credit to useful actions even if later actions cause regressions. The abstract reports consistent pass@1/2/4/8 improvements over binary-reward and pass-rate baselines on SWE benchmarks, with analyses finding dense, stable learning signals.

### Source excerpt

arXiv:2610.17224v2 Announce Type: replace Abstract: The pass-all-test binary reward has become a standard recipe for training software engineering agents, but its sparsity eliminates difference of partially correct patches. The test pass-rate is a natural alternative. However, we find that it is not a reliable indicator of patch quality. A functionality being tested can be covered by either one challenging test or one hundred trivial tests, where the majority might be already satisfied pass-to-pass tests. We introduce CodeLift, a reinforcement learning method that dynamically calibrates pass-rate rewards, turning the test suite into an adaptive curriculum. At each iteration, CodeLift groups tests that kill the same patches and patches that pass the same tests, then gives more weight to test groups that fewer patch groups pass. This reduces the impact of majority simple tests and focuses more on what is still challenging. Within each rollout, CodeLift applies the same scoring rule to repository snapshots, assigning credit to useful actions even when later ones introduce regressions. Experiments on SWE benchmarks show that CodeLift consistently improves pass@1/2/4/8 over binary and pass-rate baselines. Analyses show that CodeLift provides dense, stable learning signals and unlocks the inherent curriculum in the test suite.

## Detecting Incorrect LLM Explanations for Automated Program Repair

DevFeed: [Detecting Incorrect LLM Explanations for Automated Program Repair](<https://devfeed.tech/articles/detecting-incorrect-llm-explanations-for-automated-program-repair-107375.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.38311>)

Author: Lam Nguyen Tung, Chenxi Gu, Xiaoning Du, Haoye Tian, Aldeida Aleti, Neelofar Neelofar

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [patches](<https://devfeed.tech/topics/patches.md>), [Automated reasoning](<https://devfeed.tech/topics/automated-reasoning.md>)

Tags: [cs-se](<https://devfeed.tech/tags/cs-se.md>), [llm](<https://devfeed.tech/tags/llm.md>), [metamorphic-testing](<https://devfeed.tech/tags/metamorphic-testing.md>), [repair](<https://devfeed.tech/tags/repair.md>)

### AI overview

MetaReJ detects potentially incorrect explanations from automated program repair (APR) systems by testing whether an LLM's reasoning remains consistent across related repair scenarios. It transforms a buggy program into variants that preserve or alter its bug, asks the LLM how to repair each one, and treats contradictions captured as metamorphic-relation violations as evidence against the original explanation. This approach avoids external verifiers and aims to elicit more diverse responses than paraphrasing or re-prompting the same input, which can repeat an error. On a manually labeled set of 1,756 explanations from multiple LLM-based APR systems, MetaReJ reached an MCC of up to 0.43 and outperformed prior methods by up to 333.3% in MCC.

### Source excerpt

arXiv:2610.38311v1 Announce Type: new Abstract: LLM-generated explanations help developers understand automated program repair (APR) rationale and assess patch quality. Incorrect explanations can mislead developers about a repair's reliability, making their detection important. However, manually inspecting the growing volume of generated repairs is impractical, motivating automated detection. Existing methods mainly target incorrect answers from general question-answering LLMs, often by sampling multiple responses and detecting contradictions. Applied to APR, they face two limitations: (1) they may depend on external verifiers that are unavailable or lack APR-specific reasoning; and (2) sampling strategies such as paraphrasing or re-prompting the same input often yield superficial, redundant responses that repeat the original error rather than expose it. We propose MetaReJ, the first method designed to detect incorrect APR explanations using metamorphic testing. Inspired by debate and viva examinations, MetaReJ challenges explanations through an argumentative strategy without requiring external verifiers. It applies novel metamorphic relations (MRs) to recursively question the LLM under equivalent and non-equivalent repair scenarios and detect contradictions. To obtain diverse samples, MetaReJ transforms the buggy program into variants that preserve or alter the underlying bug, then asks the LLM how to repair each variant. Contradictions among responses are encoded as MR violations, which indicate that the original explanation is incorrect. We evaluate MetaReJ on a manually labelled dataset of 1,756 explanations from multiple LLM-based APR systems. MetaReJ achieves an MCC of up to 0.43, outperforming prior methods by up to 333.3% in MCC.

## Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements

DevFeed: [Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements](<https://devfeed.tech/articles/benchmarking-llm-agents-for-behaviorally-adequate-test-generation-from-software-requirements-103159.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.36584>)

Author: Binhang Qi, Yiming Liu, Yun Lin, Yangjie Zhou, Hailong Sun, Jin Song Dong

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Test automation](<https://devfeed.tech/topics/test-automation.md>), [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [Requirements](<https://devfeed.tech/topics/requirements.md>), [Behavior-driven development](<https://devfeed.tech/topics/bdd.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Dataset development](<https://devfeed.tech/topics/dataset-development.md>), [Spec Driven Development](<https://devfeed.tech/topics/spec-driven-development.md>), [codex](<https://devfeed.tech/topics/codex.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [bdd](<https://devfeed.tech/tags/bdd.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [codex](<https://devfeed.tech/tags/codex.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [execution](<https://devfeed.tech/tags/execution.md>), [file](<https://devfeed.tech/tags/file.md>), [frameworks](<https://devfeed.tech/tags/frameworks.md>), [generate](<https://devfeed.tech/tags/generate.md>), [generation](<https://devfeed.tech/tags/generation.md>), [harness](<https://devfeed.tech/tags/harness.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [requirements](<https://devfeed.tech/tags/requirements.md>), [software](<https://devfeed.tech/tags/software.md>), [test](<https://devfeed.tech/tags/test.md>)

### AI overview

Req2Test-Bench evaluates whether agents can generate tests from Gherkin requirements and target interfaces when behavior implementations are withheld. It contains 5,086 execution-validated instances from eight open-source projects across five languages and four BDD frameworks. Adequacy is measured strictly by kill-completeness: a generated test must kill every mutant killed by the developer-written test. Across eight agent-model-reasoning configurations, the best, Codex with DeepSeek-V4.1-Flash, generated passing tests for 82.8% of instances, but only 33.8% were behaviorally adequate. The authors report sharply diminishing adequacy gains as agent scale and generation cost increase, and identify ambiguous setup and implicit expected outcomes in underspecified requirements as sources of information gaps. They suggest requirement-grounded harnesses to steer agent nondeterminism as a possible improvement.

### Source excerpt

arXiv:2610.36584v1 Announce Type: cross Abstract: As large language model agents increasingly compile natural-language requirements into repository-level code, requirement-to-test generation grows in importance to provide validation constraints (or harness) to guide code generation. Yet it remains unclear whether current agents can generate tests that validate the required behavior when the implementation of that behavior is withheld. Existing test-generation benchmarks mostly focus on white-box testing, whereas the requirement--test-related study usually requires evaluation on semantic alignment between the given documents and the generated tests. We introduce Req2Test-Bench, a benchmark containing 5,086 execution-validated instances recovered from developer-authored behavior-driven development (BDD) artifacts in 8 open-source projects, spanning 5 programming languages and 4 BDD frameworks. Each instance pairs a natural-language requirement in Gherkin with its developer-written Test Implementation. Given the requirement, target interfaces of the required behavior, an agent must generate a complete, potentially multi-file Test Implementation. Evaluation of 8 agent--model--reasoning configurations reveals a substantial gap between execution success and behavioral adequacy, which is measured strictly as kill-completeness: a kill-complete test kills every mutant that the ground-truth test kills. The best-performing configuration, Codex with DeepSeek-V4.1-Flash, produces passing tests for 82.8% of instances, yet only 33.8% are behaviorally adequate. We find that (1) scaling up agents yields sharply diminishing returns in behavioral adequacy relative to generation cost; (2) under-specified requirements cause the information gap, including ambiguous setup and implicit expected outcomes; and (3) requirement-grounded harnesses that steer agent non-determinism offer a promising direction for improvement.

## Toward Circuit-Aware Mutation Testing for Quantum Programs: Exact Visibility and Execution Planning

DevFeed: [Toward Circuit-Aware Mutation Testing for Quantum Programs: Exact Visibility and Execution Planning](<https://devfeed.tech/articles/toward-circuit-aware-mutation-testing-for-quantum-programs-exact-visibility-and-execution-planning-103652.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.36976>)

Author: Andriy Miranskyy, Jose Campos

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>)

Tags: [behavior](<https://devfeed.tech/tags/behavior.md>), [cases](<https://devfeed.tech/tags/cases.md>), [circuit](<https://devfeed.tech/tags/circuit.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [effect](<https://devfeed.tech/tags/effect.md>), [effects](<https://devfeed.tech/tags/effects.md>), [examples](<https://devfeed.tech/tags/examples.md>), [execution](<https://devfeed.tech/tags/execution.md>), [expectation](<https://devfeed.tech/tags/expectation.md>), [framework](<https://devfeed.tech/tags/framework.md>), [gate](<https://devfeed.tech/tags/gate.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [guide](<https://devfeed.tech/tags/guide.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [mutation-testing](<https://devfeed.tech/tags/mutation-testing.md>), [planning](<https://devfeed.tech/tags/planning.md>), [quant-ph](<https://devfeed.tech/tags/quant-ph.md>), [quantum](<https://devfeed.tech/tags/quantum.md>), [testing](<https://devfeed.tech/tags/testing.md>), [visibility](<https://devfeed.tech/tags/visibility.md>)

### AI overview

The framework separates surviving quantum-program mutants into behaviorally unchanged cases, changes hidden by the selected input or measurement, and cases unresolved by insufficient statistical evidence. For ideal, noiseless programs with mutations replacing one gate or subcircuit, it traces effects on measured outputs and combines mathematical checks with statistical estimates. It gives conditions proving when a test cannot reveal a mutation, exact effect formulas when required state information is available, and--under stated assumptions--a mutation-score bound that retains surviving and unresolved mutants. A restricted circuit family permits efficient classical classification. The work also provides a decision procedure, worked test-change examples and implementation requirements, plus a small-circuit reference implementation for differential expectation tests. Evidence is limited to proofs and executable examples under the ideal model.

### Source excerpt

arXiv:2610.36976v1 Announce Type: new Abstract: Mutation testing assesses test suites by introducing program changes and checking whether tests detect them. In quantum programs, a surviving mutant may leave program behavior unchanged, produce a change hidden by the chosen input or measurement, or remain undetected because statistical evidence is insufficient. We develop a circuit-aware framework to distinguish these causes and guide execution planning and test changes. The framework covers ideal, noiseless programs and mutations that replace one gate or subcircuit. The method traces how a mutation can affect measured outputs and combines established mathematical checks with statistical estimates. For supported cases, we establish conditions under which a test cannot reveal a mutation and exact formulas for its effect when the required state information is available. Under stated assumptions, we bound the mutation score while retaining surviving and unresolved mutants, and identify a restricted circuit family whose mutation effects admit efficient classical classification. The deliverables include a decision procedure, mathematical guarantees, worked examples of test changes and their implementation requirements, and a small-circuit reference implementation for differential expectation tests. These results help testers decide what to execute, what to change, and what remains unknown. The evidence is limited to proofs and executable examples under the ideal model.

[Next page](<https://devfeed.tech/topics/mutation-testing.md?cursor=WyIyMDI2LTEwLTI4VDA0OjAwOjAwKzAwOjAwIiwgIjBiNDdhMzU4LTNiZGQtNGVhNS1iYjJiLWQ4YTRkNTJlOTA4MiJd>)