# Learning from Mutants: Teaching Small Language Models to Synthesise Loop Invariants

DevFeed: [Learning from Mutants: Teaching Small Language Models to Synthesise Loop Invariants](<https://devfeed.tech/articles/learning-from-mutants-teaching-small-language-models-to-synthesise-loop-invariants-137467.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19569>)

Author: Daragh King, Vasileios Koutavas

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Model Development](<https://devfeed.tech/topics/model-development.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [framework](<https://devfeed.tech/tags/framework.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [invariants](<https://devfeed.tech/tags/invariants.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [mutation](<https://devfeed.tech/tags/mutation.md>), [mutation-testing](<https://devfeed.tech/tags/mutation-testing.md>), [operators](<https://devfeed.tech/tags/operators.md>), [program](<https://devfeed.tech/tags/program.md>), [qwen3-5-4b](<https://devfeed.tech/tags/qwen3-5-4b.md>), [small-language-models](<https://devfeed.tech/tags/small-language-models.md>), [synthesis](<https://devfeed.tech/tags/synthesis.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [training-data](<https://devfeed.tech/tags/training-data.md>), [verification](<https://devfeed.tech/tags/verification.md>)

## AI overview

The method turns verifier-breaking mutations of correct Dafny loop invariants into fine-tuning examples: each mutant is paired with verifier feedback and the correct invariant, teaching models to refine candidates. On 472 DafnyBench programs, the best fine-tuned Gemma-4-E4B configuration verified 50.8% zero-shot, versus 8.1% for its base model; Qwen3.5-4B rose from 9.3% to 53.6%. Higher-order mutants were more effective training examples than first-order mutants, and combining mutation-based examples with examples where invariants were removed improved performance during iterative refinement. The evidence is limited to the reported Dafny evaluation and transfer result.

## Source excerpt

arXiv:2611.19569v1 Announce Type: cross Abstract: Large language models (LLMs) have proven promising for synthesising loop invariants, a long-standing obstacle to automated program verification. Yet, frontier LLMs are typically accessible only through third-party APIs, hampering reproducibility and local deployment. Small language models (SLMs) sidestep these obstacles but require task-specific fine-tuning, for which high-quality training data is needed. We present a failure-driven data curation method for fine-tuning language models to synthesise loop invariants. Inspired by mutation testing, our approach applies semantic mutation operators to ground-truth invariants, producing semantic mutants that induce verification failure. Each mutant is paired with verifier feedback and its ground-truth counterpart, yielding training examples for feedback-guided invariant refinement. Whereas the vast space of mutants is a long-standing burden in mutation testing, it becomes an asset when generating training data, multiplying the examples obtainable from a fixed set of verified programs. We implement our approach for Dafny, augmenting an existing synthetic dataset of verified programs, and introduce a framework that iteratively refines candidate invariants using verifier feedback. On 472 DafnyBench programs, our best-performing fine-tuned Gemma-4-E4B configuration verifies 50.8% of programs zero-shot, compared to 8.1% for the base model, a 6.3$\times$ improvement. Higher-order mutants prove more effective as training data than first-order mutants, and adding mutation-based examples to direct synthesis examples, in which invariants are removed rather than mutated, improves performance under iterative refinement. Our approach also transfers to Qwen3.5-4B, whose zero-shot verification rate rises from 9.3% to 53.6%.