# SeVerus: Context-Aware, Gamified Benchmarking for Rust Verification

DevFeed: [SeVerus: Context-Aware, Gamified Benchmarking for Rust Verification](<https://devfeed.tech/articles/severus-context-aware-gamified-benchmarking-for-rust-verification-120168.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07194>)

Author: \\'Alvaro F. Silva, Martim Afonso, Nuno Saavedra, Emily First, Alexandra Mendes, Jo\\~ao F. Ferreira

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Formal verification](<https://devfeed.tech/topics/formal-verification.md>), [Rust](<https://devfeed.tech/topics/rust.md>), [codex](<https://devfeed.tech/topics/codex.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [binary-search](<https://devfeed.tech/tags/binary-search.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [codex](<https://devfeed.tech/tags/codex.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [projects](<https://devfeed.tech/tags/projects.md>), [proofs](<https://devfeed.tech/tags/proofs.md>), [quest](<https://devfeed.tech/tags/quest.md>), [rust](<https://devfeed.tech/tags/rust.md>), [sol](<https://devfeed.tech/tags/sol.md>), [solve](<https://devfeed.tech/tags/solve.md>), [using](<https://devfeed.tech/tags/using.md>), [verification](<https://devfeed.tech/tags/verification.md>), [verus](<https://devfeed.tech/tags/verus.md>)

## AI overview

SEVERUS builds Verus verification benchmarks by turning each target into a quest and progressively removing its original proof dependencies, from a single missing lemma to no supporting lemmas. An independent checker rejects executable-code changes, weakened specifications, and unsound proof constructs; noisy binary search estimates the highest level an agent can solve reliably. Across six open-source projects, it generated 819 quests and 10,095 levels. Tests with Codex (GPT-6 Sol and GPT-6 Luna) and Claude Code (Sonnet 4.6) found that success fell and cost rose as context was withdrawn. The strongest agent solved the final level in 52 of 60 sampled quests but reliably solved only 35% of their levels, with failures concentrated in longer quests. Successful GPT-6 Sol proofs usually used alternative decompositions rather than recreating withheld lemmas.

## Source excerpt

arXiv:2611.07194v1 Announce Type: cross Abstract: Coding agents achieve high success rates on existing benchmarks for Verus, a verifier for Rust programs. These benchmarks typically provide the auxiliary lemmas needed to complete a proof. In practice, however, such lemmas must be discovered and proved. To capture this challenge, we present SEVERUS, a context-aware technique to generate Verus benchmarks that organizes verification tasks into quests, each divided into levelsin a gamified manner. Using the original proof dependencies in verified projects, SEVERUS constructs one quest per verification target and progressively removes proof context to create individual levels. These levels range from the easiest, requiring completing only a single missing lemma, to the hardest, requiring proving the target without any supporting lemmas. An independent checker rejects changes to executable code, weakened specifications, and unsound proof constructs. A noisy binary search estimates the highest level that an agent can solve reliably. Applied to six open-source Verus projects, SEVERUS generates 819 quests comprising 10,095 levels. Experiments with Codex using GPT-6 Sol and GPT-6 Luna, and with Claude Code using Sonnet 4.6, show that success decreases and cost increases as proof context is withdrawn. The strongest agent solves the final level of 52 of 60 sampled quests, yet can only reliably solve 35% of their levels, with failures concentrated in longer quests. Successful GPT-6 Sol proofs rarely reproduce the withheld lemmas, instead, using alternative proof decompositions. By systematically varying proof context, SEVERUS provides a foundation for constructing verification benchmarks that can continue to challenge increasingly capable coding agents.