# Beyond QASecClaw: Agent Scaling, Verifier Feedback, and Evidence Reliability in Software Assurance

DevFeed: [Beyond QASecClaw: Agent Scaling, Verifier Feedback, and Evidence Reliability in Software Assurance](<https://devfeed.tech/articles/beyond-qasecclaw-agent-scaling-verifier-feedback-and-evidence-reliability-in-software-assurance-103207.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.36947>)

Author: S M Abdulla Al Mamun, Akif Islam, Mohd. Ruhul Ameen

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [Code quality](<https://devfeed.tech/topics/code-quality.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [generation](<https://devfeed.tech/tags/generation.md>), [java](<https://devfeed.tech/tags/java.md>), [jvm](<https://devfeed.tech/tags/jvm.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [limits](<https://devfeed.tech/tags/limits.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [owasp](<https://devfeed.tech/tags/owasp.md>), [owasp-benchmark](<https://devfeed.tech/tags/owasp-benchmark.md>), [packages](<https://devfeed.tech/tags/packages.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [software](<https://devfeed.tech/tags/software.md>), [test](<https://devfeed.tech/tags/test.md>), [token](<https://devfeed.tech/tags/token.md>), [validation](<https://devfeed.tech/tags/validation.md>)

## AI overview

Across five software assurance experiments, adding specialist model contexts did not improve unsafe-warning clearance: on eight warnings, 50 contexts used 49 times as many tokens as one. On 160 OWASP cases, configurations traded clearance against unresolved results, and the ten-context arm assessed only 96 cases. For 32 SV-COMP tasks, raw verifier feedback increased mean correct model verdicts from 12.7 to 22.0, but incorrect verdicts also rose from one to seven; a JBMC-plus-JVM baseline produced 26 correct and six unresolved verdicts. Models rejected 70-100% of applicable counterexamples in two 120-package sets, while construction-specific scope checks classified all packages correctly. No target defect was found in three Defects4J bugs, including one unsupported by the test adapter. The descriptive results favor deterministic evidence admission in these tested settings, while highlighting limits in verdict aggregation, feedback interpretation, and evaluation coverage.

## Source excerpt

arXiv:2610.36947v1 Announce Type: cross Abstract: Can additional specialist language-model contexts improve software assurance when deterministic tool evidence is already available? We study this question through five experiments spanning authored Java warning controls, OWASP Benchmark assessment, Defects4J test generation, SV-COMP assertion verdicts, and deliberately misleading evidence packages. The evaluated systems use Qwen2.5-Coder-7B-Instruct, with a 1.5B sensitivity analysis, and complete 13,638 model requests. On eight related development warnings, increasing the context count from 1 to 50 increases token use 49-fold without improving unsafe clearance. On 160 OWASP cases, model configurations trade clearance against unresolved outcomes; the ten-context arm assesses only 96 cases. On 32 SV-COMP tasks, raw verifier feedback raises mean correct model verdicts from 12.7 to 22.0, but also raises incorrect verdicts from one to seven; a JBMC-plus-JVM validation baseline returns 26 correct verdicts and six unresolved outcomes. On two sets of 120 evidence packages, models reject 70-100% of applicable counterexamples, while construction-specific scope checks classify every package correctly. No target defect is discovered in three Defects4J bugs, one unsupported by the test adapter. These descriptive results support deterministic evidence admission for the tested settings, while exposing limits of verdict aggregation, feedback interpretation, and the available evaluation coverage.