# self-testing

Published articles for self-testing.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

DevFeed: [VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents](<https://devfeed.tech/articles/verifierbench-benchmarking-the-generation-of-executable-verifiers-for-llm-agents-94611.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.28679>)

Author: Jianing Yin, Ziyi Chen, Hao Wang, Yiyou Sun, Zijian Jin, Dawn Song, Daoan Zhang

Published: 2026-10-22T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [harvestbench](<https://devfeed.tech/topics/harvestbench.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-harness](<https://devfeed.tech/tags/agent-harness.md>), [agents](<https://devfeed.tech/tags/agents.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [audit](<https://devfeed.tech/tags/audit.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [case-studies](<https://devfeed.tech/tags/case-studies.md>), [code](<https://devfeed.tech/tags/code.md>), [code-verification](<https://devfeed.tech/tags/code-verification.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [environment](<https://devfeed.tech/tags/environment.md>), [errors](<https://devfeed.tech/tags/errors.md>), [generation](<https://devfeed.tech/tags/generation.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [models](<https://devfeed.tech/tags/models.md>), [self-testing](<https://devfeed.tech/tags/self-testing.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

VerifierBench evaluates executable verifiers generated by LLM agents across 222 tasks from six agent benchmarks, using 4,993 hidden tests that include alternative valid solutions and attacks. Across 19 models, normalized balanced accuracy ranges from 10.6 to 62.1; verifiers reject invalid solutions more reliably than they accept valid ones. The authors report that many false rejections arise because a verifier accepts none of the tested solutions, often due to unsupported assumptions about solution formats, tool use, or code changes. Self-testing may miss these errors when test solutions share the verifier's assumptions or agents never confirm that a correct solution passes. The findings recommend checking both acceptance of a correct solution and acceptance of other solutions the task permits.

### Source excerpt

arXiv:2610.28679v1 Announce Type: new Abstract: Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution and reject invalid ones. We introduce VerifierBench, a benchmark of 222 tasks from six agent benchmarks spanning tool-call, state-based, and code verification, with 4,993 hidden test cases that include alternative valid solutions and attacks; Verifier4Verifier (V4V), a six-check audit, screens its source verifiers and test suites. Across 19 models in a common agent harness, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers consistently rejecting invalid solutions more reliably than they accept valid ones. This apparent strength can mask overly restrictive verification: most false rejections in the analyzed records come from verifiers that accept no tested solution to a task. Case studies reveal unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions. These errors survive self-testing because agents test solutions that share the verifier's mistaken assumptions, or submit verifiers without confirming that any correct solution passes. These findings suggest testing both whether a verifier accepts a correct solution and whether it accepts other solutions permitted by the task. The project page and dataset are publicly available.

## How to Test a Compiler Against Itself: Thirteen Generators and 7,800 Programs

DevFeed: [How to Test a Compiler Against Itself: Thirteen Generators and 7,800 Programs](<https://devfeed.tech/articles/how-to-test-a-compiler-against-itself-thirteen-generators-and-7-800-programs-93928.md>)

Original publisher: [Read original article](<https://hackernoon.com/how-to-test-a-compiler-against-itself-thirteen-generators-and-7800-programs?source=rss>)

Author: Tech\_World

Published: 2026-10-21T20:01:48Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Software Testing](<https://devfeed.tech/topics/software-testing.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [Fuzzing/Fuzz testing](<https://devfeed.tech/topics/fuzzing.md>), [Compilers and Linkers](<https://devfeed.tech/topics/compilers-and-linkers.md>)

Tags: [back-end](<https://devfeed.tech/tags/back-end.md>), [backends](<https://devfeed.tech/tags/backends.md>), [bytecode](<https://devfeed.tech/tags/bytecode.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [chuks-programming-language](<https://devfeed.tech/tags/chuks-programming-language.md>), [chukslang](<https://devfeed.tech/tags/chukslang.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [errors](<https://devfeed.tech/tags/errors.md>), [free-software](<https://devfeed.tech/tags/free-software.md>), [generators](<https://devfeed.tech/tags/generators.md>), [http](<https://devfeed.tech/tags/http.md>), [native](<https://devfeed.tech/tags/native.md>), [nightly](<https://devfeed.tech/tags/nightly.md>), [programming](<https://devfeed.tech/tags/programming.md>), [release-notes](<https://devfeed.tech/tags/release-notes.md>), [self-testing](<https://devfeed.tech/tags/self-testing.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [test](<https://devfeed.tech/tags/test.md>), [tests](<https://devfeed.tech/tags/tests.md>)

### AI overview

Comparing Chuks's bytecode VM with its native compiler can miss bugs shared by both, so v0.2.0-rc.3 adds tests with answers independent of that comparison: 450 translated compiler regression programs, about 7,800 programs from 13 feature-combination generators, and native builds of Valet and CodeStudio. The tests exposed issues including cancelled tasks continuing to run, locks remaining held after errors, different `a || b` results across backends, and an HTTP parser accepting forbidden framing; each was fixed and made a permanent check. The generators pass on the full tree, and a higher-strength nightly run has caught a regression. Based on this evidence, the project says rc.3 will be its last candidate and plans to publish v0.2.0 from this build, while acknowledging that tests cannot prove bugs are absent.

### Source excerpt

Comparing two backends only catches disagreement. How Chuks tests against known answers with 13 generators and about 7,800 programs.