# RubricCraft: Where Solutions Diverge, Criteria Emerge

DevFeed: [RubricCraft: Where Solutions Diverge, Criteria Emerge](<https://devfeed.tech/articles/rubriccraft-where-solutions-diverge-criteria-emerge-99887.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.31707>)

Author: Hosein Hasanbeig, Priyanshu Gupta, Ashish Tiwari, Vu Le, Sumit Gulwani

Published: 2026-10-26T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [audit](<https://devfeed.tech/topics/audit.md>), [Parallel reasoning](<https://devfeed.tech/topics/parallel-reasoning.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [audit](<https://devfeed.tech/tags/audit.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [improvements](<https://devfeed.tech/tags/improvements.md>), [language](<https://devfeed.tech/tags/language.md>), [model](<https://devfeed.tech/tags/model.md>), [reference](<https://devfeed.tech/tags/reference.md>), [repair](<https://devfeed.tech/tags/repair.md>), [requirements](<https://devfeed.tech/tags/requirements.md>), [solution](<https://devfeed.tech/tags/solution.md>), [specifications](<https://devfeed.tech/tags/specifications.md>)

## AI overview

RubricCraft builds natural-language evaluation rubrics by comparing independently sampled solutions, then revises criteria that miss errors or reject valid alternatives. This is intended to capture partial correctness without treating one solution as a complete reference. On SpreadsheetBench v2, compared with a prompt-only rubric written and applied by the same model, RubricCraft passed fewer incorrect outputs (29.5% versus 42.6%), rejected none accepted by the benchmark checker, and ranked outputs by correctness more often, while separating models as often. Repair based on one attempt did not change grading; attempts from three model families prompted repairs on different tasks, and an evidence audit judged most repairs improvements, including many on tasks where the reference solution was wrong.

## Source excerpt

arXiv:2610.31707v1 Announce Type: cross Abstract: Evaluating language agents on open-ended, artifact-producing tasks is difficult: model-based graders can be inconsistent, while exhaustive programmatic checks are often impractical. Textual rubrics make evaluation criteria explicit, but writing them requires domain expertise. Rubrics generated from task descriptions alone can miss requirements and failure modes that become apparent in attempted solutions. We introduce RubricCraft, which treats rubric construction as the generation of natural-language specifications guided by candidate solutions. Given a task description and independently sampled solver trajectories, RubricCraft compares how each attempt satisfies or violates the task requirements and uses these differences to construct rubric criteria. It applies the criteria to the collected solutions and revises checks that overlook errors or penalize valid alternatives. The resulting rubrics capture partial correctness without requiring any single solution to serve as a complete reference. On SpreadsheetBench v2, against a prompt-only rubric written and applied by the same model, RubricCraft rubrics pass fewer wrong outputs (29.5% against 42.6%), fail none of the outputs the benchmark's own checker accepts, and rank outputs by correctness more often, while separating models as often. Repair on a single attempt leaves grading unchanged, but repair driven by attempts from three model families acts on different tasks, and an evidence audit judges most of those repairs improvements, many on tasks whose reference solution is itself wrong.