# JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment

DevFeed: [JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment](<https://devfeed.tech/articles/judgmentbench-comparing-rubric-and-preference-evaluation-for-quality-assessment-120038.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.25240>)

Author: Russell Yang, Ruishi Chen, Pierce Kelaita, Riya Ranjan, Sibo Ma, Charles Dickens, Matthew Guillod, Megan Ma, Julian Nyarko

Published: 2026-11-06T05:00:00Z

Content type: comparison

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [Dataset development](<https://devfeed.tech/topics/dataset-development.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>)

Tags: [assessment](<https://devfeed.tech/tags/assessment.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [domain](<https://devfeed.tech/tags/domain.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [experts](<https://devfeed.tech/tags/experts.md>), [law](<https://devfeed.tech/tags/law.md>), [legal](<https://devfeed.tech/tags/legal.md>), [llm](<https://devfeed.tech/tags/llm.md>), [metric](<https://devfeed.tech/tags/metric.md>), [ordering](<https://devfeed.tech/tags/ordering.md>), [quality](<https://devfeed.tech/tags/quality.md>), [rank](<https://devfeed.tech/tags/rank.md>), [research](<https://devfeed.tech/tags/research.md>), [supervision](<https://devfeed.tech/tags/supervision.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [time](<https://devfeed.tech/tags/time.md>), [used](<https://devfeed.tech/tags/used.md>)

## AI overview

JudgmentBench compares rubric scoring with pairwise preference judgments using annotations from practicing attorneys on 30 real-world legal tasks. In this initial study, outputs were constructed at three quality levels. Pairwise judgments recovered their intended quality ordering better than rubric scores: mean Spearman correlation was 0.908 versus 0.150, and pairwise win rate was 0.669 versus 0.542. They also took less than half the annotation time. The reported pattern held for human annotators and LLM autograders. These findings describe the benchmark's initial comparison; the paired dataset is also intended to support research on eliciting and using expert supervision where ground truth is not verifiable.

## Source excerpt

arXiv:2605.25240v4 Announce Type: replace-cross Abstract: Two methodologies dominate current practices of benchmarking: rubric-based scoring evaluates items against predefined criteria, whereas comparative judgment elicits pairwise preferences between outputs. Although both methodologies are widely used, the choice between them is rarely justified. We release JudgmentBench, a benchmark of 30 real-world legal tasks, paired with 1,539 rubric scores and 1,530 pairwise preference judgments collected from practicing attorneys--including at major U.S. law firms--with substantial experience. The annotations constitute the first publicly available dataset in a high-expertise domain in which both supervision signals are elicited from the same pool of experts on the same tasks. Using LLM-generated outputs at three constructed quality levels, we provide an initial empirical comparison: comparative judgments recover the intended quality ordering substantially better than rubrics under both a per-task rank-correlation metric (mean Spearman's rank correlation of 0.908 vs. 0.150, estimated difference = 0.758 [0.494, 1.021]) and a per-judgment pairwise win-rate metric (0.669 vs. 0.542, estimated difference = 0.127 [0.067, 0.186]), while requiring less than half the annotation time. The patterns hold for human annotators and LLM autograders. Beyond this initial comparison, the paired structure of the dataset supports a broader research agenda on how expert judgment should be elicited, aggregated, and used as supervision in domains without verifiable ground truth.