# DramaGrow: A Regime-Aware Benchmark for Auditing LLM Judges of Multi-Episode Drama

DevFeed: [DramaGrow: A Regime-Aware Benchmark for Auditing LLM Judges of Multi-Episode Drama](<https://devfeed.tech/articles/dramagrow-a-regime-aware-benchmark-for-auditing-llm-judges-of-multi-episode-drama-125847.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09938>)

Author: Hui Zhang, Yibo Su

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [audit](<https://devfeed.tech/topics/audit.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Dataset development](<https://devfeed.tech/topics/dataset-development.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [human feedback](<https://devfeed.tech/topics/human-feedback.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [auditing](<https://devfeed.tech/tags/auditing.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [constraint](<https://devfeed.tech/tags/constraint.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [data](<https://devfeed.tech/tags/data.md>), [drama](<https://devfeed.tech/tags/drama.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [framework](<https://devfeed.tech/tags/framework.md>), [glm](<https://devfeed.tech/tags/glm.md>), [llm](<https://devfeed.tech/tags/llm.md>), [margin](<https://devfeed.tech/tags/margin.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [reliability](<https://devfeed.tech/tags/reliability.md>)

## AI overview

DramaGrow audits LLM judges of AI-generated Chinese five-episode dramas by separating verifiable constraint violations, clear human preferences, and genuine panel disagreement into distinct evaluation regimes. Its benchmark includes 800 episodes, 120 byte-preserving interventions, and 1,155 blinded five-rater records. On frozen sets of 114 preference pairs and 35 constraint interventions, repeated absolute-score rubrics achieved 53.8-55.8% mean accuracy and only 29.8-32.5% agreement across three calls; widening the tie band reduced accuracy from 71.6-71.9% to 34.8-37.1%. DirectPair-GLM scored 85.1% on preference accuracy, 89.5% on constraint F1, and 81.9% on Detect+Locate on the same items. Exact subpanel enumeration found four of five votes to be the smallest stable human margin. The results indicate that tie calibration, repeated-call variation, and the human vote threshold materially affect reported performance alongside rubric design.

## Source excerpt

arXiv:2611.09938v1 Announce Type: cross Abstract: LLMs can now draft complete multi-episode dramas, and LLM judges are increasingly used to evaluate them. Assessing the reliability of these judges is difficult because drama quality encompasses specification-backed errors, reproducible preferences, and legitimate disagreement among human raters. Most existing evaluations reduce these cases to a single score from one judge call and leave tie calibration, call-to-call variation, and the vote margin underlying the human reference unreported. We introduce DramaGrow, a regime-aware benchmark and audit protocol for AI-generated Chinese five-episode drama. The benchmark assigns verifiable constraint violations, four-of-five preferences, and within-panel disagreements to separate denominators and evidence rules. It contains 800 episodes, 120 byte-preserving interventions, and 1,155 blinded five-rater records, including a frozen extension with 114 supermajority pairs and 21 disagreement pairs. On the frozen sets of 114 preference pairs and 35 constraint interventions, the repeated absolute-score rubrics yield overlapping performance, with mean accuracies of 53.8-55.8% and three-call agreement of 29.8-32.5%, and their accuracy falls from 71.6-71.9% to 34.8-37.1% as the tie band widens. DirectPair-GLM reaches 85.1% preference accuracy, 89.5% constraint F1, and 81.9% Detect+Locate on the same items, and exact subpanel enumeration identifies four-of-five as the smallest stable human vote margin. Evaluation protocol, tie calibration, judge-call variation, and panel margin therefore affect performance as much as rubric design. We will release the specification packages, scripts, interventions, anonymized human votes and rationales, frozen judge runs, and analysis code.