# Don't Let Claude Code Design Your LLM Judges

DevFeed: [Don't Let Claude Code Design Your LLM Judges](<https://devfeed.tech/articles/don-t-let-claude-code-design-your-llm-judges-127871.md>)

Original publisher: [Read original article](<https://www.decodingai.com/p/write-your-own-llm-judges>)

Author: Louis-François Bouchard

Published: 2026-11-10T12:21:59Z

Content type: tutorial

Language: en

Sources: [Decoding ML](<https://devfeed.tech/sources/decoding-ml.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Reliable AI outputs](<https://devfeed.tech/topics/reliable-ai-outputs.md>), [AI Engineering](<https://devfeed.tech/topics/ai-engineering.md>), [phoenix](<https://devfeed.tech/topics/phoenix.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>), [codex](<https://devfeed.tech/topics/codex.md>)

Tags: [1-4](<https://devfeed.tech/tags/1-4.md>), [2](<https://devfeed.tech/tags/2.md>), [2-2](<https://devfeed.tech/tags/2-2.md>), [3](<https://devfeed.tech/tags/3.md>), [about](<https://devfeed.tech/tags/about.md>), [account](<https://devfeed.tech/tags/account.md>), [added](<https://devfeed.tech/tags/added.md>), [adoption](<https://devfeed.tech/tags/adoption.md>), [after](<https://devfeed.tech/tags/after.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-builder](<https://devfeed.tech/tags/agent-builder.md>), [agents](<https://devfeed.tech/tags/agents.md>), [claude](<https://devfeed.tech/tags/claude.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [code](<https://devfeed.tech/tags/code.md>), [don-t](<https://devfeed.tech/tags/don-t.md>), [let](<https://devfeed.tech/tags/let.md>), [llm](<https://devfeed.tech/tags/llm.md>), [me](<https://devfeed.tech/tags/me.md>), [noise](<https://devfeed.tech/tags/noise.md>)

## AI overview

ToneBench's 28,200 LLM judgments showed that judge prompts and scores need human calibration before they can support a trustworthy ranking. The author recommends measuring both generation variance and judge reliability: ToneBench now scores five drafts per model configuration and tests judges against human ratings. For subjective scoring, its most consistent approach uses detailed, hand-written rubric levels with examples, evidence quotes before scores, and checks for score clustering and whether rubric revisions separate strong from weak drafts. A panel of judges from different model families can dilute individual models' self-preference, though it does not remove bias; judges also differ in score generosity, speed, and reliability. The author's golden set uses human ratings on the same categories as the judges, but calibration results are not yet available. For ongoing runs, pin exact judge model IDs, re-evaluate after judge or rubric changes, audit inputs and outputs, and require pairwise preferences to agree in both presentation orders. ToneBench is a signal for shortlisting models, not ground truth for other use cases.

## Source excerpt

What 28,200 LLM judgments taught me about noise, rubrics, and human calibration