# When LLM judges agree, should we believe them?

DevFeed: [When LLM judges agree, should we believe them?](<https://devfeed.tech/articles/when-llm-judges-agree-should-we-believe-them-7609.md>)

Original publisher: [Read original article](<https://www.amazon.science/blog/when-llm-judges-agree-should-we-believe-them>)

Author: Krishna Balasubramanian; Sasha Podkopaev

Published: 2026-08-26T17:10:40Z

Content type: article

Language: en

Sources: [Amazon Science homepage](<https://devfeed.tech/sources/amazon-science-homepage.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Retrieval-Augmented Generation](<https://devfeed.tech/topics/retrieval-augmented-generation.md>), [Ising](<https://devfeed.tech/topics/ising.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [Network](<https://devfeed.tech/topics/network.md>), [AI, ML & Data Engineering](<https://devfeed.tech/topics/ai-ml-data-engineering.md>)

Tags: [conversational-ai](<https://devfeed.tech/tags/conversational-ai.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [ising](<https://devfeed.tech/tags/ising.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llm](<https://devfeed.tech/tags/llm.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [retrieval-augmented-generation](<https://devfeed.tech/tags/retrieval-augmented-generation.md>)

## AI overview

The article examines whether agreement among LLM judges is trustworthy when their outputs may be correlated. It presents a dependence-aware aggregation method based on Ising models that discounts shared blind spots and outperforms historical-accuracy-weighted majority voting on three tasks.

## Source excerpt

Discounting the opinions of LLM judges with highly correlated outputs ensures that panels of judges reflect a true diversity of perspectives.