# Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge

DevFeed: [Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge](<https://devfeed.tech/articles/grounded-but-not-immune-confirmation-bias-in-agent-as-a-judge-118360.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05662>)

Author: Drake Benjamin Brown, Khushbu Pahwa, Yubin Ge, Salvatore Romeo, Akhil Agnihotri, Yi Zhang, Xiaofei Ma, Daniele Bonadiman

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [agent observability](<https://devfeed.tech/topics/agent-observability.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-tasks](<https://devfeed.tech/tags/agent-tasks.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [bias](<https://devfeed.tech/tags/bias.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [environment](<https://devfeed.tech/tags/environment.md>), [error](<https://devfeed.tech/tags/error.md>), [failure](<https://devfeed.tech/tags/failure.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [investigation](<https://devfeed.tech/tags/investigation.md>), [note](<https://devfeed.tech/tags/note.md>), [open](<https://devfeed.tech/tags/open.md>), [over](<https://devfeed.tech/tags/over.md>), [properties](<https://devfeed.tech/tags/properties.md>), [shift](<https://devfeed.tech/tags/shift.md>), [signal](<https://devfeed.tech/tags/signal.md>), [support](<https://devfeed.tech/tags/support.md>), [tool](<https://devfeed.tech/tags/tool.md>), [tools](<https://devfeed.tech/tags/tools.md>), [trust](<https://devfeed.tech/tags/trust.md>), [under](<https://devfeed.tech/tags/under.md>)

## AI overview

Tool use does not shield Agent-as-a-Judge from confirmation bias: a false, authority-attributed pre-screening note overturned up to 40% of correct verdicts across four judges and five datasets, even when judges collected decisive evidence. The study identifies three ways the cue changes behavior: over-trust (verify, reach the correct conclusion, then defer), hallucination (invent support for a changed verdict), and under-investigation (stop after initial corroboration). Over-trust accounted for 67-75% of harmful flips for the two Claude judges; smaller judges showed all three failure modes. Lightweight mitigations reduced but did not eliminate the bias. The authors frame the cue as a controlled diagnostic, not an adversarial attack.

## Source excerpt

arXiv:2611.05662v1 Announce Type: new Abstract: Evaluating open-ended agent tasks increasingly falls to an Agent-as-a-Judge: an evaluator handed tools and the agent's full trajectory so it can verify a result against the environment. The premise is that a judge that gathers its own evidence should be grounded by it. We show that this grounding does not immunize the judge against confirmation bias: a single false pre-screening note, attributed to an authority and carrying no evidential bearing on the correct verdict, overturns up to 40% of a tool-using judge's correct verdicts across four judges and five datasets, even though the judge collects decisive evidence. The shift is driven by specific properties of the cue. We treat this cue as a controlled diagnostic of confirmation bias rather than an adversarial attack. Our central contribution is to explain how confirmation bias changes the judge's behavior in a trajectory. We classify changes in judge behavior and attribute changes to one of three failure modes: over-trust (verify, conclude correctly, then defer), hallucination (fabricate support for the flip), and under-investigation (stop at the first corroborating signal). For the two Claude judges, the failure is predominantly over-trust (67-75% of harmful flips), while smaller judges have error modes spread across all three. Lightweight mitigations blunt but never remove the bias.