# Widening Harm Evaluation Coverage with a Judge that Reads the World

DevFeed: [Widening Harm Evaluation Coverage with a Judge that Reads the World](<https://devfeed.tech/articles/widening-harm-evaluation-coverage-with-a-judge-that-reads-the-world-115337.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.02624>)

Author: Prabhpreet Singh Setia, Bhavesh Neekhra, Debayan Gupta

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [least privilege](<https://devfeed.tech/topics/least-privilege.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [llm](<https://devfeed.tech/tags/llm.md>), [osworld](<https://devfeed.tech/tags/osworld.md>), [robust](<https://devfeed.tech/tags/robust.md>), [safety](<https://devfeed.tech/tags/safety.md>), [state](<https://devfeed.tech/tags/state.md>), [taxonomy](<https://devfeed.tech/tags/taxonomy.md>), [union](<https://devfeed.tech/tags/union.md>)

## AI overview

The study evaluates an LLM harm judge with read-only access to environment state alongside a judge that inspects an agent's trajectory. Across two DecodingTrust-Arena environments, combining the two channel-specific verdicts recalled 94% of confirmed harms, versus 53% for trajectory-only judging; on OS-Harm, the union recalled 77%, versus 67%. A single judge observing both channels did not improve performance and produced more false negatives than the state-only judge. The channels also missed different episodes. The abstract's results suggest that combining separate trajectory and state-only judges can broaden harm detection in these tested settings.

## Source excerpt

arXiv:2611.02624v1 Announce Type: new Abstract: An agent acting in a stateful environment can leave evidence of harm in two places: the record of what it did and said, or the world it changed. Safety evaluations for discovering harm examine one or the other in limited ways, using either an LLM judge over the trajectory or deterministic checks over the final state. Although LLM-based verifiers that inspect the environment exist, they have been used to verify task completion, not to detect and confirm harm in a safety setting. We address this gap with an LLM that has read-only access to the environment state and is given a harm taxonomy. We evaluate this approach in three environments: two from DecodingTrust-Arena and one from OSWorld. We hold the judge model and harm taxonomy fixed, varying only what the judge observes: the trajectory, read-only tools for inspecting the environment, or both channels together. In the two DecodingTrust-Arena environments, combining the trajectory and state-only judges recalls 94 percent of confirmed harms, compared with 53 percent for the trajectory judge alone. On OS-Harm, the union recalls 77 percent, compared with 67 percent. A single judge given both channels does not perform better and produces more false negatives than the state-only judge. We also record disagreements and find that the two channels miss different episodes. These results suggest that combining verdicts from two channel-specific judges is a robust and flexible approach to detecting and verifying harm in agentic settings.