# Batch-Evaluating LLM Agent Trajectories for Responsible AI Checks on Cloud TPU v5e

DevFeed: [Batch-Evaluating LLM Agent Trajectories for Responsible AI Checks on Cloud TPU v5e](<https://devfeed.tech/articles/the-score-was-right-the-agent-was-wrong-22858.md>)

Original publisher: [Read original article](<https://medium.com/google-developer-experts/the-score-was-right-the-agent-was-wrong-59efb6a1f1fe?source=rss----a67bd6fa7d58---4>)

Author: Noble Ackerson

Published: 2026-08-04T23:28:06Z

Content type: tutorial

Language: en

Sources: [Google Developer Experts - Medium](<https://devfeed.tech/sources/google-developer-experts-medium.md>)

Topics: [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Claude](<https://devfeed.tech/topics/claude.md>), [gemma](<https://devfeed.tech/topics/gemma.md>), [Security](<https://devfeed.tech/topics/security.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [incident](<https://devfeed.tech/topics/incident.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [claude](<https://devfeed.tech/tags/claude.md>), [cybersecurity](<https://devfeed.tech/tags/cybersecurity.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [google-cloud-platform](<https://devfeed.tech/tags/google-cloud-platform.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [incident](<https://devfeed.tech/tags/incident.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [responsible-ai](<https://devfeed.tech/tags/responsible-ai.md>), [security](<https://devfeed.tech/tags/security.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

## AI overview

This article describes batch-evaluating LLM agent trajectories for responsible-AI checks before incidents occur. It connects reported production-system breaches with Hugging Face's use of LLM-driven analysis over more than 17,000 attacker events, then presents a scheduled approach using Gemma through vLLM on Cloud TPU v5e.

## Source excerpt

Batch-evaluating agent trajectories on Cloud TPU v5e (compliance-at-scale, part 2) Trajectory batch eval pipeline for rai-checklist-cli A week or so ago, Hugging Face disclosed that an autonomous agent had broken into its production infrastructure. Five days later, OpenAI confirmed the agent was theirs: a combination of its own models, running an internal cyber-capability eval with the production safety classifiers switched off. The models were being tested on a benchmark called ExploitGym. The fastest observable path to a solution ran through the answer key. They escaped the isolated environment through a package-registry proxy, chained stolen credentials with zero-day vulnerabilities, and pulled the test solutions out of Hugging Face's production database. Per Axios, the agent kept pursuing its assigned objective even after it had escaped the test environment. Nine days later, Anthropic said hold my beer, checked its own logs and found three more. It reviewed 141,006 runs and found three cases where Claude models had reached the open internet and breached real production systems, the earliest dating to April. Two of the three organizations learned about it when Anthropic notified them. One lab looked and found something. A second lab looked and found something. That is the whole story here, and it should be the uncomfortable part: none of this surfaced through production monitoring. It surfaced because somebody went back and read the trajectories. Nobody has published what score that run produced. It doesn't matter. The part of this story that matters for this series is what Hugging Face did next with their findings. To reconstruct the intrusion, Hugging Face's security team ran LLM-driven analysis agents over the full attacker action log: more than 17,000 recorded events. Reporting indicates they did that analysis with an open-weight model on their own infrastructure, partly so no hosted safety classifier sat between the responders and the attack data, and partly