# Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents

DevFeed: [Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents](<https://devfeed.tech/articles/comparative-analysis-of-agent-evaluation-frameworks-stability-detection-and-discovery-in-enterprise-analytics-agents-128437.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.14493>)

Author: Howard Zhang, Chuck Arvin, Logan Brown, Mihir Bhatia, Neelu Choudhary, Lily Miao

Published: 2026-11-11T05:00:00Z

Content type: comparison

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [Detection engineering](<https://devfeed.tech/topics/detection-engineering.md>), [threat detection](<https://devfeed.tech/topics/threat-detection.md>), [Cryptography](<https://devfeed.tech/topics/cryptography.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [adversarial](<https://devfeed.tech/tags/adversarial.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-evaluation](<https://devfeed.tech/tags/agent-evaluation.md>), [agent-evaluation-frameworks](<https://devfeed.tech/tags/agent-evaluation-frameworks.md>), [agents](<https://devfeed.tech/tags/agents.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [balance](<https://devfeed.tech/tags/balance.md>), [based](<https://devfeed.tech/tags/based.md>), [best](<https://devfeed.tech/tags/best.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [detection](<https://devfeed.tech/tags/detection.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [frameworks](<https://devfeed.tech/tags/frameworks.md>)

## AI overview

In a controlled study of two enterprise analytics agents using frozen execution traces, trace-based LLM judging achieved the strongest balance on 72 known-risk cases (F1=0.90). Text-only LLM judging matched its recall but produced five times more false alarms. Across repeated measurements, 97% of configurations met the stability threshold, with patterns independent of the agent. Functional testing and security red-teaming uncovered 24 novel failures with no overlap, suggesting they find complementary risks. The authors recommend trace-based evaluation for functional assessment and adversarial red-teaming for security validation. These findings come from the study's specific controlled setting.

## Source excerpt

arXiv:2611.14493v1 Announce Type: new Abstract: Deploying LLM-based analytics agents in enterprise settings requires evaluation frameworks that can reliably detect failures across complex, multi-tool workflows. We present a three-phase comparative study of three evaluation frameworks, each representing a distinct evaluation paradigm (trace-based LLM judging, text-only LLM judging with red-teaming, and deterministic heuristics), applied to two analytics agents in a controlled research setting using frozen execution traces. Phase 1 quantifies evaluation harness stability through repeated measurements, finding that 97% of configurations meet the stability threshold and that stability patterns are agent-independent. Phase 2 measures known risk detection on 72 ground-truth test cases, where the trace-based framework achieves the best precision-recall balance (F1=0.90) while text-only evaluation matches on recall but generates 5x more false alarms. Phase 3 evaluates unknown risk discovery, uncovering 24 novel failures with zero overlap between functional testing and security red-teaming. Our results demonstrate that (1) the information available to the evaluation harness determines evaluation quality, (2) functional and security evaluation are complementary, and (3) LLM-based evaluation harnesses surface subtle errors that human reviewers miss. We recommend a two-framework strategy: trace-based evaluation for functional assessment and adversarial red-teaming for security validation.