# Compare AI agent changes with Online Experiments in Agent Observability

DevFeed: [Compare AI agent changes with Online Experiments in Agent Observability](<https://devfeed.tech/articles/compare-ai-agent-changes-with-online-experiments-in-agent-observability-118927.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/online-experiments-agent-observability/>)

Author: Alex Barksdale; Yakov Shapiro; Demetri Pananos; Miguel Tulla Lizardi

Published: 2026-11-04T00:00:00Z

Content type: article

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog-the-monitor-blog.md>)

Topics: [experiments](<https://devfeed.tech/topics/experiments.md>), [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [observability](<https://devfeed.tech/topics/observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-observability](<https://devfeed.tech/tags/agent-observability.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [can](<https://devfeed.tech/tags/can.md>), [changes](<https://devfeed.tech/tags/changes.md>), [compare](<https://devfeed.tech/tags/compare.md>), [cost](<https://devfeed.tech/tags/cost.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [errors](<https://devfeed.tech/tags/errors.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [latency](<https://devfeed.tech/tags/latency.md>), [learn](<https://devfeed.tech/tags/learn.md>), [observability](<https://devfeed.tech/tags/observability.md>), [online](<https://devfeed.tech/tags/online.md>), [production](<https://devfeed.tech/tags/production.md>), [quality](<https://devfeed.tech/tags/quality.md>), [test](<https://devfeed.tech/tags/test.md>), [testing](<https://devfeed.tech/tags/testing.md>), [traffic](<https://devfeed.tech/tags/traffic.md>), [with](<https://devfeed.tech/tags/with.md>)

## AI overview

Datadog Online Experiments compares AI agent variants on real production traffic, pairing evaluation scores with operational measures such as latency, cost, token use, and errors. Unlike offline experiments built on predefined datasets, online evaluations can surface unanticipated requests and behaviors; random assignment against a control helps compare variants under the same period of usage. Feature Flags can adjust traffic exposure without a new deployment, and traces let teams inspect tool calls and responses behind metric changes. In Datadog's internal comparison, GPT-6 Luna's estimated cost was 44.3% lower than GPT-5.6 Luna, while mean quality scores were close (3.86 vs. 3.84); those averages did not establish equivalent quality. Duration and workflow-error estimates were higher but uncertain, and trace review found more invalid tool parameters in GPT-6 Luna, usually corrected on retry. The team addressed that issue before proceeding and identified failures as candidates for offline regression tests. The article presents online experiments as complementary to offline testing, with results also informing future evaluation datasets.

## Source excerpt

Learn how Datadog Online Experiments, available through Agent Observability, enables you to test AI agent changes against real production traffic so you can compare response quality, cost, latency, and errors.