# Agents are users, so your software needs evals

DevFeed: [Agents are users, so your software needs evals](<https://devfeed.tech/articles/agents-are-users-so-your-software-needs-evals-136001.md>)

Original publisher: [Read original article](<https://vercel.com/blog/agent-evals>)

Author: Yury Selivanov

Published: 2026-11-16T05:00:00Z

Content type: tutorial

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [Test automation](<https://devfeed.tech/topics/test-automation.md>), [AI Bots](<https://devfeed.tech/topics/ai-bots.md>), [Integration testing](<https://devfeed.tech/topics/integration-testing.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-software](<https://devfeed.tech/tags/agentic-software.md>), [agentic-testing](<https://devfeed.tech/tags/agentic-testing.md>), [agents](<https://devfeed.tech/tags/agents.md>), [api](<https://devfeed.tech/tags/api.md>), [cli](<https://devfeed.tech/tags/cli.md>), [command](<https://devfeed.tech/tags/command.md>), [cost](<https://devfeed.tech/tags/cost.md>), [decision](<https://devfeed.tech/tags/decision.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [doing](<https://devfeed.tech/tags/doing.md>), [e2e](<https://devfeed.tech/tags/e2e.md>), [eval](<https://devfeed.tech/tags/eval.md>), [evals](<https://devfeed.tech/tags/evals.md>), [fast](<https://devfeed.tech/tags/fast.md>), [filter](<https://devfeed.tech/tags/filter.md>), [observability](<https://devfeed.tech/tags/observability.md>), [software](<https://devfeed.tech/tags/software.md>), [users](<https://devfeed.tech/tags/users.md>)

## AI overview

Agent evals measure how often software agents complete user tasks correctly across repeated trials; a successful API call or one good manual run can hide wrong outcomes. The excerpt recommends choosing the cheapest test tier that catches the target problem, then using end-to-end evals for tasks whose success depends on a sequence of actions. Build a small task set from user-relevant failures, with repeatable environments and verifiers that score each requirement. Track success rate and choose pass@k or pass^k according to whether occasional success or consistent reliability matters; measure steps, time and cost on successful trials. Run variants repeatedly, inspect failed and passing traces to identify root causes and misleading passes, then rerun after a targeted fix. The example uses Harbor with Vercel Sandboxes for concurrent trials, AI Gateway for model calls, and Vercel storage for traces. The excerpt also stresses isolating seed data and verifiers from agent writes, and notes that small trial counts and saturated evals limit what results can establish.

## Source excerpt

Agents are now users of software, doing work on people's behalf. Whether you're building the agent or the software it uses (API, CLI, MCP server, SDK), evals give you a way to measure how well that work gets done. Agents probably missed your launch video and billboard on Highway 101, but they still go through the same funnel a human does to find and use a product: Discovery, Selection, Access, and Success. You can build evals for any stage of this funnel. Here, we'll focus on the final stage, Success: whether the agent completes the user's task correctly. That's more difficult to check than it sounds. An agent can place a "successful" order for a hot latte when the user asked for an iced one. The request worked, so the logs in your Observability stack look fine. And because LLM outputs vary between runs, one good manual test can't tell you whether a fix worked. You need to repeat the same tasks and grade the outcomes, before and after the change. That's what an eval is, a test you run many times and score. Here's how we build one. The agentic testing pyramid If a test asks "did it work?", an eval asks "how often does it work, and how well?" We group agentic software tests by how many LLM calls they typically need: deterministic tests (0), one-turn evals (1), and E2E evals (2+ across a whole task). This has turned out to be a great proxy for the effort to build them, the time they take to run, and the cost of running them. Just like in the practical test pyramid, the cheap and fast tests should make up the bulk of your test suite. To use the pyramid, start with the problem you want to catch and pick the lowest tier that can catch it. For Success, that's the top. A one-turn eval checks a single decision, like which command the agent runs first. But Success depends on the whole sequence of steps (e.g. search for cafes -> filter on user criteria -> place the latte order). Each one can look reasonable on its own, and the agent can still get the wrong outcome. An E2E eval l