# Design AI Products for Verification Before Building Evals

DevFeed: [Design AI Products for Verification Before Building Evals](<https://devfeed.tech/articles/it-s-hard-to-eval-is-a-product-smell-18787.md>)

Original publisher: [Read original article](<https://hamel.dev/blog/posts/eval-smell/>)

Author: Hamel Husain

Published: 2026-06-29T07:00:00Z

Content type: opinion

Language: en

Sources: [Hamel Husain](<https://devfeed.tech/sources/hamel-husain.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [data](<https://devfeed.tech/topics/data.md>), [dashboards](<https://devfeed.tech/topics/dashboards.md>), [SQL](<https://devfeed.tech/topics/sql.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-evals](<https://devfeed.tech/tags/ai-evals.md>), [data-agents](<https://devfeed.tech/tags/data-agents.md>), [evals](<https://devfeed.tech/tags/evals.md>), [interface](<https://devfeed.tech/tags/interface.md>), [llms](<https://devfeed.tech/tags/llms.md>), [techniques](<https://devfeed.tech/tags/techniques.md>), [verification](<https://devfeed.tech/tags/verification.md>)

## AI overview

The article argues that products described as difficult to evaluate often make their outputs difficult for users to verify. Using AI data agents as an example, it recommends providing checkable artifacts--such as source comparisons, precise metric definitions, breakdowns, SQL, and uncertainty notes--before focusing on eval design.

## Source excerpt

For the past 3 years, AI evals have been my professional focus.1 The most common objection I hear to evals is "our product is hard to eval". This objection is a product smell. Artifacts that are hard for you to verify are often hard for users too. In the worst case, users have to redo the work from scratch to verify the output. More importantly, designing your product for ease of verification should come before building evals. In this post, I'll walk through three products I advised on that faced this issue. I'll also show before and after sketches to demonstrate design principles. After these examples, I'll discuss how to apply this general pattern to your product. Example 1: the AI data agent Almost every company I've worked with builds an internal AI data agent. You ask it a business question, like what was net revenue for Product A last quarter, and it finds relevant data sources, runs the queries, and provides an answer. The goal of this agent is to reduce dependency on data analysts. A common mistake when building AI data agents is to make the answer the only output, as illustrated below. Data Agent What was net revenue for Product A last quarter? Net revenue for Product A last quarter was $4.21M. Ask anything about your business...➤ Since the only output is the answer, there is nothing here to check. In the sketch above, the user has no way to verify the answer beyond redoing work.2 A better design is to provide the user with checkable artifacts, informed by how a domain expert might validate the output. Here are techniques I use to validate metrics as a data scientist: Compare the quantity and any intermediate calculations against a trusted source, like a vetted dashboard or report, or a similar analysis a colleague has already vetted.3 Confirm the metric definition precisely. A number like net revenue can include or exclude things like returns and discounts. Sanity-check a related quantity. If I can't verify the number directly, I pull a related number that s