# AI Evals: Common Questions About Model and Product Evaluation

DevFeed: [AI Evals: Common Questions About Model and Product Evaluation](<https://devfeed.tech/articles/ai-evals-everything-you-need-to-know-18789.md>)

Original publisher: [Read original article](<https://hamel.dev/blog/posts/evals-faq/>)

Author: Shreya Shankar

Published: 2025-05-28T07:00:00Z

Content type: tutorial

Language: en

Sources: [Hamel Husain](<https://devfeed.tech/sources/hamel-husain.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Code](<https://devfeed.tech/topics/code.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-evals](<https://devfeed.tech/tags/ai-evals.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [code](<https://devfeed.tech/tags/code.md>), [evals](<https://devfeed.tech/tags/evals.md>), [llms](<https://devfeed.tech/tags/llms.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [tools](<https://devfeed.tech/tags/tools.md>)

## AI overview

This FAQ explains AI evaluations as tests for determining whether an AI system meets user and business goals. It distinguishes model benchmarks from product evaluations, which measure a specific product across its model, prompts, retrieval, tools, and application code.

## Source excerpt

This document curates the most common questions Shreya and I received while teaching 700+ engineers & PMs AI Evals. Warning: These are sharp opinions about what works in most cases. They are not universal truths. Use your judgment. For a guided path through the rest of our evals work, use the AI evals topic hub. 👉 Want to learn more about AI Evals? Check out our AI Evals course. It's a live cohort with hands on exercises and office hours. Here is a 25% discount code for readers. 👈 Getting Started & Fundamentals Q: What are AI Evals? AI evals are tests that tell you whether an AI system is doing what you want. They give your team feedback when the product drifts from user needs or business goals. The failures they catch also become data you can use to improve the system. More formally, evaluation is the systematic measurement of quality. Each eval checks one behavior on relevant examples and returns a score or structured review. Most AI products need several evals because they can fail in different ways. When you hear the word "evals," it usually refers to one of two things: model benchmarks or product evals. Model benchmarks Model benchmarks compare general-purpose models on shared tasks. Model providers publish these benchmark results when they release new models. Common examples include GPQA Diamond for graduate-level science reasoning, Terminal-Bench for agents doing complex work in command-line environments, and MMLU for knowledge and reasoning across a wide range of subjects. These scores can help you choose a promising model as a starting point. To assess quality on your own tasks you need product evals, which we discuss next. Product evals Product evals measure whether your specific AI product does what you want it to do. They turn your judgment about what a good product experience looks like into metrics you can track. Product evals encompass all components of your product, including the model, prompts, retrieval, tools, and application code. This flavor of