# Your AI Product Needs Evals

DevFeed: [Your AI Product Needs Evals](<https://devfeed.tech/articles/your-ai-product-needs-evals-18791.md>)

Original publisher: [Read original article](<https://hamel.dev/blog/posts/evals/>)

Author: Hamel Husain

Published: 2024-03-29T07:00:00Z

Content type: article

Language: en

Sources: [Hamel Husain](<https://devfeed.tech/sources/hamel-husain.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [Logging](<https://devfeed.tech/topics/logging.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [code](<https://devfeed.tech/tags/code.md>), [data](<https://devfeed.tech/tags/data.md>), [debugging](<https://devfeed.tech/tags/debugging.md>), [evals](<https://devfeed.tech/tags/evals.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [llms](<https://devfeed.tech/tags/llms.md>), [logging](<https://devfeed.tech/tags/logging.md>), [prompt-engineering](<https://devfeed.tech/tags/prompt-engineering.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [tests](<https://devfeed.tech/tags/tests.md>)

## AI overview

This article argues that robust evaluation systems are essential for building and improving LLM-powered AI products. It recommends combining quality evaluation, debugging through logging and data inspection, and system changes such as prompt engineering, fine-tuning, and code changes. It also introduces a case study involving Lucy, Rechat's real estate AI assistant.

## Source excerpt

Motivation I started working with language models five years ago when I led the team that created CodeSearchNet, a precursor to GitHub CoPilot. Since then, I've seen many successful and unsuccessful approaches to building LLM products. I've found that unsuccessful products almost always share a common root cause: a failure to create robust evaluation systems. I originally wrote this while helping companies build domain-specific AI products. I hope companies can save thousands of dollars in consulting fees by reading this post carefully. This post outlines my thoughts on building evaluation systems for LLMs-powered AI products. Iterating Quickly == Success Like software engineering, success with AI hinges on how fast you can iterate. You must have processes and tools for: Evaluating quality (ex: tests). Debugging issues (ex: logging & inspecting data). Changing the behavior or the system (prompt eng, fine-tuning, writing code) Many people focus exclusively on #3 above, which prevents them from improving their LLM products beyond a demo.1 Doing all three activities well creates a virtuous cycle differentiating great from mediocre AI products (see the diagram below for a visualization of this cycle). If you streamline your evaluation process, all other activities become easy. This is very similar to how tests in software engineering pay massive dividends in the long term despite requiring up-front investment. To ground this post in a real-world situation, I'll walk through a case study in which we built a system for rapid improvement. I'll primarily focus on evaluation as that is the most critical component. Case Study: Lucy, A Real Estate AI Assistant Rechat is a SaaS application that allows real estate professionals to perform various tasks, such as managing contracts, searching for listings, building creative assets, managing appointments, and more. The thesis of Rechat is that you can do everything in one place rather than having to context switch between many diff