# Eval-driven development: Lessons from evaluating GenAI at scale

DevFeed: [Eval-driven development: Lessons from evaluating GenAI at scale](<https://devfeed.tech/articles/eval-driven-development-lessons-from-evaluating-genai-at-scale-1215.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/eval-driven-development-lessons-from-evaluating-genai-at-scale-e817e5ae5788?source=rss----53c7c27702d5---4>)

Author: Rohit Girme

Published: 2026-07-28T17:01:03Z

Content type: article

Language: en

Sources: [The Airbnb Tech Blog - Medium](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [Generative AI](<https://devfeed.tech/topics/generative-ai.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Tooling](<https://devfeed.tech/topics/tooling.md>)

Tags: [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [article](<https://devfeed.tech/tags/article.md>), [development](<https://devfeed.tech/tags/development.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [eval](<https://devfeed.tech/tags/eval.md>), [evals](<https://devfeed.tech/tags/evals.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [genai](<https://devfeed.tech/tags/genai.md>), [generation](<https://devfeed.tech/tags/generation.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [llm](<https://devfeed.tech/tags/llm.md>), [software](<https://devfeed.tech/tags/software.md>), [software-testing](<https://devfeed.tech/tags/software-testing.md>), [testing](<https://devfeed.tech/tags/testing.md>), [tooling](<https://devfeed.tech/tags/tooling.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

## AI overview

This article presents eval-driven development as a core engineering discipline for trustworthy Generative AI products. It explains why evaluating LLM systems is difficult, including non-deterministic outputs, subjective correctness, AI-based evaluation risks, and failures across retrieval, reasoning, tool calls, and generation. It shares foundational evaluation practices and cautions that teams should plan evaluation early and ground success criteria in their data.

## Source excerpt

How Airbnb teams build trustworthy Generative AI products by treating evaluation as a first-class engineering discipline; not an afterthought.Nestled into the lush hillside, this stunning modern retreat features striking natural wood architecture, terraced balconies, and a serene landscape. By: Rohit Girme, Dan Miller, Mia Zhao, Lifan Yang, Clint Kelly Introduction Generative AI breaks a lot of the assumptions that used to hold true for software testing. Unlike traditional software, LLM outputs are non-deterministic, and "correct" is subjective. Because so much judgment is involved, you often need an AI to evaluate an AI, which introduces its own potential failure modes. Making matters more complicated, a single interaction with an LLM can chain retrieval, reasoning, tool calls, and generation, each of which can fail independently. At Airbnb, we build LLM-powered features across our product, with recent launches including review highlights, AI customer support, smart communication features for guests and hosts, and more. Behind the scenes, we also use AI to help us spot trends and understand what's working, guiding where we improve the product next. Each product team may have its own evaluation criteria, process, workflows, etc. However, these are built on top of some common foundations and principles. An infrastructure team provides tooling and best practices, incorporating learnings across domains so that they are shared with everyone building products at Airbnb. In this article, we wanted to share some of these best practices and learnings with the broader engineering community. Please note that the recommendations here are not intended to be prescriptive; there is no one-size-fits all approach when it comes to running evals. 1. Foundation Evaluating LLM-based systems is challenging work, and this should be planned for at the outset. Without a deliberate strategy, three things tend to happen: False confidence: A generic "helpfulness" metric scores well, you ship,