# From weeks to a day: how we made LLM evaluation fast enough to iterate on

DevFeed: [From weeks to a day: how we made LLM evaluation fast enough to iterate on](<https://devfeed.tech/articles/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate-on-1217.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate-on-14e2d35198b4?source=rss----53c7c27702d5---4>)

Author: Baharak Saberidokht

Published: 2026-07-14T17:01:03Z

Content type: article

Language: en

Sources: [The Airbnb Tech Blog - Medium](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [data](<https://devfeed.tech/topics/data.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [data](<https://devfeed.tech/tags/data.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [eval](<https://devfeed.tech/tags/eval.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [llm](<https://devfeed.tech/tags/llm.md>), [models](<https://devfeed.tech/tags/models.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [techniques](<https://devfeed.tech/tags/techniques.md>), [technology](<https://devfeed.tech/tags/technology.md>), [validation](<https://devfeed.tech/tags/validation.md>)

## AI overview

The article explains how Airbnb made production LLM evaluation fast enough for rapid iteration by addressing infrastructure challenges across four layers. It focuses on diagnosing evaluation noise from data and judging uncertainty, validating complete system paths, and applying classical software engineering techniques to make measurements more trustworthy.

## Source excerpt

Training an LLM is the easy part. The hard part is designing experiments and evaluations that you can trust enough to know whether the new model is actually an improvement. By: Baharak Saberidokht Introduction Shipping a production LLM system means iterating fast on improvements to something that is, by construction, non-deterministic. Models drift, judges disagree with themselves, references regenerate as different strings, and bugs may persist until the next release, because retraining takes weeks. Most of this friction comes from infrastructure challenges, not model quality, and the fixes come from classical software engineering techniques. At Airbnb, we built reliable LLM infrastructure by addressing four layers. Three correspond to engineering enhancements we've made; the fourth is the integration layer that ties the rest together -- the one that is easiest to overlook, because each individual component looks fine in isolation. The approach rests on two observations: the seams are where things break, and finding those breaks requires exercising the full path, not just validating each component in isolation. Figure 1. The four layers of the production LLM stack. Bounded model mutation requires trustworthy measurement, and end-to-end validation requires the eval foundation to be fast enough to run on the combined path. Layer 1: Name it before trying to remove it Layer 1 is diagnostic framing of evaluation noise. This layer addresses two different sources of indeterminacy: data and judging uncertainty. Classical ML metrics are deterministic: F1, BLEU, and accuracy return the same number on the same input. With LLMs in the evaluation loop, that assumption dies. Judges score identical inputs differently across runs, and LLM-generated references regenerate as different strings. A two percent score movement can mean the model improved, the judge drifted, the references shifted, or some combination. We cannot tell which without naming which kind of noise we are looking