# From weeks to a day: how we made LLM evaluation fast enough to iterate on

DevFeed: [From weeks to a day: how we made LLM evaluation fast enough to iterate on](<https://devfeed.tech/articles/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate-on-1217.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate-on-14e2d35198b4?source=rss----53c7c27702d5---4>)

Author: Baharak Saberidokht

Published: 2026-07-14T17:01:03Z

Content type: article

Language: en

Sources: [Airbnb](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Integration testing](<https://devfeed.tech/topics/integration-testing.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [airbnb](<https://devfeed.tech/tags/airbnb.md>), [approach](<https://devfeed.tech/tags/approach.md>), [bleu](<https://devfeed.tech/tags/bleu.md>), [bugs](<https://devfeed.tech/tags/bugs.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [component](<https://devfeed.tech/tags/component.md>), [data](<https://devfeed.tech/tags/data.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [eval](<https://devfeed.tech/tags/eval.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [fast](<https://devfeed.tech/tags/fast.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [improvements](<https://devfeed.tech/tags/improvements.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [integration](<https://devfeed.tech/tags/integration.md>), [introduction](<https://devfeed.tech/tags/introduction.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-evaluation](<https://devfeed.tech/tags/llm-evaluation.md>), [technology](<https://devfeed.tech/tags/technology.md>)

## AI overview

Airbnb describes a four-layer infrastructure approach that reduced LLM evaluation and fix cycles to same-day iteration. First, distinguish data uncertainty from judge uncertainty, and test whether conclusions persist across judges, metrics, and sample groupings. Second, cache generated references by sample and configuration, and judge scores by sample, output, judge configuration, and metric; this makes repeated evaluations reproducible, cheaper, and resumable. Third, apply targeted rank-under-50 LoRA micro adapters that train in under an hour, with expert-domain regression checks and human review for high-uncertainty outputs, plus canary deployment and rollback. The authors caution that stacked patches can interfere, so overlapping patches should be fused, accumulated patches eventually retrained, and unused patches unloaded. Finally, run a small representative set through the full production path, combining traffic-weighted sampling with extra coverage of locales, modalities, and prior incident cases, and measure quality and tail latency. The layers depend on one another: deterministic measurement enables fast fixes and meaningful end-to-end checks.

## Source excerpt

Training an LLM is the easy part. The hard part is designing experiments and evaluations that you can trust enough to know whether the new model is actually an improvement. By: Baharak Saberidokht Introduction Shipping a production LLM system means iterating fast on improvements to something that is, by construction, non-deterministic. Models drift, judges disagree with themselves, references regenerate as different strings, and bugs may persist until the next release, because retraining takes weeks. Most of this friction comes from infrastructure challenges, not model quality, and the fixes come from classical software engineering techniques. At Airbnb, we built reliable LLM infrastructure by addressing four layers. Three correspond to engineering enhancements we've made; the fourth is the integration layer that ties the rest together -- the one that is easiest to overlook, because each individual component looks fine in isolation. The approach rests on two observations: the seams are where things break, and finding those breaks requires exercising the full path, not just validating each component in isolation. Figure 1. The four layers of the production LLM stack. Bounded model mutation requires trustworthy measurement, and end-to-end validation requires the eval foundation to be fast enough to run on the combined path. Layer 1: Name it before trying to remove it Layer 1 is diagnostic framing of evaluation noise. This layer addresses two different sources of indeterminacy: data and judging uncertainty. Classical ML metrics are deterministic: F1, BLEU, and accuracy return the same number on the same input. With LLMs in the evaluation loop, that assumption dies. Judges score identical inputs differently across runs, and LLM-generated references regenerate as different strings. A two percent score movement can mean the model improved, the judge drifted, the references shifted, or some combination. We cannot tell which without naming which kind of noise we are looking