# The Airbnb Tech Blog - Medium

Creative engineers and data scientists building a world where you can belong anywhere. http://airbnb.io - Medium

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Beyond the model: Engineering AI infra with scientific judgement

DevFeed: [Beyond the model: Engineering AI infra with scientific judgement](<https://devfeed.tech/articles/beyond-the-model-engineering-ai-infra-with-scientific-judgement-26973.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/beyond-the-model-engineering-ai-infra-with-scientific-judgement-371316d43261?source=rss----53c7c27702d5---4>)

Author: AirbnbEng

Published: 2026-09-15T17:06:18Z

Content type: article

Language: en

Sources: [The Airbnb Tech Blog - Medium](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Data Science](<https://devfeed.tech/topics/data-science.md>), [data](<https://devfeed.tech/topics/data.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [agent-harness](<https://devfeed.tech/tags/agent-harness.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [data](<https://devfeed.tech/tags/data.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [llms](<https://devfeed.tech/tags/llms.md>), [quality](<https://devfeed.tech/tags/quality.md>), [science](<https://devfeed.tech/tags/science.md>), [trust](<https://devfeed.tech/tags/trust.md>)

### AI overview

Airbnb describes an agent harness for data science that embeds scientific methodology around an AI model. The system guides agents through framing questions, selecting evidence, and recording decisions so unstructured-data investigations can be reproduced, audited, challenged, and extended across languages, geographies, and LLM-based products.

### Source excerpt

How Airbnb's agent harness transforms unstructured data exploration by encoding scientific methodology into scalable, reproducible, and audit-ready infrastructure. By: Wren Dougherty Ask a coding agent to analyze 100,000 customer support conversations and within minutes you'll have a polished taxonomy, precise prevalence numbers, and an executive-ready summary. What you can't see is the investigation that produced them: the methods it chose, the evidence it weighed, how much to trust it, or whether a second request would agree. All that reaches you is the polish. The model is undeniably intelligent, but intelligence without methodology is not science. LLMs certainly make for confident scientists, but we need them to be responsible ones. Smarter models help, but intelligence has never been the whole of science, in people or in machines. The method is as much the product as the answer. That is the idea behind the agent harness we built for data science: the methodology itself, built as infrastructure around the model. It governs how an AI agent operates, from framing a question to selecting evidence to recording decisions, so results can be reproduced, audited, and challenged, and the method shared, inspected, and built on. The challenge of unstructured data exploration In 2025, Airbnb was preparing to launch an AI customer service assistant. Before it could ship, we needed to understand exactly what kinds of situations it would face in the real world. That included rare events that could be risky for AI to interact with, and involved examining their taxonomy and prevalence to create the datasets that would help us build a more responsible product. The investigative work to do this was rigorous, but the process was deeply artisanal. Months of high-touch iteration went into each investigation, from finding the right data, reviewing samples with experts, and generating representative datasets, and the method was manually curated across notebooks, tables, docs, and indiv

## Project Lighthouse -- Part 3: Introducing project-lighthouse-anonymize

DevFeed: [Project Lighthouse -- Part 3: Introducing project-lighthouse-anonymize](<https://devfeed.tech/articles/project-lighthouse-part-3-introducing-project-lighthouse-anonymize-1220.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/project-lighthouse-part-3-introducing-project-lighthouse-anonymize-74f8b26653fb?source=rss----53c7c27702d5---4>)

Author: Adam Bloomston

Published: 2026-08-25T17:01:02Z

Content type: article

Language: en

Sources: [The Airbnb Tech Blog - Medium](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [Algorithms](<https://devfeed.tech/topics/algorithms.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Python](<https://devfeed.tech/topics/python.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [Code](<https://devfeed.tech/topics/code.md>), [Programming](<https://devfeed.tech/topics/programming.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [code](<https://devfeed.tech/tags/code.md>), [data](<https://devfeed.tech/tags/data.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [python](<https://devfeed.tech/tags/python.md>), [scale](<https://devfeed.tech/tags/scale.md>), [technology](<https://devfeed.tech/tags/technology.md>), [validation](<https://devfeed.tech/tags/validation.md>)

### AI overview

Project Lighthouse -- Part 3 introduces project-lighthouse-anonymize, an open-source Python library for privacy-preserving anonymization. The article describes the project's k-anonymity methodology, scalable Core Mondrian algorithm, parallel processing architecture, and data quality validation framework, alongside two new technical papers.

### Source excerpt

Project Lighthouse -- Part 3: Introducing project-lighthouse-anonymizeThe data in Project Lighthouse is powered by privacy-preserving anonymization code. We've put this code into open source, and published two new technical papers detailing the scalable algorithms and data quality frameworks behind it. By: Adam Bloomston Introduction In 2020, we launched Project Lighthouse, which we developed in partnership with leading civil rights and privacy organizations. As our 2020 announcement details, Project Lighthouse enables us to measure potential disparities in user experiences. This work uses perceived race data that is never linked to individual accounts; we only use this data for measuring potential disparities, and users who want to opt-out can do so by turning off the data use settings in their account's Privacy page. Our results, shared in 2024, demonstrate how we use these analyses to measure our progress in mitigating those disparities. Earlier this year, we open-sourced project-lighthouse-anonymize, the Python library that powers Project Lighthouse's anonymization process. To provide the full technical foundation for this work, we also published two new papers on arXiv alongside the code release. Together with our original 2020 paper, these three papers form a complete story: the foundational methodology, the scalable implementation, and the quality validation framework. The foundational methodology (2020) Our original 2020 paper established the privacy-by-design approach for Project Lighthouse and provides the rationale for choosing k-anonymity as the technical privacy model to prevent sensitive attribute disclosure at scale. For an introduction to this paper, see our first blog post on p-sensitive k-anonymity and our second blog post on measurement with anonymized data. Core Mondrian: Scalable partition-based anonymization (2025) The first of our new papers, Core Mondrian: Basic Mondrian beyond k-anonymity, presents the k-anonymity algorithm at the heart of th

## How we knew COVID was over (and what our models had to unlearn)

DevFeed: [How we knew COVID was over (and what our models had to unlearn)](<https://devfeed.tech/articles/how-we-knew-covid-was-over-and-what-our-models-had-to-unlearn-1218.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/how-we-knew-covid-was-over-and-what-our-models-had-to-unlearn-c606b9bdb0ab?source=rss----53c7c27702d5---4>)

Author: Harrison Katz

Published: 2026-08-19T17:01:03Z

Content type: article

Language: en

Sources: [The Airbnb Tech Blog - Medium](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [Data Science](<https://devfeed.tech/topics/data-science.md>), [data](<https://devfeed.tech/topics/data.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [Process](<https://devfeed.tech/topics/process.md>)

Tags: [company](<https://devfeed.tech/tags/company.md>), [data](<https://devfeed.tech/tags/data.md>), [data-modeling](<https://devfeed.tech/tags/data-modeling.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [models](<https://devfeed.tech/tags/models.md>), [process](<https://devfeed.tech/tags/process.md>), [technology](<https://devfeed.tech/tags/technology.md>)

### AI overview

An Airbnb forecasting data science team explains how it responds when production forecasts drift, distinguishing between refitting a model with newer data, respecifying its structure, and holding it unchanged. The article emphasizes diagnosing the source of persistent bias and managing the risks of model updates that influence company decisions.

### Source excerpt

When we retrain, when we rebuild, and when we leave a model alone. By: Harrison Katz A forecast that carries weight The Forecasting Data Science team at Airbnb produces many of the forecasts the rest of the company plans around: demand, bookings, cancellations, and a range of finer cuts by market and segment, refreshed continuously across thousands of markets. The targets differ, and the models differ, but they have one thing in common: Other teams build on top of them. This means a forecast that is casually wrong is not a clean miss, as it might be in an academic setting. That's because a small bias does not stay small once a lot of decisions are riding on it. So when one of those forecasts starts to drift, what to do about it is not really a methods question. It is a risk question, and an easy one to get wrong, which we have from time to time. One of these forecasts had been missing, compared to what actually happened after the forecast was released, in the same direction for a couple of quarters. This bias persisted after several routine refreshes. The usual solution would be to fully retrain the model: pull in the recent data, refit the model again, and ship. But we wanted to understand the source of the bias, rather than simply hoping an update would eliminate it. If you're interested in other posts on this topic, you can learn more about how COVID impacted Airbnb's financial models or how we dealt with disruption to our models during the pandemic. This post is about the discipline that came out of both: how we now decide whether a struggling forecast needs new data, a new model, or no changes at all. One word, three decisions The easy mistake is treating the choice to "retrain" a model as a single action. It is three separate actions -- refitting, respecifying, or holding -- and none of them is particularly similar to the others. Refitting is the cheaper option. Same model, same structure, same features, updated with newer data. This is what most people mean by

## Flexible Authentication: Reimagining authentication for millions of users at Airbnb

DevFeed: [Flexible Authentication: Reimagining authentication for millions of users at Airbnb](<https://devfeed.tech/articles/flexible-authentication-reimagining-authentication-for-millions-of-users-at-airbnb-1216.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/flexible-authentication-reimagining-authentication-for-millions-of-users-at-airbnb-3a8a4c917137?source=rss----53c7c27702d5---4>)

Author: Jose Santos

Published: 2026-08-12T17:01:04Z

Content type: article

Language: en

Sources: [The Airbnb Tech Blog - Medium](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [Authentication](<https://devfeed.tech/topics/authentication.md>), [App](<https://devfeed.tech/topics/app.md>)

Tags: [app](<https://devfeed.tech/tags/app.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [authentication](<https://devfeed.tech/tags/authentication.md>), [blog](<https://devfeed.tech/tags/blog.md>), [brazil](<https://devfeed.tech/tags/brazil.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [identity](<https://devfeed.tech/tags/identity.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [product-intuition](<https://devfeed.tech/tags/product-intuition.md>), [revenue](<https://devfeed.tech/tags/revenue.md>), [software-architecture](<https://devfeed.tech/tags/software-architecture.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [technology](<https://devfeed.tech/tags/technology.md>), [whatsapp](<https://devfeed.tech/tags/whatsapp.md>)

### AI overview

Airbnb describes Flexible Authentication, a redesign of login and signup for users with varied session patterns, devices, and login methods. Its central model identifies the user first and then selects the verification challenge best suited to the person and context, such as WhatsApp OTP or a regional identity provider.

### Source excerpt

Rebuilding login and signup surfaced product insights, not just technical challenges. Here's how we designed Flexible Authentication at the intersection of product intuition and technical architecture. By: Jose Santos, Mike Barry For Airbnb, logins at irregular intervals are normal. A guest books a trip in January and may not open the app again until summer. A host checks back only when a reservation comes in, and may be busy with other activities when it does. For a two-sided marketplace where a failed login means a lost booking, and lost revenue for both the guest and the host, long gaps between sessions are a structural challenge, not an edge case. Our authentication system had grown organically over a decade, adding new login methods like Social Login, Email OTP, and Phone over time. We recognized that our users have diverse needs: many, such as hosts, use Airbnb daily, while others visit less frequently, when they are planning trips or traveling. To provide a seamless experience, we needed to ensure we were doing a great job supporting all of these usage patterns, helping users pick up exactly where they left off, regardless of how much time had passed or which device they were using. In this blog post, we'll walk through how we rebuilt Airbnb's authentication flows using a new paradigm we call Flexible Authentication, and how each architectural decision was guided by insights into the user's experience. Identify first then Challenge The 'Identify first then Challenge' model was the product insight that reframed the project. Our old system treated authentication as a single question: can this person prove who they are? The real question is more nuanced: given the multiple aspects of the user session, what's the easiest way for them to verify it? A traveler in Brazil who registered with a phone number is better served by a WhatsApp one-time password (OTP) than by SMS, since WhatsApp penetration far exceeds SMS in this market. A returning host in South Korea is b

## Eval-driven development: Lessons from evaluating GenAI at scale

DevFeed: [Eval-driven development: Lessons from evaluating GenAI at scale](<https://devfeed.tech/articles/eval-driven-development-lessons-from-evaluating-genai-at-scale-1215.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/eval-driven-development-lessons-from-evaluating-genai-at-scale-e817e5ae5788?source=rss----53c7c27702d5---4>)

Author: Rohit Girme

Published: 2026-07-28T17:01:03Z

Content type: article

Language: en

Sources: [The Airbnb Tech Blog - Medium](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [Generative AI](<https://devfeed.tech/topics/generative-ai.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Tooling](<https://devfeed.tech/topics/tooling.md>)

Tags: [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [article](<https://devfeed.tech/tags/article.md>), [development](<https://devfeed.tech/tags/development.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [eval](<https://devfeed.tech/tags/eval.md>), [evals](<https://devfeed.tech/tags/evals.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [genai](<https://devfeed.tech/tags/genai.md>), [generation](<https://devfeed.tech/tags/generation.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [llm](<https://devfeed.tech/tags/llm.md>), [software](<https://devfeed.tech/tags/software.md>), [software-testing](<https://devfeed.tech/tags/software-testing.md>), [testing](<https://devfeed.tech/tags/testing.md>), [tooling](<https://devfeed.tech/tags/tooling.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

### AI overview

This article presents eval-driven development as a core engineering discipline for trustworthy Generative AI products. It explains why evaluating LLM systems is difficult, including non-deterministic outputs, subjective correctness, AI-based evaluation risks, and failures across retrieval, reasoning, tool calls, and generation. It shares foundational evaluation practices and cautions that teams should plan evaluation early and ground success criteria in their data.

### Source excerpt

How Airbnb teams build trustworthy Generative AI products by treating evaluation as a first-class engineering discipline; not an afterthought.Nestled into the lush hillside, this stunning modern retreat features striking natural wood architecture, terraced balconies, and a serene landscape. By: Rohit Girme, Dan Miller, Mia Zhao, Lifan Yang, Clint Kelly Introduction Generative AI breaks a lot of the assumptions that used to hold true for software testing. Unlike traditional software, LLM outputs are non-deterministic, and "correct" is subjective. Because so much judgment is involved, you often need an AI to evaluate an AI, which introduces its own potential failure modes. Making matters more complicated, a single interaction with an LLM can chain retrieval, reasoning, tool calls, and generation, each of which can fail independently. At Airbnb, we build LLM-powered features across our product, with recent launches including review highlights, AI customer support, smart communication features for guests and hosts, and more. Behind the scenes, we also use AI to help us spot trends and understand what's working, guiding where we improve the product next. Each product team may have its own evaluation criteria, process, workflows, etc. However, these are built on top of some common foundations and principles. An infrastructure team provides tooling and best practices, incorporating learnings across domains so that they are shared with everyone building products at Airbnb. In this article, we wanted to share some of these best practices and learnings with the broader engineering community. Please note that the recommendations here are not intended to be prescriptive; there is no one-size-fits all approach when it comes to running evals. 1. Foundation Evaluating LLM-based systems is challenging work, and this should be planned for at the outset. Without a deliberate strategy, three things tend to happen: False confidence: A generic "helpfulness" metric scores well, you ship,

## Personalizing Airbnb search by learning from the guest journey

DevFeed: [Personalizing Airbnb search by learning from the guest journey](<https://devfeed.tech/articles/personalizing-airbnb-search-by-learning-from-the-guest-journey-1219.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/personalizing-airbnb-search-by-learning-from-the-guest-journey-bcefd1915624?source=rss----53c7c27702d5---4>)

Author: Daochen Zha

Published: 2026-07-21T17:01:04Z

Content type: article

Language: en

Sources: [The Airbnb Tech Blog - Medium](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [Temporal data](<https://devfeed.tech/topics/temporal-data.md>), [Data Science](<https://devfeed.tech/topics/data-science.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [blog](<https://devfeed.tech/tags/blog.md>), [conversion](<https://devfeed.tech/tags/conversion.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [model](<https://devfeed.tech/tags/model.md>), [recommendation-system](<https://devfeed.tech/tags/recommendation-system.md>), [research](<https://devfeed.tech/tags/research.md>), [scale](<https://devfeed.tech/tags/scale.md>), [search](<https://devfeed.tech/tags/search.md>), [technology](<https://devfeed.tech/tags/technology.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

Airbnb describes a Transformer-based sequence model for personalizing search by encoding years of guest behavior, including listing views, bookings, reviews, and cancellations. The system learns richer representations of guest preferences to improve listing relevance and booking conversion while addressing very long, noisy event sequences and the cost of training on hundreds of millions of search-label pairs.

### Source excerpt

How we built a Transformer-based sequence model that encodes years of guest behavior to surface the right listings at the right time. By: Daochen Zha, Chun How Tan, Xin Liu, Bin Xu, Han Zhao, Xiaowei Liu, Jun Shi, Tracy Yu, Hui Gao, Huiji Gao, Liwei He, Michael Kinoti, Stephanie Moyerman, and Sanjeev Katariya Introduction Planning a trip on Airbnb rarely happens in a single session. A guest searching for a place to stay in San Francisco might browse dozens of listings over several days, leaving behind a trail of views. Typically, over a period of years, that same guest will have accumulated many previous bookings, reviews, and the occasional cancellation. Taken together, these events reveal a great deal about what that guest values in a stay. For years, Airbnb's search ranking captured this through hand-crafted features: aggregated statistics such as total past bookings or average listing price. These worked well, but as the feature count grew into the hundreds, the approach became harder to scale and increasingly limited in expressiveness. In this blog post, we describe how we built a sequence modeling system that encodes the full guest journey using a Transformer, learning richer representations of guest preferences to deliver more personalized search results. An example of a guest journey, which is typically long, exploratory, and complex.Challenges Event sequences per guest present three core challenges. First, they are dominated by listing views, which account for the vast majority of all events -- some guests accumulate hundreds of thousands of them -- making raw sequences computationally intractable to model directly. The distribution of event types, with the majority being listing views. Second, unlike social media platforms, which optimize for engagement, Airbnb optimizes for booking conversion. Bookings are rare, compared to events, and deliberate, whereas a listing view could reflect genuine intent or simply idle browsing. Building a model that generalizes

## From weeks to a day: how we made LLM evaluation fast enough to iterate on

DevFeed: [From weeks to a day: how we made LLM evaluation fast enough to iterate on](<https://devfeed.tech/articles/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate-on-1217.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate-on-14e2d35198b4?source=rss----53c7c27702d5---4>)

Author: Baharak Saberidokht

Published: 2026-07-14T17:01:03Z

Content type: article

Language: en

Sources: [The Airbnb Tech Blog - Medium](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [data](<https://devfeed.tech/topics/data.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [data](<https://devfeed.tech/tags/data.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [eval](<https://devfeed.tech/tags/eval.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [llm](<https://devfeed.tech/tags/llm.md>), [models](<https://devfeed.tech/tags/models.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [techniques](<https://devfeed.tech/tags/techniques.md>), [technology](<https://devfeed.tech/tags/technology.md>), [validation](<https://devfeed.tech/tags/validation.md>)

### AI overview

The article explains how Airbnb made production LLM evaluation fast enough for rapid iteration by addressing infrastructure challenges across four layers. It focuses on diagnosing evaluation noise from data and judging uncertainty, validating complete system paths, and applying classical software engineering techniques to make measurements more trustworthy.

### Source excerpt

Training an LLM is the easy part. The hard part is designing experiments and evaluations that you can trust enough to know whether the new model is actually an improvement. By: Baharak Saberidokht Introduction Shipping a production LLM system means iterating fast on improvements to something that is, by construction, non-deterministic. Models drift, judges disagree with themselves, references regenerate as different strings, and bugs may persist until the next release, because retraining takes weeks. Most of this friction comes from infrastructure challenges, not model quality, and the fixes come from classical software engineering techniques. At Airbnb, we built reliable LLM infrastructure by addressing four layers. Three correspond to engineering enhancements we've made; the fourth is the integration layer that ties the rest together -- the one that is easiest to overlook, because each individual component looks fine in isolation. The approach rests on two observations: the seams are where things break, and finding those breaks requires exercising the full path, not just validating each component in isolation. Figure 1. The four layers of the production LLM stack. Bounded model mutation requires trustworthy measurement, and end-to-end validation requires the eval foundation to be fast enough to run on the combined path. Layer 1: Name it before trying to remove it Layer 1 is diagnostic framing of evaluation noise. This layer addresses two different sources of indeterminacy: data and judging uncertainty. Classical ML metrics are deterministic: F1, BLEU, and accuracy return the same number on the same input. With LLMs in the evaluation loop, that assumption dies. Judges score identical inputs differently across runs, and LLM-generated references regenerate as different strings. A two percent score movement can mean the model improved, the judge drifted, the references shifted, or some combination. We cannot tell which without naming which kind of noise we are looking

## Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world

DevFeed: [Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world](<https://devfeed.tech/articles/scaling-beyond-one-how-airbnb-evolved-its-data-architecture-for-a-multi-product-world-1222.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/scaling-beyond-one-how-airbnb-evolved-its-data-architecture-for-a-multi-product-world-6125645d470c?source=rss----53c7c27702d5---4>)

Author: Patrick Lam

Published: 2026-06-09T17:01:02Z

Content type: article

Language: en

Sources: [The Airbnb Tech Blog - Medium](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [Data Infrastructure](<https://devfeed.tech/topics/data-infrastructure.md>), [data-modeling](<https://devfeed.tech/topics/data-modeling.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [data](<https://devfeed.tech/tags/data.md>), [data-architecture](<https://devfeed.tech/tags/data-architecture.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-infrastructure](<https://devfeed.tech/tags/data-infrastructure.md>), [data-modeling](<https://devfeed.tech/tags/data-modeling.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [offline](<https://devfeed.tech/tags/offline.md>), [post](<https://devfeed.tech/tags/post.md>), [technology](<https://devfeed.tech/tags/technology.md>)

### AI overview

Airbnb's data and analytics engineering teams evolved a decade-old offline data warehouse to support Homes, Experiences, and Services. The article examines the trade-offs between separate product-specific data models and a unified monolithic model while describing the need for a consistent, flexible, and scalable data foundation.

### Source excerpt

How Airbnb's data engineers and analytics engineers built a consistent and flexible data modeling framework to support the expansion into Homes, Experiences, and Services. By: Patrick Lam, Namrata Lamba, Jamie Stober With the May 2025 Summer Release, Airbnb redesigned its app, relaunched Experiences, and debuted Services, pushing us beyond our traditional Homes focus. For the data teams, this meant rapidly evolving a decade-old infrastructure to integrate two brand-new product pillars. Our data engineers and analytics engineers rose to the challenge by building a consistent and flexible framework to serve as a robust and scalable data foundation for the next decade of growth. But getting there wasn't straightforward. This fundamental shift surfaced a critical question for our data organization: How do you evolve your offline data architecture to support new product lines without introducing disorder in vital analytics services? We knew the approach we took would have long-lasting implications. A fragmented strategy risked creating data silos, inconsistent analytics, and a tangled web of technical debt that would likely slow down future innovation. In this post, we'll take you behind the scenes to share key decisions that we made, the framework that emerged, and the lessons that helped reshape our offline data warehouse for the future. Note that we focus specifically on our offline data warehouse (the analytics-oriented data infrastructure owned by our data engineers and analytics engineers) rather than the online data systems that serve the app directly, as the two domains have fundamentally different requirements, constraints, and design philosophies that warrant separate treatment. The core dilemma: separate vs. monolithic The first and most critical question was how to structure offline data for the new, three-product world, with Homes, a refreshed Experiences product, and the new Services offering. This involved a trade-off between two main approaches: Separate

## Sitar-agent: Building a reliable dynamic configuration sidecar at scale

DevFeed: [Sitar-agent: Building a reliable dynamic configuration sidecar at scale](<https://devfeed.tech/articles/sitar-agent-building-a-reliable-dynamic-configuration-sidecar-at-scale-1223.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/sitar-agent-building-a-reliable-dynamic-configuration-sidecar-at-scale-b7e00c152068?source=rss----53c7c27702d5---4>)

Author: Bo Teng

Published: 2026-06-04T17:01:04Z

Content type: article

Language: en

Sources: [The Airbnb Tech Blog - Medium](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [configuration](<https://devfeed.tech/topics/configuration.md>), [Kubernetes](<https://devfeed.tech/topics/kubernetes.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [s3](<https://devfeed.tech/tags/s3.md>), [scale](<https://devfeed.tech/tags/scale.md>), [software-architecture](<https://devfeed.tech/tags/software-architecture.md>), [software-development](<https://devfeed.tech/tags/software-development.md>), [sync](<https://devfeed.tech/tags/sync.md>)

### AI overview

Airbnb describes Sitar-agent, a Kubernetes sidecar that reliably delivers dynamic configuration to service pods. It bootstraps from S3 snapshots, then synchronizes newer configuration from the Sitar Service.

### Source excerpt

How Airbnb built a Kubernetes sidecar to deliver dynamic configuration reliably at scale. By: Bo Teng, Cosmo Qiu, Siyuan Zhou, Ankur Soni, Xin Huang, Willis Harvey Introduction In our previous post, we explored Airbnb's dynamic configuration system, Sitar, with a focus on service architecture and configuration change safety. Now for the harder question: once a config change is committed, which happens several times each minute, how does it actually reach the thousands of Airbnb's service instances reliably, quickly, and without redeploying the services? This post describes sitar agent: a lightweight Kubernetes sidecar that runs alongside every subscribed service pod, continuously synchronizing the latest configurations from the service backend and making them available on the local filesystem for reads. In this post, we will first go through the configuration delivery life cycle, and then discuss some key design choices for the sitar-agent sidecar. Config delivery life cycle The diagram below illustrates the end-to-end journey of a configuration change, from the developer-facing layer to the production service fleet. Sitar config delivery lifecycle Step 1 -- Config creation/update Developers create or update configuration values through either Git flow or the web UI. These changes are committed to the Sitar Service, where they are stored with full versioning, change logs, and ACL enforcement. Step 2 -- Hourly snapshot upload The Snapshot Service periodically packages the full state of all config groups and uploads compressed snapshots to AWS S3. Step 3.1 -- Preload snapshot from S3 (on pod startup) When a production service pod starts, the sitar-agent sidecar runs first. It downloads the latest snapshot for each subscribed tenant's configs from S3 to the mounted disk (shared between sitar-agent and the main container). This allows the agent to bootstrap from a known-good state without fetching every config from the Sitar Service from scratch on every restart. Preloadin

## When history fails you, borrow from geography

DevFeed: [When history fails you, borrow from geography](<https://devfeed.tech/articles/when-history-fails-you-borrow-from-geography-1224.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/when-history-fails-you-borrow-from-geography-915a72b91b5c?source=rss----53c7c27702d5---4>)

Author: Harrison Katz

Published: 2026-06-02T17:01:04Z

Content type: article

Language: en

Sources: [The Airbnb Tech Blog - Medium](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [data](<https://devfeed.tech/topics/data.md>), [datasets](<https://devfeed.tech/topics/datasets.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [building](<https://devfeed.tech/tags/building.md>), [data](<https://devfeed.tech/tags/data.md>), [data-modeling](<https://devfeed.tech/tags/data-modeling.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [post](<https://devfeed.tech/tags/post.md>), [technology](<https://devfeed.tech/tags/technology.md>), [travel-industry](<https://devfeed.tech/tags/travel-industry.md>)

### AI overview

Airbnb describes a forecasting approach for the uneven post-COVID travel recovery, using sequential signals from other geographies and propagating prior information to produce corridor-level forecasts when local post-shock data was scarce.

### Source excerpt

How Airbnb used sequential geographic recovery signals and prior propagation to generate reliable corridor-level forecasts when local data was scarce. By: Harrison Katz The problem with unprecedented shocks Almost every forecasting system is built on the same implicit assumption: the future will resemble the past. You train on historical data, you validate on holdout periods, and you trust that past patterns will at least roughly indicate future performance. When this assumption breaks, the model does not gracefully degrade; it fails confidently. It produces precise, well-calibrated intervals around the wrong answer. The acute phase of COVID, from early to late 2020, was a clear illustration of this, and we wrote about it in a previous post. But the more interesting forecasting problem was not the shutdown. It was everything that came after. The period from late 2020 through 2022 was not a single coherent regime. It was a sequence of overlapping, asynchronous changes: vaccine rollouts that reached some markets months before others, border reopenings that followed their own country-level timelines, reclosures triggered by new variants that hit different corridors (a pairing of the traveler's origin city and destination city) at different moments. Demand was not recovering uniformly. It was rebounding unevenly across every corner of the world, in ways that had no historical precedent and no single governing pattern. The standard response to a shock is to wait for each affected market to accumulate its own post-shock data and retrain locally. But Covid was among the biggest shocks the travel industry has faced in decades. With markets worldwide reopening and reclosing on staggered schedules, waiting for markets to settle meant forecasting blind for months at a time, across all markets, just when timely projections were most needed, in the circumstances. So we started building something different. When we could not simply look backward in time for relevant examples, we

## Scaling Airbnb's identity graph with a unified knowledge graph infrastructure

DevFeed: [Scaling Airbnb's identity graph with a unified knowledge graph infrastructure](<https://devfeed.tech/articles/scaling-airbnb-s-identity-graph-with-a-unified-knowledge-graph-infrastructure-1221.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/scaling-airbnbs-identity-graph-with-a-unified-knowledge-graph-infrastructure-ebac467b7836?source=rss----53c7c27702d5---4>)

Author: Lucen Zhao

Published: 2026-05-19T17:01:01Z

Content type: article

Language: en

Sources: [The Airbnb Tech Blog - Medium](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [Graphs](<https://devfeed.tech/topics/graphs.md>), [graph-database](<https://devfeed.tech/topics/graph-database.md>), [Architecture & Design](<https://devfeed.tech/topics/architecture-design.md>), [Scalability](<https://devfeed.tech/topics/scalability.md>), [Caching](<https://devfeed.tech/topics/caching.md>), [Low Latency](<https://devfeed.tech/topics/low-latency.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [caching](<https://devfeed.tech/tags/caching.md>), [data](<https://devfeed.tech/tags/data.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [graph-database](<https://devfeed.tech/tags/graph-database.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [knowledge-graph](<https://devfeed.tech/tags/knowledge-graph.md>), [latency](<https://devfeed.tech/tags/latency.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [safety](<https://devfeed.tech/tags/safety.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [technology](<https://devfeed.tech/tags/technology.md>)

### AI overview

Airbnb describes scaling its identity graph into a unified, internally managed knowledge graph infrastructure. The article covers the graph database and KV caching storage layer, graph service, near-real-time ingestion, low-latency access, and architectural evolution driven by increasing graph density and query complexity.

### Source excerpt

How Airbnb shifts from PaaS to an internal knowledge graph infrastructure at scale. By: Lucen Zhao, Shukun Yang, Ashish Jain Knowledge graphs offer a natural and powerful way to represent relationships between entities. Many real-world systems are fundamentally about connections. Airbnb's identity graph captures relationships between users in a graph database. The identity graph serves aggregated insights that enable user identity resolution and relationship understanding. These capabilities support a wide range of Trust and Safety use cases, from detecting suspicious activities to identifying linked accounts. Over time, the identity graph has grown into one of the largest and most complex graph data products at Airbnb, both in terms of scale and the complexity of queries it supports. In 2024, Airbnb began investing in a new, internally managed, paved-path graph data platform to build a unified knowledge graph infrastructure. Airbnb's identity graph became one of the first systems to adopt this platform. In this post, we'll walk through the foundations and challenges of the identity graph, introduce the architecture behind the graph infrastructure, and highlight several key optimizations that emerged during the onboarding process. Airbnb's identity graph Airbnb's identity graph is a critical foundation layer, playing an important role in Trust and Safety applications. It contains two major components: Graph data storage: a storage layer composed of a graph database and a key-value (KV) caching layer. It models users and relationships as vertices and edges. Most data is ingested in near real-time through asynchronous events and served through low-latency, real-time service calls. Graph service: this service provides a unified interface for accessing graph data. It retrieves data from underlying sources, including the graph database, applies aggregation logic or models as needed, and serves the results to downstream customer services. Evolution of the identity graph a