# How we knew COVID was over (and what our models had to unlearn)

DevFeed: [How we knew COVID was over (and what our models had to unlearn)](<https://devfeed.tech/articles/how-we-knew-covid-was-over-and-what-our-models-had-to-unlearn-1218.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/how-we-knew-covid-was-over-and-what-our-models-had-to-unlearn-c606b9bdb0ab?source=rss----53c7c27702d5---4>)

Author: Harrison Katz

Published: 2026-08-19T17:01:03Z

Content type: article

Language: en

Sources: [The Airbnb Tech Blog - Medium](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [Data Science](<https://devfeed.tech/topics/data-science.md>), [data](<https://devfeed.tech/topics/data.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [Process](<https://devfeed.tech/topics/process.md>)

Tags: [company](<https://devfeed.tech/tags/company.md>), [data](<https://devfeed.tech/tags/data.md>), [data-modeling](<https://devfeed.tech/tags/data-modeling.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [models](<https://devfeed.tech/tags/models.md>), [process](<https://devfeed.tech/tags/process.md>), [technology](<https://devfeed.tech/tags/technology.md>)

## AI overview

An Airbnb forecasting data science team explains how it responds when production forecasts drift, distinguishing between refitting a model with newer data, respecifying its structure, and holding it unchanged. The article emphasizes diagnosing the source of persistent bias and managing the risks of model updates that influence company decisions.

## Source excerpt

When we retrain, when we rebuild, and when we leave a model alone. By: Harrison Katz A forecast that carries weight The Forecasting Data Science team at Airbnb produces many of the forecasts the rest of the company plans around: demand, bookings, cancellations, and a range of finer cuts by market and segment, refreshed continuously across thousands of markets. The targets differ, and the models differ, but they have one thing in common: Other teams build on top of them. This means a forecast that is casually wrong is not a clean miss, as it might be in an academic setting. That's because a small bias does not stay small once a lot of decisions are riding on it. So when one of those forecasts starts to drift, what to do about it is not really a methods question. It is a risk question, and an easy one to get wrong, which we have from time to time. One of these forecasts had been missing, compared to what actually happened after the forecast was released, in the same direction for a couple of quarters. This bias persisted after several routine refreshes. The usual solution would be to fully retrain the model: pull in the recent data, refit the model again, and ship. But we wanted to understand the source of the bias, rather than simply hoping an update would eliminate it. If you're interested in other posts on this topic, you can learn more about how COVID impacted Airbnb's financial models or how we dealt with disruption to our models during the pandemic. This post is about the discipline that came out of both: how we now decide whether a struggling forecast needs new data, a new model, or no changes at all. One word, three decisions The easy mistake is treating the choice to "retrain" a model as a single action. It is three separate actions -- refitting, respecifying, or holding -- and none of them is particularly similar to the others. Refitting is the cheaper option. Same model, same structure, same features, updated with newer data. This is what most people mean by