# How we knew COVID was over (and what our models had to unlearn)

DevFeed: [How we knew COVID was over (and what our models had to unlearn)](<https://devfeed.tech/articles/how-we-knew-covid-was-over-and-what-our-models-had-to-unlearn-1218.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/how-we-knew-covid-was-over-and-what-our-models-had-to-unlearn-c606b9bdb0ab?source=rss----53c7c27702d5---4>)

Author: Harrison Katz

Published: 2026-08-19T17:01:03Z

Content type: article

Language: en

Sources: [Airbnb](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [Finance](<https://devfeed.tech/topics/finance.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [dataset](<https://devfeed.tech/topics/dataset.md>)

Tags: [airbnb](<https://devfeed.tech/tags/airbnb.md>), [bias](<https://devfeed.tech/tags/bias.md>), [cancellations](<https://devfeed.tech/tags/cancellations.md>), [covid](<https://devfeed.tech/tags/covid.md>), [data](<https://devfeed.tech/tags/data.md>), [data-modeling](<https://devfeed.tech/tags/data-modeling.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [demand](<https://devfeed.tech/tags/demand.md>), [easy](<https://devfeed.tech/tags/easy.md>), [features](<https://devfeed.tech/tags/features.md>), [financial](<https://devfeed.tech/tags/financial.md>), [forecast](<https://devfeed.tech/tags/forecast.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [leave](<https://devfeed.tech/tags/leave.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [markets](<https://devfeed.tech/tags/markets.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [rebuild](<https://devfeed.tech/tags/rebuild.md>), [risk](<https://devfeed.tech/tags/risk.md>), [routine](<https://devfeed.tech/tags/routine.md>), [science](<https://devfeed.tech/tags/science.md>), [technology](<https://devfeed.tech/tags/technology.md>)

## AI overview

Airbnb's forecasting team treats a persistent miss as a choice among refitting, respecifying, or holding a model. Refit when the data-generating process remains the same and parameters have drifted; respecify when the model's structure cannot represent changed conditions; hold when a miss falls within expected variation, especially when an unexplained surprise could distort an off-cycle update. Routine refits can absorb temporary spikes or preserve outdated assumptions, while rebuilding in response to a short-lived shock can create a fragile model. During COVID recovery, markets that had once moved together diverged, making the old geographic pooling structure unstable: refitting alone produced about three times the pre-shock baseline error. Changing the model to borrow across geographic neighbors with shared recovery patterns cut error by a little over half versus refitting and brought it close to the pre-shock baseline. The article's central lesson is to reassess and sometimes remove assumptions after a shock, since updating data alone does not correct stale model structure.

## Source excerpt

When we retrain, when we rebuild, and when we leave a model alone. By: Harrison Katz A forecast that carries weight The Forecasting Data Science team at Airbnb produces many of the forecasts the rest of the company plans around: demand, bookings, cancellations, and a range of finer cuts by market and segment, refreshed continuously across thousands of markets. The targets differ, and the models differ, but they have one thing in common: Other teams build on top of them. This means a forecast that is casually wrong is not a clean miss, as it might be in an academic setting. That's because a small bias does not stay small once a lot of decisions are riding on it. So when one of those forecasts starts to drift, what to do about it is not really a methods question. It is a risk question, and an easy one to get wrong, which we have from time to time. One of these forecasts had been missing, compared to what actually happened after the forecast was released, in the same direction for a couple of quarters. This bias persisted after several routine refreshes. The usual solution would be to fully retrain the model: pull in the recent data, refit the model again, and ship. But we wanted to understand the source of the bias, rather than simply hoping an update would eliminate it. If you're interested in other posts on this topic, you can learn more about how COVID impacted Airbnb's financial models or how we dealt with disruption to our models during the pandemic. This post is about the discipline that came out of both: how we now decide whether a struggling forecast needs new data, a new model, or no changes at all. One word, three decisions The easy mistake is treating the choice to "retrain" a model as a single action. It is three separate actions -- refitting, respecifying, or holding -- and none of them is particularly similar to the others. Refitting is the cheaper option. Same model, same structure, same features, updated with newer data. This is what most people mean by