# How Whatnot's Hourly ML Feature Pipeline Safely Handled a Failed Update

DevFeed: [How Whatnot's Hourly ML Feature Pipeline Safely Handled a Failed Update](<https://devfeed.tech/articles/the-ml-feature-pipeline-that-got-slower-and-no-one-noticed-23713.md>)

Original publisher: [Read original article](<https://medium.com/whatnot-engineering/the-ml-feature-pipeline-that-got-slower-and-no-one-noticed-8e90c224eae3?source=rss----162aeca881b0---4>)

Author: Whatnot Engineering

Published: 2026-05-07T16:01:01Z

Content type: article

Language: en

Sources: [Whatnot Engineering](<https://devfeed.tech/sources/whatnot-engineering.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [data](<https://devfeed.tech/topics/data.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [Processes](<https://devfeed.tech/topics/processes.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [feature](<https://devfeed.tech/tags/feature.md>), [feature-engineering](<https://devfeed.tech/tags/feature-engineering.md>), [feature-store](<https://devfeed.tech/tags/feature-store.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [ml](<https://devfeed.tech/tags/ml.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [recommendation-system](<https://devfeed.tech/tags/recommendation-system.md>), [recommendations](<https://devfeed.tech/tags/recommendations.md>), [run](<https://devfeed.tech/tags/run.md>)

## AI overview

Whatnot describes how its hourly ML feature pipeline handled a bad pull request that caused a feature group to produce zero rows. The online feature store served last-known-good features for up to two days, anomaly monitors detected the row-count drop, and the team reverted the change before the freshness gap became significant. The post shares lessons about safe degradation and operating frequently updated recommendation features.

## Source excerpt

David Kawashima, Miguel Fernandez-Montes Cuberta, Ford Bohrmann | Discovery A bad PR went out just as our hourly feature pipeline was about to run. The pipeline was responsible for recomputing fresh user behavior signals used in livestream ranking on the home feed. These features capture recent user intent: watches, bids, orders aggregated across users, sellers, and categories. A missed feature update means that the system stops responding to user signals immediately. This PR release caused one of our feature groups to suddenly produce zero rows ... but nothing broke! Even though a critical feature group for the model stopped updating, the online feature store continued serving the last-known-good features, buffered by a 2-day TTL and nothing broke explicitly. Our anomaly monitors went off immediately, flagging the drop in row counts. The team identified the bad change and reverted it before the freshness gap grew large enough to matter. No corrupted data reached production. The system absorbed the failure and degraded safely. This graceful degradation didn't happen by accident; it was the result of hardened processes from years of lived experience, seeing the way these systems can degrade in obvious or subtle ways. In this post, we share some of those lessons learned. The system: hourly feature pipeline Whatnot is a highly dynamic marketplace where buyers' preferences change at a moment's notice. A buyer might research a laptop, join a football-card break a few hours later, and buy chocolate before logging off for the night. Daily feature updates capture long-term preferences but miss intra-day shifts. To keep up with these rapidly changing buyer interests (in 2025) we moved from daily to hourly updates for features that power our ML feed ranking models. Fresher features are a critical component for capturing the live nature of the Whatnot marketplace, where the inventory available to purchase changes in real time. Moving from daily to hourly updates was a meaningful