# Whatnot Engineering

Whatnot is a livestream shopping platform and marketplace. We're building the future of social commerce. Our mission is to enable anyone to turn their passion into a business and bring people together through commerce. - Medium

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Don't let overload protection be the enemy of scaling

DevFeed: [Don't let overload protection be the enemy of scaling](<https://devfeed.tech/articles/don-t-let-overload-protection-be-the-enemy-of-scaling-131132.md>)

Original publisher: [Read original article](<https://medium.com/whatnot-engineering/dont-let-overload-protection-be-the-enemy-of-scaling-7f59abbcb070?source=rss----162aeca881b0---4>)

Author: Whatnot Engineering

Published: 2026-11-12T13:05:42Z

Content type: article

Language: en

Sources: [Whatnot Engineering](<https://devfeed.tech/sources/whatnot-engineering.md>)

Topics: [consistency](<https://devfeed.tech/topics/consistency.md>), [read-replica](<https://devfeed.tech/topics/read-replica.md>), [database-optimization](<https://devfeed.tech/topics/database-optimization.md>), [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [backend](<https://devfeed.tech/tags/backend.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [database](<https://devfeed.tech/tags/database.md>), [fleet](<https://devfeed.tech/tags/fleet.md>), [graphql](<https://devfeed.tech/tags/graphql.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [ip](<https://devfeed.tech/tags/ip.md>), [limit](<https://devfeed.tech/tags/limit.md>), [load](<https://devfeed.tech/tags/load.md>), [protection](<https://devfeed.tech/tags/protection.md>), [query](<https://devfeed.tech/tags/query.md>), [queue](<https://devfeed.tech/tags/queue.md>), [redis](<https://devfeed.tech/tags/redis.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [request](<https://devfeed.tech/tags/request.md>), [rest](<https://devfeed.tech/tags/rest.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [site-reliability-engineer](<https://devfeed.tech/tags/site-reliability-engineer.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [sre](<https://devfeed.tech/tags/sre.md>), [traffic](<https://devfeed.tech/tags/traffic.md>)

### AI overview

Whatnot replaced its concurrency limiter's precise, synchronous Redis counting after the old design remained a bottleneck even across nine primaries. The new distributed semaphore tolerates slightly stale counts: pods check current and previous window counters on replicas, accumulate request deltas locally, and batch INCRBY updates to primaries every 100 ms. Window keys can land on different shards, spreading hot traffic; Redis LRU eviction replaces per-request TTL updates. The limiter still throttles slow request types, returning 503 when a type exceeds its cap and 429 when an IP does. It can miss very short requests that start and finish between syncs, a tradeoff the team accepted because slow requests are the overload concern. In pre-Black-Friday 2025 load tests, the hottest primary across eight shards stayed below 50% CPU; ahead of Black Friday 2026, the semaphore handled four times the traffic that would overwhelm the old limiter, with headroom. Limits remain static; an adaptive controller is planned.

### Source excerpt

Marcin Mierzejewski | Reliability Platform (Poland) The concurrency limiter in front of our main API at Whatnot has one job: protect the backend from overload. As we scaled, though, it became the component most likely to fall over under a traffic surge. We had grown its Redis fleet from one instance to nine, and the hottest instance still peaked at 92% CPU in load tests. We had even configured that instance to send 60% of its traffic elsewhere, but it still stayed the hottest. Keeping track of every request was a losing effort. The way out was to stop counting so precisely. Concurrency limiting in a nutshell In short, we limit concurrency rather than rate. We count in-flight requests per request type (GraphQL operation names or REST endpoints), so a request type that slows down gets throttled harder. Why count requests in flight instead of requests per second? The number in flight is roughly the arrival rate times how long each request takes. If a database query slows from, say, 50 ms to 500 ms, the same traffic keeps ten times as many requests in flight. Each one holds a worker and a database connection while it waits, so the pile-up slows everything else down, which makes requests slower still. A rate limiter doesn't notice, because the request rate hasn't changed. A concurrency limiter does. The slow request type hits its cap on in-flight requests, and the extras get a fast 503 instead of joining the queue. That request type sheds load early, and the rest of the API keeps its workers and connections. A separate per-IP cap stops any one client from taking more than its share. For simplicity, we'll only talk about request-type concurrency limiting, unless stated otherwise. How the limiter became the bottleneck In 2024, my colleague Karol Gil wrote about how we initially built the limiter. That design worked, and a few weeks later it throttled a query storm that nearly tripled request volume. However, it paid for a precise, synchronous count on every request. Each A

## Driver Mode: Jumping levels of abstraction with agents

DevFeed: [Driver Mode: Jumping levels of abstraction with agents](<https://devfeed.tech/articles/driver-mode-jumping-levels-of-abstraction-with-agents-90002.md>)

Original publisher: [Read original article](<https://medium.com/whatnot-engineering/driver-mode-jumping-levels-of-abstraction-with-agents-8073087bd10a?source=rss----162aeca881b0---4>)

Author: Whatnot Engineering

Published: 2026-10-19T15:51:01Z

Content type: article

Language: en

Sources: [Whatnot Engineering](<https://devfeed.tech/sources/whatnot-engineering.md>)

Topics: [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [agent orchestration](<https://devfeed.tech/topics/agent-orchestration.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [background](<https://devfeed.tech/tags/background.md>), [background-work](<https://devfeed.tech/tags/background-work.md>), [context](<https://devfeed.tech/tags/context.md>), [details](<https://devfeed.tech/tags/details.md>), [driver](<https://devfeed.tech/tags/driver.md>), [intervene](<https://devfeed.tech/tags/intervene.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [map](<https://devfeed.tech/tags/map.md>), [model](<https://devfeed.tech/tags/model.md>), [need](<https://devfeed.tech/tags/need.md>), [overview](<https://devfeed.tech/tags/overview.md>), [pi](<https://devfeed.tech/tags/pi.md>), [research](<https://devfeed.tech/tags/research.md>), [results](<https://devfeed.tech/tags/results.md>), [run](<https://devfeed.tech/tags/run.md>), [running](<https://devfeed.tech/tags/running.md>), [setup](<https://devfeed.tech/tags/setup.md>), [signal](<https://devfeed.tech/tags/signal.md>), [status](<https://devfeed.tech/tags/status.md>), [tools](<https://devfeed.tech/tags/tools.md>)

### AI overview

The author's "driver mode" workflow helps one person oversee several agents by keeping a clear status overview and opening detailed sessions when judgment is needed. In Pi, a small default toolset, manual skills, deferred tools, and project settings keep unused instructions out of each session while making capabilities available on demand. Herdr organizes visible sessions and shows whether agents are working, blocked, or done; background agents surface status and results for work that can wait. These signals direct attention but do not verify correctness: the author checks evidence, uses a separate model to audit completion requirements, and still relies on tests and personal review. Separate git worktrees prevent parallel writers from colliding. A cleanup skill proposes what to close, merge, or archive for the author's approval. The account concerns personal coding, research, and data exploration workflows; it explicitly excludes production agents, which need different guardrails.

### Source excerpt

Eli Bressert | Research (US) The hard part of running several agents isn't starting them. It's knowing which one needs your judgment. During an agent-driven research session, one of my background agents turned red in its status line and kept retrying. I checked on it, saw it was stuck, and told the orchestrator to launch a plain agent instead. The work went on. Without that signal, I'd have waited on a result that wasn't coming. I didn't need to follow every step of the background work. I needed a signal that something was wrong, access to the details, and a way to intervene. That's what I mean by driver mode. My setup uses Pi and Herdr. Below, I'll walk through what I load into each session, which work I keep visible, and how I check results and keep the workspace usable. The operating principles apply beyond those tools. This is for people already using agents to write code, do research, or explore data, who want to run several without losing track of the work. Autonomous agents in production need different guardrails and aren't what I'm covering here. Move between the overview and the decision Donald Knuth called it "jumping levels of abstraction": seeing things at many levels of detail, from high levels down to the lowest. With agents, I need both a view of where the work is headed and access to the specific decision that needs my context. Think of driving with a map app. In the city, you zoom in to see the next turn. On a five-hour drive, you zoom out to check the route. You might reroute because you know about road construction the map doesn't. Agents have similar gaps: a product's cultural nuances, a codebase's history, or what a research field already knows. An overview helps me notice where that context might be missing; the detailed view lets me supply it. Two scarce budgets The model's context is limited. Tool definitions and instructions loaded up front take space alongside the task. Research shows that longer inputs can degrade a model's reasoning, even

## All Dashboards Green, All Screens Black

DevFeed: [All Dashboards Green, All Screens Black](<https://devfeed.tech/articles/all-dashboards-green-all-screens-black-26982.md>)

Original publisher: [Read original article](<https://medium.com/whatnot-engineering/all-dashboards-green-all-screens-black-bcdb4a175633?source=rss----162aeca881b0---4>)

Author: Whatnot Engineering

Published: 2026-09-15T16:31:01Z

Content type: article

Language: en

Sources: [Whatnot Engineering](<https://devfeed.tech/sources/whatnot-engineering.md>)

Topics: [observability](<https://devfeed.tech/topics/observability.md>), [digital experience monitoring](<https://devfeed.tech/topics/digital-experience-monitoring.md>), [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [Grafana Cloud Frontend Observability](<https://devfeed.tech/topics/grafana-cloud-frontend-observability.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [black](<https://devfeed.tech/tags/black.md>), [cdn](<https://devfeed.tech/tags/cdn.md>), [client](<https://devfeed.tech/tags/client.md>), [client-side-security](<https://devfeed.tech/tags/client-side-security.md>), [component](<https://devfeed.tech/tags/component.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [e2e](<https://devfeed.tech/tags/e2e.md>), [experience](<https://devfeed.tech/tags/experience.md>), [external](<https://devfeed.tech/tags/external.md>), [green](<https://devfeed.tech/tags/green.md>), [incident](<https://devfeed.tech/tags/incident.md>), [journey](<https://devfeed.tech/tags/journey.md>), [livestream](<https://devfeed.tech/tags/livestream.md>), [observability](<https://devfeed.tech/tags/observability.md>), [party](<https://devfeed.tech/tags/party.md>), [product](<https://devfeed.tech/tags/product.md>), [production](<https://devfeed.tech/tags/production.md>), [site-reliability-engineer](<https://devfeed.tech/tags/site-reliability-engineer.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>)

### AI overview

A live update to a third-party security script caused black video screens for about 3,000 users in the first 30 minutes, while most component dashboards stayed green. End-to-end SLO monitoring detected the broken livestream journey and paged teams within five minutes. The system models each user action as a story: correlated events must show video, auction details, and chat within 60 seconds. A worker checks completed stories each minute and emits success metrics to Datadog. To prevent retries from skewing results, each client contributes its worst result per minute; expected outcomes such as a seller blocking entry are excluded from failures. Client telemetry uses local storage, compression, and retries, but fully offline clients cannot appear in live graphs. Instrumented stories reach about 99.5% SLI, and hundreds of thousands of events per minute are pushing ClickHouse queries toward their limit.

### Source excerpt

Karol Gil | Reliability Platform (Poland) On June 8, 2026, newly loaded web clients began showing black screens instead of livestream video. For our platform, that's a serious problem: it's pretty hard to sell Pokémon cards that no one can see. It turned out that a third-party script we use for client-side security monitoring wasn't bundled with our release, but was rather fetched live from the provider's CDN. When the provider updated the script all new web clients fetched it, and it included an additional URL validation which our video provider didn't pass. The result? Black screens for users of the affected web clients, with most internal dashboards staying green. 3,000 users were impacted in the first 30 minutes of the incident. One system did catch it. Our end-to-end service-level objective (E2E SLO) monitoring was already in production and paged the owning teams within five minutes. Here's what it saw. The real problem Most of our dashboards stayed green because they monitor component-level health: a server, an endpoint, a specific function. These are all useful, but can all be healthy while the actual user experience is completely broken. This problem gets worse the more external dependencies there are, or the more sophisticated an experience you want to deliver. In complex, integrated product experiences like ours, a "small" problem can have an outsize impact on the user experience. Measuring this requires a different approach to observability, namely, to model the user journey across multiple surfaces that must be true for a customer to have a good experience. So how do we measure this in a complex distributed application? Joining a livestream is not one thing Joining a livestream sounds like one action, but the user expects at least three things: Video to be playing Auction details to be shown Chat to be visible and up to da Each of those can succeed or fail completely independently of the other two. Our video depends on third-party providers and CDN netwo

## Every animation was one rename away from breaking a live show

DevFeed: [Every animation was one rename away from breaking a live show](<https://devfeed.tech/articles/every-animation-was-one-rename-away-from-breaking-a-live-show-23709.md>)

Original publisher: [Read original article](<https://medium.com/whatnot-engineering/every-animation-was-one-rename-away-from-breaking-a-live-show-a5139b9c6bad?source=rss----162aeca881b0---4>)

Author: Whatnot Engineering

Published: 2026-09-10T16:31:01Z

Content type: tutorial

Language: en

Sources: [Whatnot Engineering](<https://devfeed.tech/sources/whatnot-engineering.md>)

Topics: [iOS](<https://devfeed.tech/topics/ios.md>), [Swift](<https://devfeed.tech/topics/swift.md>), [Code](<https://devfeed.tech/topics/code.md>), [SwiftUI](<https://devfeed.tech/topics/swiftui.md>)

Tags: [animation](<https://devfeed.tech/tags/animation.md>), [code](<https://devfeed.tech/tags/code.md>), [ios-app-development](<https://devfeed.tech/tags/ios-app-development.md>), [software-development](<https://devfeed.tech/tags/software-development.md>), [swift](<https://devfeed.tech/tags/swift.md>), [swiftui](<https://devfeed.tech/tags/swiftui.md>)

### AI overview

This engineering post explains how a designer's property rename could silently break a production iOS animation because Swift code referenced asset properties through string literals. It describes replacing hand-written bindings with generated animation binding code from asset files so the compiler can enforce the contract and reduce reliance on tribal knowledge.

### Source excerpt

Seth Sandler There's a bug we never shipped, and it kept me up anyway. Here's how it goes: a designer renames a property from purchaseCount to purchase_count inside an animation file. Every place our Swift code references that property is a string literal. The build stays green. Code review passes; nothing in the diff even mentions animations. And the celebration animation at the heart of our live auctions silently stops firing, in production, in front of a live audience. Nothing stood between us and that outcome except everyone remembering to be careful. And the strings were only the most visible tripwire. Driving our animation runtime correctly also meant knowing things the compiler never checks: call order, load timing, which properties were designer-internal and untouchable, how not to leak memory. Every engineer touching an animation had to know all of it, and every new animation was another chance to get it wrong. If you've done animation work on iOS, some version of this unease is familiar. This post is the story of how we defused it: by generating the entire animation binding layer from the asset files themselves, so the compiler enforces the contract and the generated code encodes the nuance. The methodology isn't specific to our stack, our vendor, or our app. If your code talks to any designer-owned asset through string literals and tribal knowledge, you can build this too. Why hand-rolled animation code collapses Our first instinct, like every iOS team's, was to build the animations ourselves. SwiftUI has withAnimation. Core Animation has CAKeyframeAnimation. We'd shipped complex UI before. How hard could it be? The animation we were building wasn't a loop. It was state-driven: it needed a user's avatar, their name, a running count that ramps up the visuals, and different behavior depending on who's watching. Each input changes the intro, the escalation, the outro. In native Swift, that decomposed into: State explosion. A @State for the image, a @Binding

## Whatnot at Snowflake Summit 2026

DevFeed: [Whatnot at Snowflake Summit 2026](<https://devfeed.tech/articles/whatnot-at-snowflake-summit-2026-23716.md>)

Original publisher: [Read original article](<https://medium.com/whatnot-engineering/whatnot-at-snowflake-summit-2026-a18a855f529d?source=rss----162aeca881b0---4>)

Author: Whatnot Engineering

Published: 2026-08-12T17:54:54Z

Content type: article

Language: en

Sources: [Whatnot Engineering](<https://devfeed.tech/sources/whatnot-engineering.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [dataset](<https://devfeed.tech/topics/dataset.md>), [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [AI-Ready Data](<https://devfeed.tech/topics/ai-ready-data.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [developer-productivity](<https://devfeed.tech/topics/developer-productivity.md>), [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [auction](<https://devfeed.tech/tags/auction.md>), [business](<https://devfeed.tech/tags/business.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [chat](<https://devfeed.tech/tags/chat.md>), [customer-support](<https://devfeed.tech/tags/customer-support.md>), [data](<https://devfeed.tech/tags/data.md>), [data-platform](<https://devfeed.tech/tags/data-platform.md>), [data-products](<https://devfeed.tech/tags/data-products.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [developer](<https://devfeed.tech/tags/developer.md>), [developer-velocity](<https://devfeed.tech/tags/developer-velocity.md>), [developers](<https://devfeed.tech/tags/developers.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [employees](<https://devfeed.tech/tags/employees.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [snowflake-summit](<https://devfeed.tech/tags/snowflake-summit.md>), [summit](<https://devfeed.tech/tags/summit.md>)

### AI overview

Whatnot says it is using Snowflake Cortex AI and Semantic Views, grounded in governed business definitions and trusted metadata, so builders can find datasets and metrics and investigate issues with natural-language questions. The excerpt reports weekly AI use among full-time employees and broader adoption figures, including over 80% of 1,000+ employees within 90 days, while describing teams using conversational AI for trend analysis, messy-field matching, and seller churn models. For observability, Whatnot replaced delayed utilization logs and costly, frequent diagnostic scans with Snowflake event tables, which it says improved event ingestion speed tenfold. Data products now receive default freshness and error-rate alerts routed to owners through CI-enforced metadata; developers can add custom quality checks and asset-level SLAs. The stated next steps are a company-wide platform-health view and faster incident repair.

### Source excerpt

Earlier this summer, several members of the Whatnot Data Platform team joined Snowflake on stage at Snowflake Summit to share how we're building the data platform behind Whatnot's live marketplace. This post covers what we talked about, why we built it, and future direction. Caption: The Whatnot team were everywhere at the Summit! Snowflake recently published a recap of our sessions, which you can read here: Observability at Scale: Whatnot at Snowflake Summit. Data at Whatnot Every live auction, bid, chat message, purchase, moderation action and recommendation produces data. That translates into billions of events each day powering recommendations, seller analytics, and customer support. For us, data isn't just for internal analysis. We use it to power the product while millions of people are buying and selling in real time. That creates an unique set of engineering challenges. As Whatnot has grown, we've had to evolve our platform to support thousands of datasets, hundreds of developers, and an increasingly diverse set of workloads while maintaining reliability, governance, and developer velocity. At Summit, we talked about the investments we have made to get engineers to vetted data faster Accelerating Builders with AI The Whatnot data platform is a living system. Schemas evolve, pipelines are deployed continuously, business definitions change, and new data products are published every day. Keeping up with that pace of change has become one of the biggest challenges for developers. In our first session, we shared how we're using Snowflake Cortex AI and Semantic Views to make that knowledge more accessible. By grounding AI in governed business definitions and trusted metadata (see this earlier post for more details), builders can ask natural language questions about their data, quickly identify the right datasets and metrics, and investigate issues without manually searching documentation or relying on the Data team. Whatnot's full-time employees now use AI weekly,

## The Soufflé Problem: One Regex, 30% Fewer Tokens

DevFeed: [The Soufflé Problem: One Regex, 30% Fewer Tokens](<https://devfeed.tech/articles/the-souffle-problem-one-regex-30-fewer-tokens-23715.md>)

Original publisher: [Read original article](<https://medium.com/whatnot-engineering/the-souffl%C3%A9-problem-one-regex-30-fewer-tokens-5026e6841cf4?source=rss----162aeca881b0---4>)

Author: Bartek Lipinski

Published: 2026-06-23T16:01:03Z

Content type: article

Language: en

Sources: [Whatnot Engineering](<https://devfeed.tech/sources/whatnot-engineering.md>)

Topics: [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [Tool](<https://devfeed.tech/topics/tool.md>), [Terminal](<https://devfeed.tech/topics/terminal.md>)

Tags: [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-coding-agents](<https://devfeed.tech/tags/ai-coding-agents.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [budget](<https://devfeed.tech/tags/budget.md>), [build](<https://devfeed.tech/tags/build.md>), [command](<https://devfeed.tech/tags/command.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [compute](<https://devfeed.tech/tags/compute.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cost](<https://devfeed.tech/tags/cost.md>), [error](<https://devfeed.tech/tags/error.md>), [filter](<https://devfeed.tech/tags/filter.md>), [llms](<https://devfeed.tech/tags/llms.md>), [logs](<https://devfeed.tech/tags/logs.md>), [noise](<https://devfeed.tech/tags/noise.md>), [recipe](<https://devfeed.tech/tags/recipe.md>), [regex](<https://devfeed.tech/tags/regex.md>), [token-efficiency](<https://devfeed.tech/tags/token-efficiency.md>), [tokens](<https://devfeed.tech/tags/tokens.md>)

### AI overview

A documented output filter can reduce coding agents' repeated processing of noisy build logs and curb unproductive follow-up commands. The article's Android/Gradle example pipes combined output through one regex that retains compiler errors, failures, test results, exceptions, and final build status; its patterns were added to address specific confusions such as mistaking nonexistent tasks for code errors. Across 60-plus runs on two constructed Android fix tasks, filtered and unfiltered agents reached the same outcomes, while transcript size fell about 20% on a simpler task and 30% on a cascade failure. The evidence is limited to those seeded tasks, not observed production sessions. The authors recommend documenting when to use the filter, keeping one shared version, updating it as the environment changes, and telling agents to apply it unchanged; if it misses something, run an unfiltered command separately.

### Source excerpt

Our AI agent was burning tokens on every turn by processing build noise that provided no signal. Context windows clogged with irrelevant logs, forcing the agent to waste cycles and thrash through unnecessary tool calls. We solved this with one line of regex; a filter that cut token consumption by up to 30% and, more importantly, stopped the agent from flailing. It's not just a cost-saving trick; it makes the agent's behavior more predictable, spending its budget on the fix instead of the hunt. To grok this, imagine a chef tasked with making a soufflé with ingredients but no recipe. An expert chef, using school-honed skills, might succeed immediately or through a few refined attempts. Success is inevitable because they possess the competence to iterate. A mediocre chef eventually succeeds too, but wastes far more food in the process. AI coding agents operate similarly when facing unknowns. They iterate, try, and fail before reaching an answer. They are capable enough to succeed, but they waste significant compute and tokens during the process. The solution isn't rocket science, but just really good filtering that is specific to tasks that can be token-heavy. Here's an example of that below. For the rest of the post we will do a deep dive into what we did, how it works, and you can ultimately use the recipe for your coding agents. The asymmetry Coding agents with capable LLMs are good at knowing which tool to call. They pick the right command as they work towards their objective. But they are bad at predicting what the output will look like and where in that output to focus. The agent fires off a build command. Hundreds of lines of build output land in its context window. The relevant compiler error is buried somewhere in the middle. Reading through a log is second nature to you; you scan the text without a second thought. You scroll. Your eye snags on red text, stack traces, the word FAILED. You find what matters in seconds, and the rest of the output costs you nothi

## Building, and Whatnot

DevFeed: [Building, and Whatnot](<https://devfeed.tech/articles/building-and-whatnot-23708.md>)

Original publisher: [Read original article](<https://medium.com/whatnot-engineering/building-and-whatnot-772c33c57f63?source=rss----162aeca881b0---4>)

Author: Whatnot Engineering

Published: 2026-05-14T16:01:00Z

Content type: opinion

Language: en

Sources: [Whatnot Engineering](<https://devfeed.tech/sources/whatnot-engineering.md>)

Topics: [Product Management](<https://devfeed.tech/topics/product-management.md>), [Business, Industry, Usability, User Experience, User Research, Web Strategy](<https://devfeed.tech/topics/business-industry-usability-user-experience-user-research-web-strategy.md>), [User Experience](<https://devfeed.tech/topics/user-experience.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [building](<https://devfeed.tech/tags/building.md>), [building-products](<https://devfeed.tech/tags/building-products.md>), [customer](<https://devfeed.tech/tags/customer.md>), [ecommerce](<https://devfeed.tech/tags/ecommerce.md>), [how-we-build](<https://devfeed.tech/tags/how-we-build.md>), [leadership](<https://devfeed.tech/tags/leadership.md>), [process](<https://devfeed.tech/tags/process.md>), [product](<https://devfeed.tech/tags/product.md>), [product-management](<https://devfeed.tech/tags/product-management.md>), [products](<https://devfeed.tech/tags/products.md>), [scale](<https://devfeed.tech/tags/scale.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>)

### AI overview

Whatnot describes a small, senior product team organized around customer problems, with PMs and leaders expected to do hands-on work and employees required to sell, buy, and handle customer experience tickets. Its approach favors broad business context, direct customer understanding, and shipping without procedural gatekeeping. The building process starts with a valuable user and business problem, combines qualitative and quantitative research, works across org boundaries, and seeks the simplest solution. Teams test with the smallest useful audience, then iterate rapidly and launch independently viable components. The document says product review is the sole development gate, while emphasizing strategic alignment, early risk visibility, and autonomy in implementation. It argues that AI can accelerate product work, but useful leverage depends on people still knowing how to do that work. The excerpt presents these as Whatnot's stated practices and rationale; it does not provide independent evidence for its impact claims.

### Source excerpt

Tom Verilli | Chief Product Officer In the last two years, 31,832 people applied to be a Product Manager at Whatnot. We hired one. You're twice as likely to hit a hole in one as you are to get a job by simply applying. That's not a process failure. I've been building products -- and product teams -- for over a decade, and one of the biggest factors in deciding to come to Whatnot ~3 years ago was the very deliberate product culture. No one knows what it means to be a PM in the world of AI, but everything I see says the industry is moving towards us and how we build here -- because no tool will make you useful if you aren't doing the right job. First: we have to acknowledge that the average PM is deeply average. The product function emerged in response to scale -- engineering teams got too big for CEOs or GMs to manage directly, so a business <> tech conduit was needed. Over time, we lazily generalized the role to "every time you hire an Engineering Manager, you hire a PM". But where an Eng Director managed 30-40 people through their EMs, a PM Director just managed five. Incentives govern the world, so those directors' jobs became "justify growing my eng partners headcount" so they could, in turn, grow theirs to become a VP. Slowly, the role of junior PMs shifted from "CEOs of the product" to "babysitters of buttons" and product-minded engineers to infantilized order takers. Then COVID hit, and the industry hired a mind-boggling 500,000 new software engineers in just four years, and ~80,000 new PMs were minted to match. That's 80,000 PMs buried within gargantuan teams at FAANG, far from any customer, 50 layers from the zoom where it happens, taught paint-by-number PM'ing at a product school, in an era of unearned engagement growth where seemingly anything worked. The likelihood of someone emerging from that with great product instincts, experience, and grit actually feels less likely than hitting a hole in one. Second: we made our best, worse. When your job is supervising

## How Whatnot's Hourly ML Feature Pipeline Safely Handled a Failed Update

DevFeed: [How Whatnot's Hourly ML Feature Pipeline Safely Handled a Failed Update](<https://devfeed.tech/articles/the-ml-feature-pipeline-that-got-slower-and-no-one-noticed-23713.md>)

Original publisher: [Read original article](<https://medium.com/whatnot-engineering/the-ml-feature-pipeline-that-got-slower-and-no-one-noticed-8e90c224eae3?source=rss----162aeca881b0---4>)

Author: Whatnot Engineering

Published: 2026-05-07T16:01:01Z

Content type: article

Language: en

Sources: [Whatnot Engineering](<https://devfeed.tech/sources/whatnot-engineering.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [dataset](<https://devfeed.tech/topics/dataset.md>), [data](<https://devfeed.tech/topics/data.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [Process](<https://devfeed.tech/topics/process.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [feature](<https://devfeed.tech/tags/feature.md>), [feature-engineering](<https://devfeed.tech/tags/feature-engineering.md>), [feature-store](<https://devfeed.tech/tags/feature-store.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [ml](<https://devfeed.tech/tags/ml.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [recommendation-system](<https://devfeed.tech/tags/recommendation-system.md>), [recommendations](<https://devfeed.tech/tags/recommendations.md>), [run](<https://devfeed.tech/tags/run.md>)

### AI overview

Whatnot describes how its hourly ML feature pipeline handled a bad pull request that caused a feature group to produce zero rows. The online feature store served last-known-good features for up to two days, anomaly monitors detected the row-count drop, and the team reverted the change before the freshness gap became significant. The post shares lessons about safe degradation and operating frequently updated recommendation features.

### Source excerpt

David Kawashima, Miguel Fernandez-Montes Cuberta, Ford Bohrmann | Discovery A bad PR went out just as our hourly feature pipeline was about to run. The pipeline was responsible for recomputing fresh user behavior signals used in livestream ranking on the home feed. These features capture recent user intent: watches, bids, orders aggregated across users, sellers, and categories. A missed feature update means that the system stops responding to user signals immediately. This PR release caused one of our feature groups to suddenly produce zero rows ... but nothing broke! Even though a critical feature group for the model stopped updating, the online feature store continued serving the last-known-good features, buffered by a 2-day TTL and nothing broke explicitly. Our anomaly monitors went off immediately, flagging the drop in row counts. The team identified the bad change and reverted it before the freshness gap grew large enough to matter. No corrupted data reached production. The system absorbed the failure and degraded safely. This graceful degradation didn't happen by accident; it was the result of hardened processes from years of lived experience, seeing the way these systems can degrade in obvious or subtle ways. In this post, we share some of those lessons learned. The system: hourly feature pipeline Whatnot is a highly dynamic marketplace where buyers' preferences change at a moment's notice. A buyer might research a laptop, join a football-card break a few hours later, and buy chocolate before logging off for the night. Daily feature updates capture long-term preferences but miss intra-day shifts. To keep up with these rapidly changing buyer interests (in 2025) we moved from daily to hourly updates for features that power our ML feed ranking models. Fresher features are a critical component for capturing the live nature of the Whatnot marketplace, where the inventory available to purchase changes in real time. Moving from daily to hourly updates was a meaningful

## The model is the easy part: Building the LLM Platform at Whatnot

DevFeed: [The model is the easy part: Building the LLM Platform at Whatnot](<https://devfeed.tech/articles/the-model-is-the-easy-part-building-the-llm-platform-at-whatnot-23714.md>)

Original publisher: [Read original article](<https://medium.com/whatnot-engineering/the-model-is-the-easy-part-building-the-llm-platform-at-whatnot-ec8730fa9bdf?source=rss----162aeca881b0---4>)

Author: Whatnot Engineering

Published: 2026-04-14T15:01:05Z

Content type: article

Language: en

Sources: [Whatnot Engineering](<https://devfeed.tech/sources/whatnot-engineering.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [function calling](<https://devfeed.tech/topics/function-calling.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Developer Tools](<https://devfeed.tech/topics/developer-tools.md>)

Tags: [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [api](<https://devfeed.tech/tags/api.md>), [diagram](<https://devfeed.tech/tags/diagram.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [foundations](<https://devfeed.tech/tags/foundations.md>), [introduction](<https://devfeed.tech/tags/introduction.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-evaluation](<https://devfeed.tech/tags/llm-evaluation.md>), [machine](<https://devfeed.tech/tags/machine.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [ml](<https://devfeed.tech/tags/ml.md>), [operational](<https://devfeed.tech/tags/operational.md>), [platform](<https://devfeed.tech/tags/platform.md>), [platform-engineering](<https://devfeed.tech/tags/platform-engineering.md>), [product](<https://devfeed.tech/tags/product.md>), [production](<https://devfeed.tech/tags/production.md>), [prompt-engineering](<https://devfeed.tech/tags/prompt-engineering.md>), [quality](<https://devfeed.tech/tags/quality.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [small](<https://devfeed.tech/tags/small.md>), [teams](<https://devfeed.tech/tags/teams.md>), [thing](<https://devfeed.tech/tags/thing.md>), [trust](<https://devfeed.tech/tags/trust.md>), [velocity](<https://devfeed.tech/tags/velocity.md>)

### AI overview

Whatnot's LLM platform treats the model call as the easy part; its main challenge is making LLM workflows reliable, improvable, and trustworthy. It builds on existing data, logging, analytics, and internal tooling foundations, aided by a consolidated codebase. To speed iteration, prompt experiments are self-serve within the company's existing experimentation guardrails; exposures count only when prompt variants produce different outputs, which the article says makes iteration 10x+ faster. A shared tool registry lets engineers define and register Python tools for prompt authors to inspect and attach. For trust, deterministic checks cover mechanical requirements, while calibrated LLM judges score open-ended outputs; production agreement monitoring and mining unusual or valuable cases for human review help detect degradation and refresh evaluation sets. Reliability features include multi-provider support, defaulting toward fallback behavior, observability, caching, rate limiting, and guardrails. The article describes these as platform investments, without providing service-level metrics.

### Source excerpt

Stas Sajin, Faithful Alabi, Peiyun Zhang, Peicheng Yu | AI Platform Introduction A decade ago, one of the more useful ways to explain machine learning systems was with the diagram below: the ML model itself was a tiny box in the middle, surrounded by everything else you actually had to build to make it work in production. Figure 1: Complexity surfaced by ML systems. Source. The same thing is happening with LLMs now. Making the API call is the small box, maybe even smaller than the one in that original diagram. Calling the model is the easy part. The hard part is everything around it: there is less stable ground truth, inputs are harder to constrain, outputs are non-deterministic, and the system is much easier for users to push in unintended directions. The hard part is giving teams the ability to iterate fast, trust that the system is working, and know that it is getting better. From our perspective, this is what an LLM platform actually has to solve for. It has to be reliable enough to support real product and operational workflows. It has to enable velocity, because many of the highest-leverage improvements are small changes that need to move quickly. And it has to create trust, so teams can understand output quality, catch regressions, and ship with confidence. These pillars reinforce each other. Reliability makes teams willing to depend on the platform in production. That production usage creates the data and feedback loops needed to build trust. And trust, in turn, makes it much easier for teams to move with velocity, because they can tell whether a change actually helped. The rest of this post is about those three strategic pillars and the concrete actions behind each of them that allowed us to build the LLM Platform at Whatnot. Figure 2: The LLM platform is organized around three self-reinforcing pillars: velocity, reliability, and trust, each supported by a distinct set of platform enablers.We built on foundations that were already there A big reason we were

## Scaling Whatnot: Behind the Largest Live Shopping Stream in US History

DevFeed: [Scaling Whatnot: Behind the Largest Live Shopping Stream in US History](<https://devfeed.tech/articles/scaling-whatnot-behind-the-largest-live-shopping-stream-in-us-history-23712.md>)

Original publisher: [Read original article](<https://medium.com/whatnot-engineering/scaling-whatnot-behind-the-largest-live-shopping-stream-in-us-history-040a458f538c?source=rss----162aeca881b0---4>)

Author: Whatnot Engineering

Published: 2026-02-24T14:33:16Z

Content type: article

Language: en

Sources: [Whatnot Engineering](<https://devfeed.tech/sources/whatnot-engineering.md>)

Topics: [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [Cloud Architecture](<https://devfeed.tech/topics/cloud-architecture.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [site-reliability-engineering](<https://devfeed.tech/topics/site-reliability-engineering.md>), [Chaos Engineering](<https://devfeed.tech/topics/chaos-engineering.md>), [Load Balancing](<https://devfeed.tech/topics/load-balancing.md>), [Cloudflare](<https://devfeed.tech/topics/cloudflare.md>)

Tags: [admission](<https://devfeed.tech/tags/admission.md>), [cloud-computing](<https://devfeed.tech/tags/cloud-computing.md>), [connection-pooling](<https://devfeed.tech/tags/connection-pooling.md>), [devops](<https://devfeed.tech/tags/devops.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [live](<https://devfeed.tech/tags/live.md>), [load-testing](<https://devfeed.tech/tags/load-testing.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [shopping](<https://devfeed.tech/tags/shopping.md>), [site-reliability-engineer](<https://devfeed.tech/tags/site-reliability-engineer.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [stream](<https://devfeed.tech/tags/stream.md>), [technology](<https://devfeed.tech/tags/technology.md>)

### AI overview

Whatnot prepared for a target of 1 million viewers on one stream and 1.35 million across the platform, validating production systems at 1.46 million total users and roughly 970,000 in one stream. Its Client Admission Service (Go) capped sessions while protecting people already watching or buying; Envoy proxies pooled Redis and OpenSearch connections, while a custom Kafka proxy reduced broker connection pressure. A cached Doomsday Feed served more than 708,000 responses during the event, and Agora delivered video to 582,225 viewers in one RTC channel with time to first frame under one second. Weekly production load tests exposed bottlenecks and prompted durable changes, including a separate analytical read replica cluster. The event ran without major incidents, though Live Service timeouts, DynamoDB throttling, and profile traffic surges led to earlier rate limiting and temporary reporting interruption. Whatnot says future tests will model more sudden bursts, since gradual ramps missed the speed of audience actions.

### Source excerpt

On February 8, 2026, over a half million viewers tuned in to watch MrBeast give away 1 million dollars in prizes on Whatnot. On Big Game Sunday 2026, MrBeast went live on Whatnot for a giveaway show that would become the largest live shopping event in US history. At peak, 583,000 concurrent viewers were watching a single show on our platform. Over 555k people entered a single giveaway. We drove hundreds of thousands of new signups in 24 hours. If any one of a dozen systems buckled, it would have happened live on camera. We pulled it off with zero major incidents. But that outcome was never guaranteed. It took months of preparation, 60+ engineers across every major engineering org, and some of the most significant infrastructure investments we've ever made. In this post, we'll walk through the biggest technical challenges we faced and how we solved them, not with throwaway scaffolding, but with durable platform improvements that raise our scalability ceiling for every seller and buyer on the platform. We'll cover the work in three parts. First, the key architectural investments we made to handle this scale: admission control, connection pooling, feed resilience, and video infrastructure. Then, how we validated it all through progressive production load testing. Finally, what happened on event day, what we learned, and what we're carrying forward. Setting the Stage If you've followed our blog, you might remember our Post Malone "Post-Poned" post from 2022 or our three-part series on preparing for the 2024 Big Game. Each of those events pushed us to improve, and each one revealed new limits. As our community has grown, scaling our infrastructure to match has been a consistent priority. The MrBeast event was on a different order of magnitude entirely, but it accelerated work that was already underway. Our target was to support 1 million concurrent viewers on a single stream and 1.35 million across the platform. To put that in perspective, our previous largest event had

## Lessons learned from scaling data scientists with AI

DevFeed: [Lessons learned from scaling data scientists with AI](<https://devfeed.tech/articles/lessons-learned-from-scaling-data-scientists-with-ai-23711.md>)

Original publisher: [Read original article](<https://medium.com/whatnot-engineering/lessons-learned-from-scaling-data-scientists-with-ai-e7aa7b3235b4?source=rss----162aeca881b0---4>)

Author: Whatnot Engineering

Published: 2026-01-29T18:59:15Z

Content type: article

Language: en

Sources: [Whatnot Engineering](<https://devfeed.tech/sources/whatnot-engineering.md>)

Topics: [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [Data Science](<https://devfeed.tech/topics/data-science.md>), [dataset](<https://devfeed.tech/topics/dataset.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>)

Tags: [2025](<https://devfeed.tech/tags/2025.md>), [adoption](<https://devfeed.tech/tags/adoption.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-systems](<https://devfeed.tech/tags/ai-systems.md>), [bot](<https://devfeed.tech/tags/bot.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [consumer](<https://devfeed.tech/tags/consumer.md>), [conversational-ai](<https://devfeed.tech/tags/conversational-ai.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [data-scientists](<https://devfeed.tech/tags/data-scientists.md>), [data-team](<https://devfeed.tech/tags/data-team.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [design](<https://devfeed.tech/tags/design.md>), [employees](<https://devfeed.tech/tags/employees.md>), [experts](<https://devfeed.tech/tags/experts.md>), [flexibility](<https://devfeed.tech/tags/flexibility.md>), [flexible](<https://devfeed.tech/tags/flexible.md>), [interfaces](<https://devfeed.tech/tags/interfaces.md>), [launch](<https://devfeed.tech/tags/launch.md>), [livestream](<https://devfeed.tech/tags/livestream.md>), [semantic-view](<https://devfeed.tech/tags/semantic-view.md>)

### AI overview

Whatnot's LLM-powered Slack bot answered more than 10,000 employee data questions, usually within tens of seconds, and redirected them from the data team's queue. To limit confident errors, the team restricted it to SQL support and data scientist-endorsed metrics in Snowflake semantic views, accepting that about 30% of questions would go unanswered; public questions also let experts review responses and identify missing context. The authors found that SQL generation needs more than column documentation: semantic views specify relationships and metric definitions, while Cortex search services index record values to improve matching. Broader use exposed unresolved modeling problems, including slow queries, ambiguous source selection, and conflicting metric definitions. They integrated semantic view management with dbt and argue these assets need production practices such as version control, testing, monitoring, and ownership. Their conclusion is that AI gives data scientists more capacity while making reliable data modeling and context definition more essential; different use cases may need different levels of access.

### Source excerpt

Alice Leach, Anthony Sedarous, Evan Hou, Faithful Alabi, Stephen Bailey, Tanisha Kore | Data "Can you check my SQL?" "Which table has the livestream data?" "How do I calculate GMV again?" At Whatnot, our data scientists were drowning in these sorts of questions. At the same time, leaders asking them might wait hours for answers that should take seconds. In 2025, we shipped an LLM-powered Slack bot to break this tension. The results were immediate: the help channel converted from "ask a data scientist" to "ask the bot" overnight. Since launch, employees have had over ten thousand data questions answered by AI systems, with most answered in tens of seconds and all deflected from the data team's queue. Widespread adoption (and proliferation outside of Slack) has revealed limitations that have forced us to rethink how we architect data access for AI systems. Sample conversation from the first verison of our Slackbot. Below, we share a few of the key lessons learned and how we plan to make our team even faster in the coming year. Lesson 1: Flexibility vs. trustworthiness is the key design tradeoff Data experts are valued because they are both trustworthy and flexible: ask them a question, and you can get back not just an answer but their confidence in it as well. LLMs, on the other hand, are famous for being confidently clueless. The first question facing a team employing them for data science work is, "How risky is a wrong answer?" It's not a new question: every data product provides consumers a degree of flexibility, whether that's almost none (public metrics) or a significant amount (raw or unstructured datasets). What's new is that, with AI interfaces, teams can control the consumer's full aperture. Many teams default to maximum flexibility by giving the LLM access to everything and letting it figure it out. If the goal is to replace experts, this is backwards. We went the opposite direction: locked down access to pre-vetted datasets, accepting that we'd miss around