# Whatnot Engineering Whatnot is a livestream shopping platform and marketplace. We're building the future of social commerce. Our mission is to enable anyone to turn their passion into a business and bring people together through commerce. - Medium This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## How end-to-end SLO monitoring detected a livestream failure that component dashboards missed DevFeed: [How end-to-end SLO monitoring detected a livestream failure that component dashboards missed]() Original publisher: [Read original article]() Author: Whatnot Engineering Published: 2026-09-15T16:31:01Z Content type: article Language: en Sources: [Whatnot Engineering]() Topics: [dashboards](), [observability](), [incident](), [Monitoring](), [Web](), [client](), [Network]() Tags: [dashboards](), [distributed-systems](), [incident](), [monitoring](), [observability](), [security](), [server](), [site-reliability-engineer](), [software-engineering](), [web]() ### AI overview The article examines a June 8, 2026 incident in which a third-party client-side security script fetched from a provider CDN added URL validation that the video provider did not pass, causing black screens for newly loaded web clients. Most component-level dashboards remained green, while end-to-end service-level objective monitoring detected the broken livestream experience and paged the owning teams within five minutes. ### Source excerpt Karol Gil | Reliability Platform (Poland) On June 8, 2026, newly loaded web clients began showing black screens instead of livestream video. For our platform, that's a serious problem: it's pretty hard to sell Pokémon cards that no one can see. It turned out that a third-party script we use for client-side security monitoring wasn't bundled with our release, but was rather fetched live from the provider's CDN. When the provider updated the script all new web clients fetched it, and it included an additional URL validation which our video provider didn't pass. The result? Black screens for users of the affected web clients, with most internal dashboards staying green. 3,000 users were impacted in the first 30 minutes of the incident. One system did catch it. Our end-to-end service-level objective (E2E SLO) monitoring was already in production and paged the owning teams within five minutes. Here's what it saw. The real problem Most of our dashboards stayed green because they monitor component-level health: a server, an endpoint, a specific function. These are all useful, but can all be healthy while the actual user experience is completely broken. This problem gets worse the more external dependencies there are, or the more sophisticated an experience you want to deliver. In complex, integrated product experiences like ours, a "small" problem can have an outsize impact on the user experience. Measuring this requires a different approach to observability, namely, to model the user journey across multiple surfaces that must be true for a customer to have a good experience. So how do we measure this in a complex distributed application? Joining a livestream is not one thing Joining a livestream sounds like one action, but the user expects at least three things: Video to be playing Auction details to be shown Chat to be visible and up to da Each of those can succeed or fail completely independently of the other two. Our video depends on third-party providers and CDN netwo ## Every animation was one rename away from breaking a live show DevFeed: [Every animation was one rename away from breaking a live show]() Original publisher: [Read original article]() Author: Whatnot Engineering Published: 2026-09-10T16:31:01Z Content type: tutorial Language: en Sources: [Whatnot Engineering]() Topics: [iOS](), [Swift](), [Code](), [SwiftUI]() Tags: [animation](), [code](), [ios-app-development](), [software-development](), [swift](), [swiftui]() ### AI overview This engineering post explains how a designer's property rename could silently break a production iOS animation because Swift code referenced asset properties through string literals. It describes replacing hand-written bindings with generated animation binding code from asset files so the compiler can enforce the contract and reduce reliance on tribal knowledge. ### Source excerpt Seth Sandler There's a bug we never shipped, and it kept me up anyway. Here's how it goes: a designer renames a property from purchaseCount to purchase_count inside an animation file. Every place our Swift code references that property is a string literal. The build stays green. Code review passes; nothing in the diff even mentions animations. And the celebration animation at the heart of our live auctions silently stops firing, in production, in front of a live audience. Nothing stood between us and that outcome except everyone remembering to be careful. And the strings were only the most visible tripwire. Driving our animation runtime correctly also meant knowing things the compiler never checks: call order, load timing, which properties were designer-internal and untouchable, how not to leak memory. Every engineer touching an animation had to know all of it, and every new animation was another chance to get it wrong. If you've done animation work on iOS, some version of this unease is familiar. This post is the story of how we defused it: by generating the entire animation binding layer from the asset files themselves, so the compiler enforces the contract and the generated code encodes the nuance. The methodology isn't specific to our stack, our vendor, or our app. If your code talks to any designer-owned asset through string literals and tribal knowledge, you can build this too. Why hand-rolled animation code collapses Our first instinct, like every iOS team's, was to build the animations ourselves. SwiftUI has withAnimation. Core Animation has CAKeyframeAnimation. We'd shipped complex UI before. How hard could it be? The animation we were building wasn't a loop. It was state-driven: it needed a user's avatar, their name, a running count that ramps up the visuals, and different behavior depending on who's watching. Each input changes the intro, the escalation, the outro. In native Swift, that decomposed into: State explosion. A @State for the image, a @Binding ## Whatnot at Snowflake Summit 2026 DevFeed: [Whatnot at Snowflake Summit 2026]() Original publisher: [Read original article]() Author: Whatnot Engineering Published: 2026-08-12T17:54:54Z Content type: article Language: en Sources: [Whatnot Engineering]() Topics: [data](), [datasets](), [Artificial Intelligence](), [developer velocity]() Tags: [ai](), [ai-agent](), [analytics](), [data](), [data-science](), [datasets](), [developer-velocity](), [developers](), [machine-learning]() ### AI overview Whatnot's Data Platform team describes the data platform behind its live marketplace and its use of Snowflake Cortex AI and Semantic Views. The approach grounds AI in governed business definitions and trusted metadata so developers can find datasets and metrics and investigate issues using natural-language questions. ### Source excerpt Earlier this summer, several members of the Whatnot Data Platform team joined Snowflake on stage at Snowflake Summit to share how we're building the data platform behind Whatnot's live marketplace. This post covers what we talked about, why we built it, and future direction. Caption: The Whatnot team were everywhere at the Summit! Snowflake recently published a recap of our sessions, which you can read here: Observability at Scale: Whatnot at Snowflake Summit. Data at Whatnot Every live auction, bid, chat message, purchase, moderation action and recommendation produces data. That translates into billions of events each day powering recommendations, seller analytics, and customer support. For us, data isn't just for internal analysis. We use it to power the product while millions of people are buying and selling in real time. That creates an unique set of engineering challenges. As Whatnot has grown, we've had to evolve our platform to support thousands of datasets, hundreds of developers, and an increasingly diverse set of workloads while maintaining reliability, governance, and developer velocity. At Summit, we talked about the investments we have made to get engineers to vetted data faster Accelerating Builders with AI The Whatnot data platform is a living system. Schemas evolve, pipelines are deployed continuously, business definitions change, and new data products are published every day. Keeping up with that pace of change has become one of the biggest challenges for developers. In our first session, we shared how we're using Snowflake Cortex AI and Semantic Views to make that knowledge more accessible. By grounding AI in governed business definitions and trusted metadata (see this earlier post for more details), builders can ask natural language questions about their data, quickly identify the right datasets and metrics, and investigate issues without manually searching documentation or relying on the Data team. Whatnot's full-time employees now use AI weekly, ## Filtering Build Output Reduced AI Coding Agent Token Consumption by Up to 30% DevFeed: [Filtering Build Output Reduced AI Coding Agent Token Consumption by Up to 30%]() Original publisher: [Read original article]() Author: Bartek Lipinski Published: 2026-06-23T16:01:03Z Content type: tutorial Language: en Sources: [Whatnot Engineering]() Topics: [AI-assisted coding](), [AI Agent](), [context window](), [context](), [Tool]() Tags: [agent](), [agentic-ai](), [ai](), [ai-agent](), [ai-coding-agents](), [bash](), [build](), [commands](), [compiler](), [context-window](), [llms](), [token](), [token-efficiency](), [tool]() ### AI overview The article explains how filtering irrelevant build output with a regex can reduce token consumption for AI coding agents by up to 30%. It describes how noisy logs fill the context window, causing repeated costs and unnecessary tool calls, and presents filtering as a way to make agent behavior more predictable. ### Source excerpt Our AI agent was burning tokens on every turn by processing build noise that provided no signal. Context windows clogged with irrelevant logs, forcing the agent to waste cycles and thrash through unnecessary tool calls. We solved this with one line of regex; a filter that cut token consumption by up to 30% and, more importantly, stopped the agent from flailing. It's not just a cost-saving trick; it makes the agent's behavior more predictable, spending its budget on the fix instead of the hunt. To grok this, imagine a chef tasked with making a soufflé with ingredients but no recipe. An expert chef, using school-honed skills, might succeed immediately or through a few refined attempts. Success is inevitable because they possess the competence to iterate. A mediocre chef eventually succeeds too, but wastes far more food in the process. AI coding agents operate similarly when facing unknowns. They iterate, try, and fail before reaching an answer. They are capable enough to succeed, but they waste significant compute and tokens during the process. The solution isn't rocket science, but just really good filtering that is specific to tasks that can be token-heavy. Here's an example of that below. For the rest of the post we will do a deep dive into what we did, how it works, and you can ultimately use the recipe for your coding agents. The asymmetry Coding agents with capable LLMs are good at knowing which tool to call. They pick the right command as they work towards their objective. But they are bad at predicting what the output will look like and where in that output to focus. The agent fires off a build command. Hundreds of lines of build output land in its context window. The relevant compiler error is buried somewhere in the middle. Reading through a log is second nature to you; you scan the text without a second thought. You scroll. Your eye snags on red text, stack traces, the word FAILED. You find what matters in seconds, and the rest of the output costs you nothi ## Whatnot's Product Culture and the Changing Role of Product Managers DevFeed: [Whatnot's Product Culture and the Changing Role of Product Managers]() Original publisher: [Read original article]() Author: Whatnot Engineering Published: 2026-05-14T16:01:00Z Content type: opinion Language: en Sources: [Whatnot Engineering]() Topics: [Job](), [Software]() Tags: [ai](), [ecommerce](), [industry](), [job](), [leadership](), [people](), [product-management](), [software](), [software-engineering]() ### AI overview Tom Verilli discusses Whatnot's deliberate product culture and argues that the product manager role has become diluted by organizational scale, hiring growth, and management incentives. He suggests that succeeding in an AI-driven industry depends on doing the right product work rather than relying on tools alone. ### Source excerpt Tom Verilli | Chief Product Officer In the last two years, 31,832 people applied to be a Product Manager at Whatnot. We hired one. You're twice as likely to hit a hole in one as you are to get a job by simply applying. That's not a process failure. I've been building products -- and product teams -- for over a decade, and one of the biggest factors in deciding to come to Whatnot ~3 years ago was the very deliberate product culture. No one knows what it means to be a PM in the world of AI, but everything I see says the industry is moving towards us and how we build here -- because no tool will make you useful if you aren't doing the right job. First: we have to acknowledge that the average PM is deeply average. The product function emerged in response to scale -- engineering teams got too big for CEOs or GMs to manage directly, so a business <> tech conduit was needed. Over time, we lazily generalized the role to "every time you hire an Engineering Manager, you hire a PM". But where an Eng Director managed 30-40 people through their EMs, a PM Director just managed five. Incentives govern the world, so those directors' jobs became "justify growing my eng partners headcount" so they could, in turn, grow theirs to become a VP. Slowly, the role of junior PMs shifted from "CEOs of the product" to "babysitters of buttons" and product-minded engineers to infantilized order takers. Then COVID hit, and the industry hired a mind-boggling 500,000 new software engineers in just four years, and ~80,000 new PMs were minted to match. That's 80,000 PMs buried within gargantuan teams at FAANG, far from any customer, 50 layers from the zoom where it happens, taught paint-by-number PM'ing at a product school, in an era of unearned engagement growth where seemingly anything worked. The likelihood of someone emerging from that with great product instincts, experience, and grit actually feels less likely than hitting a hole in one. Second: we made our best, worse. When your job is supervising ## How Whatnot's Hourly ML Feature Pipeline Safely Handled a Failed Update DevFeed: [How Whatnot's Hourly ML Feature Pipeline Safely Handled a Failed Update]() Original publisher: [Read original article]() Author: Whatnot Engineering Published: 2026-05-07T16:01:01Z Content type: article Language: en Sources: [Whatnot Engineering]() Topics: [Machine learning](), [recommendation systems](), [datasets](), [data](), [real-time](), [Processes]() Tags: [ai](), [data](), [data-engineering](), [feature](), [feature-engineering](), [feature-store](), [machine-learning](), [ml](), [pipeline](), [ranking](), [real-time](), [recommendation-system](), [recommendations](), [run]() ### AI overview Whatnot describes how its hourly ML feature pipeline handled a bad pull request that caused a feature group to produce zero rows. The online feature store served last-known-good features for up to two days, anomaly monitors detected the row-count drop, and the team reverted the change before the freshness gap became significant. The post shares lessons about safe degradation and operating frequently updated recommendation features. ### Source excerpt David Kawashima, Miguel Fernandez-Montes Cuberta, Ford Bohrmann | Discovery A bad PR went out just as our hourly feature pipeline was about to run. The pipeline was responsible for recomputing fresh user behavior signals used in livestream ranking on the home feed. These features capture recent user intent: watches, bids, orders aggregated across users, sellers, and categories. A missed feature update means that the system stops responding to user signals immediately. This PR release caused one of our feature groups to suddenly produce zero rows ... but nothing broke! Even though a critical feature group for the model stopped updating, the online feature store continued serving the last-known-good features, buffered by a 2-day TTL and nothing broke explicitly. Our anomaly monitors went off immediately, flagging the drop in row counts. The team identified the bad change and reverted it before the freshness gap grew large enough to matter. No corrupted data reached production. The system absorbed the failure and degraded safely. This graceful degradation didn't happen by accident; it was the result of hardened processes from years of lived experience, seeing the way these systems can degrade in obvious or subtle ways. In this post, we share some of those lessons learned. The system: hourly feature pipeline Whatnot is a highly dynamic marketplace where buyers' preferences change at a moment's notice. A buyer might research a laptop, join a football-card break a few hours later, and buy chocolate before logging off for the night. Daily feature updates capture long-term preferences but miss intra-day shifts. To keep up with these rapidly changing buyer interests (in 2025) we moved from daily to hourly updates for features that power our ML feed ranking models. Fresher features are a critical component for capturing the live nature of the Whatnot marketplace, where the inventory available to purchase changes in real time. Moving from daily to hourly updates was a meaningful ## The model is the easy part: Building the LLM Platform at Whatnot DevFeed: [The model is the easy part: Building the LLM Platform at Whatnot]() Original publisher: [Read original article]() Author: Whatnot Engineering Published: 2026-04-14T15:01:05Z Content type: article Language: en Sources: [Whatnot Engineering]() Topics: [AI Platform](), [LLMs](), [AI Development](), [Tooling]() Tags: [ai-agent](), [ai-platform](), [building](), [llm](), [llm-evaluation](), [llms](), [platform](), [platform-engineering](), [production](), [prompt-engineering](), [quality](), [teams](), [trust](), [workflows]() ### AI overview This article explains that building an LLM platform involves much more than calling a model. At Whatnot, the platform is organized around reliability, velocity, and trust, supported by existing data, logging, analytics, integration, and internal tooling foundations. ### Source excerpt Stas Sajin, Faithful Alabi, Peiyun Zhang, Peicheng Yu | AI Platform Introduction A decade ago, one of the more useful ways to explain machine learning systems was with the diagram below: the ML model itself was a tiny box in the middle, surrounded by everything else you actually had to build to make it work in production. Figure 1: Complexity surfaced by ML systems. Source. The same thing is happening with LLMs now. Making the API call is the small box, maybe even smaller than the one in that original diagram. Calling the model is the easy part. The hard part is everything around it: there is less stable ground truth, inputs are harder to constrain, outputs are non-deterministic, and the system is much easier for users to push in unintended directions. The hard part is giving teams the ability to iterate fast, trust that the system is working, and know that it is getting better. From our perspective, this is what an LLM platform actually has to solve for. It has to be reliable enough to support real product and operational workflows. It has to enable velocity, because many of the highest-leverage improvements are small changes that need to move quickly. And it has to create trust, so teams can understand output quality, catch regressions, and ship with confidence. These pillars reinforce each other. Reliability makes teams willing to depend on the platform in production. That production usage creates the data and feedback loops needed to build trust. And trust, in turn, makes it much easier for teams to move with velocity, because they can tell whether a change actually helped. The rest of this post is about those three strategic pillars and the concrete actions behind each of them that allowed us to build the LLM Platform at Whatnot. Figure 2: The LLM platform is organized around three self-reinforcing pillars: velocity, reliability, and trust, each supported by a distinct set of platform enablers.We built on foundations that were already there A big reason we were ## How Whatnot's Taxonomy Keeps Pace with a Marketplace That Won't Sit Still DevFeed: [How Whatnot's Taxonomy Keeps Pace with a Marketplace That Won't Sit Still]() Original publisher: [Read original article]() Author: Whatnot Engineering Published: 2026-03-25T14:22:19Z Content type: article Language: en Sources: [Whatnot Engineering]() Topics: [recommendations](), [bug](), [real-time]() Tags: [2025](), [ai](), [bug](), [ecommerce](), [machine-learning](), [onboarding](), [platform](), [real-time](), [recommendation-system](), [recommendations](), [taxonomy]() ### AI overview Whatnot describes how it redesigned its marketplace taxonomy to change more rapidly across countries and discovery surfaces. The article explains why onboarding, browse, search, and recommendations need different taxonomy treatments, using supply-aware onboarding and the Labubu category as examples. ### Source excerpt Marcus Gee | Discovery The hottest trends don't happen over months, but start suddenly. At Whatnot, our best sellers update their strategies in real time to stay in step with the market. Connecting buyers and sellers in this fast-changing landscape is one of the most important problems Whatnot solves, but in 2025, it was a major challenge. In this post, we're going to walk through how we overhauled the taxonomy system to evolve more rapidly, so our team could help new communities grow. There is No Single Taxonomy Whatnot supports over 200 categories across seven countries, and that number increases weekly. Our taxonomy is a platform primitive that underpins nearly every discovery experience. It shapes onboarding (what categories new users see), browse (how they navigate), search (how results are ranked), and recommendations (what their feed looks like). A bug in taxonomy doesn't break one feature; it degrades the entire discovery surface. That makes every change high-stakes. Here's what we learned the hard way: maintaining a single taxonomy doesn't work. A single taxonomy can't optimally serve onboarding, browse, and search at the same time, because these surfaces have different goals. Let's look at two examples to understand why: When new users join Whatnot, they choose categories they're interested in. That choice shapes the first feed they see. If a user selects a niche category with limited supply, their initial experience may feel sparse or low quality. For this reason, onboarding taxonomies are country-specific and supply-aware. We fold categories into broader parents, reorder them, or exclude them based on local marketplace density. Categories play a critical role in connecting enthusiasts and sellers. When Labubu surged in 2025, it was subcategorized under Toys, and new users struggled to find relevant shows. When we elevated Labubu to the first page of onboarding (a similar treatment given to other big categories like Pokémon Cards), it reduced this frictio ## How Whatnot Built Atlas to Represent and Automate Support SOPs as Code DevFeed: [How Whatnot Built Atlas to Represent and Automate Support SOPs as Code]() Original publisher: [Read original article]() Author: Whatnot Engineering Published: 2026-03-10T15:01:01Z Content type: article Language: en Sources: [Whatnot Engineering]() Topics: [Code](), [Automation](), [Graphs](), [Python]() Tags: [ai](), [ai-agent](), [automation](), [code](), [customer-experience](), [graph](), [graphs](), [python](), [trust]() ### AI overview Whatnot describes Atlas, a workflow execution and automation engine that represents support procedures as directed graphs in code using a small Python DSL. The team first formalized workflows as manual agent-guided steps before adding automation. ### Source excerpt Your SOPs Are Already Code... You Just Can't Run Them Yet Paul Gut | CX & Trust Engineering Every support org has them: thousands of pages of branching, conditional logic scattered across flowcharts, docs, and the heads of your most tenured agents. "If the label's been generated, check the tracking status; if it shows delivered, verify the address on file..." The Standard Operating Procedures (SOPs) grow until they can no longer be managed. And when you're the fastest-growing live commerce platform in history, it all catches up with you even faster. At Whatnot, our agents handle everything from shipping questions on vintage sneaker orders to address changes on trading card shipments. Until recently, they were spending the majority of their time on work that required no human insight, like checking order prices, validating addresses, looking up tracking statuses -- routine data checks interspersed with the actual decision-making. Even worse, our more experienced agents had memorized the SOPs and stopped referencing them altogether, which made propagating even the slightest changes a nightmare. We needed SOPs to behave like code: automatable, deployable, version-controlled. And there's no better way to make something behave like code than to simply turn it into code. So in mid-2025, we built Atlas: our new workflow execution and automation engine. Formalize first, automate later The idea behind Atlas is simple: for every ticket, the engine traverses a directed graph of nodes, eventually reaching a resolution the agent can execute. The graph is represented in code (more on that decision later). For every ticket, the engine traverses a directed graph of nodes, eventually reaching a resolution the agent can execute. We built a small Python DSL to represent these graphs. Each node could be a decision ("Has a label been generated for this order?"), an action ("Generate a shipping label"), or a resolution ("Send the label to the user"). We also made data points -- reusable pieces ## Scaling Whatnot: Behind the Largest Live Shopping Stream in US History DevFeed: [Scaling Whatnot: Behind the Largest Live Shopping Stream in US History]() Original publisher: [Read original article]() Author: Whatnot Engineering Published: 2026-02-24T14:33:16Z Content type: article Language: en Sources: [Whatnot Engineering]() Topics: [Scalability](), [Architecture & Design](), [Platform Engineering](), [SRE](), [Elixir](), [Python]() Tags: [cloud-computing](), [devops](), [elixir](), [infrastructure](), [load-testing](), [python](), [resilience](), [scalability](), [scale](), [site-reliability-engineer](), [software-engineering](), [sre](), [technology]() ### AI overview Whatnot describes how it prepared its platform for a MrBeast giveaway stream that reached 583,000 concurrent viewers and became the largest live shopping event in US history. The article covers architectural investments, progressive production load testing, event-day results, and lessons for future scalability. ### Source excerpt On February 8, 2026, over a half million viewers tuned in to watch MrBeast give away 1 million dollars in prizes on Whatnot. On Big Game Sunday 2026, MrBeast went live on Whatnot for a giveaway show that would become the largest live shopping event in US history. At peak, 583,000 concurrent viewers were watching a single show on our platform. Over 555k people entered a single giveaway. We drove hundreds of thousands of new signups in 24 hours. If any one of a dozen systems buckled, it would have happened live on camera. We pulled it off with zero major incidents. But that outcome was never guaranteed. It took months of preparation, 60+ engineers across every major engineering org, and some of the most significant infrastructure investments we've ever made. In this post, we'll walk through the biggest technical challenges we faced and how we solved them, not with throwaway scaffolding, but with durable platform improvements that raise our scalability ceiling for every seller and buyer on the platform. We'll cover the work in three parts. First, the key architectural investments we made to handle this scale: admission control, connection pooling, feed resilience, and video infrastructure. Then, how we validated it all through progressive production load testing. Finally, what happened on event day, what we learned, and what we're carrying forward. Setting the Stage If you've followed our blog, you might remember our Post Malone "Post-Poned" post from 2022 or our three-part series on preparing for the 2024 Big Game. Each of those events pushed us to improve, and each one revealed new limits. As our community has grown, scaling our infrastructure to match has been a consistent priority. The MrBeast event was on a different order of magnitude entirely, but it accelerated work that was already underway. Our target was to support 1 million concurrent viewers on a single stream and 1.35 million across the platform. To put that in perspective, our previous largest event had ## Lessons learned from scaling data scientists with AI DevFeed: [Lessons learned from scaling data scientists with AI]() Original publisher: [Read original article]() Author: Whatnot Engineering Published: 2026-01-29T18:59:15Z Content type: opinion Language: en Sources: [Whatnot Engineering]() Topics: [data](), [Large Language Model](), [Slack](), [datasets](), [SQL]() Tags: [agent](), [ai](), [conversational-ai](), [data](), [data-engineering](), [data-science](), [datasets](), [llms](), [scope](), [semantic-view](), [slack](), [trust]() ### AI overview Whatnot describes lessons from deploying an LLM-powered Slack bot for data questions. The team limited access to pre-vetted datasets and data scientist-endorsed metrics to reduce incorrect answers and protect trust, while acknowledging that this conservative approach would miss some questions. ### Source excerpt Alice Leach, Anthony Sedarous, Evan Hou, Faithful Alabi, Stephen Bailey, Tanisha Kore | Data "Can you check my SQL?" "Which table has the livestream data?" "How do I calculate GMV again?" At Whatnot, our data scientists were drowning in these sorts of questions. At the same time, leaders asking them might wait hours for answers that should take seconds. In 2025, we shipped an LLM-powered Slack bot to break this tension. The results were immediate: the help channel converted from "ask a data scientist" to "ask the bot" overnight. Since launch, employees have had over ten thousand data questions answered by AI systems, with most answered in tens of seconds and all deflected from the data team's queue. Widespread adoption (and proliferation outside of Slack) has revealed limitations that have forced us to rethink how we architect data access for AI systems. Sample conversation from the first verison of our Slackbot. Below, we share a few of the key lessons learned and how we plan to make our team even faster in the coming year. Lesson 1: Flexibility vs. trustworthiness is the key design tradeoff Data experts are valued because they are both trustworthy and flexible: ask them a question, and you can get back not just an answer but their confidence in it as well. LLMs, on the other hand, are famous for being confidently clueless. The first question facing a team employing them for data science work is, "How risky is a wrong answer?" It's not a new question: every data product provides consumers a degree of flexibility, whether that's almost none (public metrics) or a significant amount (raw or unstructured datasets). What's new is that, with AI interfaces, teams can control the consumer's full aperture. Many teams default to maximum flexibility by giving the LLM access to everything and letting it figure it out. If the goal is to replace experts, this is backwards. We went the opposite direction: locked down access to pre-vetted datasets, accepting that we'd miss around