# Beyond the model: Engineering AI infra with scientific judgement

DevFeed: [Beyond the model: Engineering AI infra with scientific judgement](<https://devfeed.tech/articles/beyond-the-model-engineering-ai-infra-with-scientific-judgement-26973.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/beyond-the-model-engineering-ai-infra-with-scientific-judgement-371316d43261?source=rss----53c7c27702d5---4>)

Author: AirbnbEng

Published: 2026-09-15T17:06:18Z

Content type: article

Language: en

Sources: [Airbnb](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [AI for customer service automation](<https://devfeed.tech/topics/ai-for-customer-service-automation.md>), [audit](<https://devfeed.tech/topics/audit.md>), [dataset](<https://devfeed.tech/topics/dataset.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-harness](<https://devfeed.tech/tags/agent-harness.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-infra](<https://devfeed.tech/tags/ai-infra.md>), [airbnb](<https://devfeed.tech/tags/airbnb.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [audit](<https://devfeed.tech/tags/audit.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agent](<https://devfeed.tech/tags/coding-agent.md>), [conversations](<https://devfeed.tech/tags/conversations.md>), [customer](<https://devfeed.tech/tags/customer.md>), [customer-service](<https://devfeed.tech/tags/customer-service.md>), [customer-support](<https://devfeed.tech/tags/customer-support.md>), [data](<https://devfeed.tech/tags/data.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [model](<https://devfeed.tech/tags/model.md>), [science](<https://devfeed.tech/tags/science.md>)

## AI overview

Airbnb's Insight Miner packages a repeatable method for exploring unstructured text around an AI agent. It combines extract, embed, and cluster workflows with prompt tuning, hard-example mining, contrastive labeling, and a path from unsupervised exploration to classification. Researchers start from a question in chat; the harness handles scalable analysis while experts inspect ambiguous or strategic data, test groupings, and build qualitative and quantitative understanding. Airbnb reports that investigations supporting expansion of its customer service assistant to new languages and countries went from months to days, with rigor and speed both increasing. A UI also lets nontechnical subject matter experts run scaled analyses. The excerpt says teams use the tool for survey responses, model evaluation, and fraud-pattern analysis, but gives no comparative benchmark or detailed evaluation results.

## Source excerpt

How Airbnb's agent harness transforms unstructured data exploration by encoding scientific methodology into scalable, reproducible, and audit-ready infrastructure. By: Wren Dougherty Ask a coding agent to analyze 100,000 customer support conversations and within minutes you'll have a polished taxonomy, precise prevalence numbers, and an executive-ready summary. What you can't see is the investigation that produced them: the methods it chose, the evidence it weighed, how much to trust it, or whether a second request would agree. All that reaches you is the polish. The model is undeniably intelligent, but intelligence without methodology is not science. LLMs certainly make for confident scientists, but we need them to be responsible ones. Smarter models help, but intelligence has never been the whole of science, in people or in machines. The method is as much the product as the answer. That is the idea behind the agent harness we built for data science: the methodology itself, built as infrastructure around the model. It governs how an AI agent operates, from framing a question to selecting evidence to recording decisions, so results can be reproduced, audited, and challenged, and the method shared, inspected, and built on. The challenge of unstructured data exploration In 2025, Airbnb was preparing to launch an AI customer service assistant. Before it could ship, we needed to understand exactly what kinds of situations it would face in the real world. That included rare events that could be risky for AI to interact with, and involved examining their taxonomy and prevalence to create the datasets that would help us build a more responsible product. The investigative work to do this was rigorous, but the process was deeply artisanal. Months of high-touch iteration went into each investigation, from finding the right data, reviewing samples with experts, and generating representative datasets, and the method was manually curated across notebooks, tables, docs, and indiv