# CASTLE: training natural language search before the first real query

DevFeed: [CASTLE: training natural language search before the first real query](<https://devfeed.tech/articles/castle-training-natural-language-search-before-the-first-real-query-86238.md>)

Original publisher: [Read original article](<https://medium.com/airbnb-engineering/castle-training-natural-language-search-before-the-first-real-query-3818ef8b4828?source=rss----53c7c27702d5---4>)

Author: Wendy Ran Wei

Published: 2026-10-13T17:01:02Z

Content type: article

Language: en

Sources: [Airbnb](<https://devfeed.tech/sources/the-airbnb-tech-blog-medium.md>)

Topics: [search-engines](<https://devfeed.tech/topics/search-engines.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [airbnb](<https://devfeed.tech/tags/airbnb.md>), [cold-start-problem](<https://devfeed.tech/tags/cold-start-problem.md>), [data](<https://devfeed.tech/tags/data.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [few-shot](<https://devfeed.tech/tags/few-shot.md>), [generate](<https://devfeed.tech/tags/generate.md>), [generation](<https://devfeed.tech/tags/generation.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [llm](<https://devfeed.tech/tags/llm.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [model](<https://devfeed.tech/tags/model.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [relevance](<https://devfeed.tech/tags/relevance.md>), [search](<https://devfeed.tech/tags/search.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [technology](<https://devfeed.tech/tags/technology.md>), [traffic](<https://devfeed.tech/tags/traffic.md>), [train](<https://devfeed.tech/tags/train.md>), [training](<https://devfeed.tech/tags/training.md>), [training-data](<https://devfeed.tech/tags/training-data.md>)

## AI overview

CASTLE tackles cold-start natural language search by using de-identified booking sessions to form positive and negative listing pairs, then prompting an LLM to generate queries that fit the booked listing but not the alternatives. This makes relevance labels part of the construction, avoiding a separate LLM judging step for the core data. Template prompts, about 500 research seed queries, attribute-focused prompts, and a variety mechanism are used to improve query variety and match guest phrasing. For listings without booking history, a Virtual Judge assigns labels; CASTLE-R is intended for settings with some real search traffic. In the reported evaluation, CASTLE achieved 1.01 KL divergence against real query distributions, versus 9.33 for the best baseline, and experts agreed with its labels 91-93% of the time across 200 sampled triplets. These results are reported for the described evaluations.

## Source excerpt

How we addressed the cold-start problem for Airbnb's natural language search by jointly generating synthetic training data -- queries and relevance labels -- from de-identified booking and search signals. By: Wendy Ran Wei, Hao Li, Weiwei Guo, Xiaowei Liu, Xueyin Chen, Chun How Tan, Malay Haldar, Soumyadip Banerjee, Kedar Bellare, Huiji Gao, Sanjeev Katariya, and Stephanie Moyerman Introduction Natural language search lets guests express intent in their own words, such as "romantic cabin near skiing" or "pet-friendly beach house that sleeps eight," rather than navigating filters and categories. We needed to introduce this kind of search for Airbnb, and to have it work well from the start. However, this required training the model before we had actual user queries to train the model on, and a reasonable set of relevance labels (inferred from the queries) to map queries to matching listings. Without prior traffic, both must be synthesized from scratch. This is the cold-start problem for natural-language search. Other organizations have faced similar challenges and have created methods to address them. However, prior approaches address the need for a body of queries and the need for relevance labels separately. Passage-based methods such as InPars [1] prompt a large language model (LLM) to generate queries conditioned on a single document, then rely on additional LLM judgments to assign relevance labels, introducing a second source of error. Promptagator [2] improves query diversity using few-shot examples, but is still limited by the gap between its generation distribution and real user behavior. In this post, we describe Contrastive And Seed-guided Training for natural Language sEarch (CASTLE), our framework for jointly generating queries and producing relevance labels from Airbnb's own de-identified booking data. Our paper describing CASTLE was accepted as a full research paper at the 35th ACM International Conference on Information and Knowledge Management, more wide