# Booking.com Data Science The Machine learning and Data Science blog of Booking.com - Medium This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## How we selected the next vector database at Booking.com DevFeed: [How we selected the next vector database at Booking.com]() Original publisher: [Read original article]() Author: Başak Tuğçe Eskili Published: 2026-08-11T10:31:50Z Content type: article Language: en Sources: [Booking.com Data Science]() Topics: [Database](), [Retrieval Augmented Generation (RAG)](), [Embeddings](), [Back end](), [opensearch](), [AI search]() Tags: [ai](), [database](), [embeddings](), [featured](), [genai](), [hybrid-search](), [retrieval-augmented-generation](), [retrieval-augmented-generation-rag](), [semantic](), [vector-database]() ### AI overview Booking.com explains why selecting a vector database became an infrastructure decision as embeddings and vector search expanded across its machine learning and GenAI systems. The article describes diverse functional and operational requirements, including hybrid search, multi-vector support, capacity, request rates, metadata filtering, and concurrency, and introduces OpenSearch as the initial choice. ### Source excerpt This work was done in collaboration with Klaus Schaefers. Over the past few years, embeddings and vector search have become an important capability in many of our machine learning and GenAI systems at Booking.com. We initially started with a handful of use cases and experiments, and later this capability has grown into shared infrastructure that powers similarity search, semantic filtering, and retrieval-augmented generation (RAG). We used to treat vector search as a backend implementation detail, but today it directly drives the user experience. The real win isn't only speed but also the context. Expanding the variety of domain data we can retrieve efficiently gives our system the depth of context it needs to deliver accurate, and personalized experiences across the platform. This makes selecting the underlying vector database an infrastructure decision similar to choosing a primary datastore or message queue. It has to be predictable and scalable. As more teams started using our vector store, we began seeing highly diverse functional and operational requirements across different use cases. Some teams needed advanced capabilities like hybrid search or multi-vector support, while others demanded larger vector capacities and higher RPS metrics. These architectural needs ultimately brought us to a point where we needed to reassess whether our current setup could support this next phase of growth. Context: how embeddings fit into our stack Embeddings are vectors: fixed-length arrays of numbers produced by a model to represent an item (text, image, etc.). Each vector can be seen as a point in a high-dimensional space, where distance (or similarity) between points approximates semantic relatedness. By searching for the nearest vectors to a query vector, we retrieve items that are semantically "similar". This simple mechanism enables a wide range of use cases for us due its ability to do semantic similarity search. RAG-based use cases are the most well known examples. Ano ## A Recap of the 2026 Experimentation Conference at Booking.com DevFeed: [A Recap of the 2026 Experimentation Conference at Booking.com]() Original publisher: [Read original article]() Author: Mel JI Mueller Published: 2026-07-16T08:18:03Z Content type: article Language: en Sources: [Booking.com Data Science]() Topics: [experiments](), [decision-making](), [Artificial Intelligence](), [Tooling]() Tags: [2025](), [ai](), [analysis](), [data-science](), [events](), [experimentation](), [organizational](), [parallel](), [recap](), [themes]() ### AI overview A recap of Booking.com's 2026 Experimentation Conference, which brought together more than 150 experimentation practitioners from 49 companies. The article summarizes survey findings, conference themes, and sessions on AI-assisted experimentation, experimentation quality and velocity, and organizational culture. ### Source excerpt By Kevin Anderson, Angelica Goetzen, Jorden Lentze, and Melanie Mueller On May 18, 2026, we hosted the third annual Experimentation Conference at Booking.com on our Amsterdam campus. What started in 2024 as an experiment itself -- would large-scale experimentation practitioners come together to learn from each other? -- has grown into an event which brings together over 150 practitioners from 49 companies which run experiments at scale. About one third of attendees came back a second or third time. The room collectively ran 56,000 experiments per year. It's a unique crowd, and that's exactly the point. The day opened with sharing the results of the survey data we collected from the participating companies on the state of experimentation across the room, revealing some interesting findings: most teams operate a centre of excellence model, roughly a third release over 90% of features through controlled experiments, and the top challenges are scaling, coordination, platform tooling, and culture. These shared experiences helped shape the programme. We had three sessions, grouped by the three conference themes: AI and experimentation: AI-assisted analysis, no-code experimentation Quality / velocity tradeoff: High-quality vs high-speed experimentation Experimentation culture: Build organizational buy-in and data-driven decision-making Each session followed the same format: two talks, then a panel discussion on the same topic. We closed with nine parallel breakout groups for deeper conversation. Below is a recap of the key sessions. Read the recap of 2025 | Read the recap of 2024 Session 1: AI and experimentation The conference started off with the hot topic of AI in experimentation. AI is changing how we experiment and how we support experimenters. How Experimentation Protects Decisions in an AI-Written World -- Marcel Toben Marcel Toben, Head of Engineering at Zalando, opened with a provocation he'd recently heard from software engineers in Berlin: nobody on his team had wr ## From Traditional ML to AI Agents: How Booking.com Scales AI Observability With Arize AI DevFeed: [From Traditional ML to AI Agents: How Booking.com Scales AI Observability With Arize AI]() Original publisher: [Read original article]() Author: Amir Bitaraf Published: 2026-07-10T07:52:18Z Content type: article Language: en Sources: [Booking.com Data Science]() Topics: [ai observability](), [agent observability](), [Machine learning](), [observability](), [human review]() Tags: [agents](), [ai](), [ai-agents](), [ai-observability](), [blog-posts](), [data-quality](), [human-review](), [latency](), [ml](), [observability]() ### AI overview Booking.com describes building an AI-native observability stack for traditional machine learning systems and agentic AI workflows. The article explains that its diverse systems require observability to detect changes, regressions, data quality issues, misconfigurations, and responsible-AI concerns across different operating constraints and user contexts. ### Source excerpt Building an AI-native observability stack for agentic AI and traditional ML at Booking.com Authors: Amir Bitaraf, Shahaf Veber Why AI Observability Matters at Booking.com At Booking.com, AI helps travellers and partners in every step of their journey, from how people discover destinations to the way we support them while they're on the road. Rather than a single flagship model, we rely on a large and growing collection of systems that each solve a specific problem at scale. To make this concrete, consider a few examples: Trip planning assistants that help travelers turn vague ideas ("somewhere warm in April with good hiking") into concrete, bookable itineraries. On-site helpers that turn property details, amenities, reviews, and options into plain-language guidance, so people can choose the right stay with confidence. Partner copilots that help accommodation partners and other suppliers respond to guest messages faster and more consistently, while still staying in control of the final reply. Ranking systems that decide which options to show first in search and recommendation to surfaces, balancing user relevance with experimentation needs. Fraud detection models that quietly protect customers and partners in the background by flagging suspicious activity before it turns into real harm. Each of these systems is built and iterated on by different teams, uses different data, and runs under different constraints such as real-time vs batch, strict latency budgets vs more relaxed ones, fully automated vs human-in-the-loop. As we scale this ecosystem, observability becomes a first-class requirement, not a nice-to-have as we need to: Know when something changes in the real world, a new travel pattern, a data quality issue, a misconfiguration and how that affects model behaviour and user experience. Detect regressions early: slower responses, more confusing answers, drops in relevance or conversion, or subtle shifts that only show up for specific geographies, devices, or use ## Outlier Handling at Scale in Experimentation DevFeed: [Outlier Handling at Scale in Experimentation]() Original publisher: [Read original article]() Author: Margarida Moreira da Silva Published: 2026-07-01T13:44:26Z Content type: article Language: en Sources: [Booking.com Data Science]() Topics: [experiments](), [data](), [Simulation](), [Ground truth / benchmark quality](), [plotting]() Tags: [data](), [experimentation](), [experiments](), [false-positive](), [outlier-detection](), [simulation](), [variance]() ### AI overview The article examines how extreme values affect experimentation at Booking.com. It describes permutation tests and simulated A/A experiments for diagnosing distorted p-value distributions, and reports that increasing outlier magnitude and frequency can cause test failures. ### Source excerpt At Booking.com, thousands of experiments run simultaneously across highly heterogeneous users, from individual travellers to large travel agencies. This means our experiment data regularly contains legitimate but extreme values. When these go unhandled, they distort the statistical conclusions we draw, leading us to scale ideas that don't create value, or to discard ones that do. So, we need outlier handling methods that are reliable, automated, and applicable across diverse metrics without manual intervention. The Problem When extreme values are present in experiment data, they can compromise the estimation of average treatment effects (ATE), leading to unreliable test results and reduced statistical power. Even a single observation can inflate variance enough to mask a real effect or produce a spurious one. In practice, this means we risk shipping changes that appear positive but are not, or killing promising features because noise masked their real effect. At Booking.com's scale, this increase in false conclusions quickly compounds into a meaningful impact on customer experience and business outcomes. A Diagnostic Tool: the Permutation Test One way to assess whether extreme values are distorting results is the permutation test. By permuting over experiment data, we generate hundreds of simulated AA experiments where we know the ground truth: there is no real effect. Plotting the resulting p-values, we expect a uniform distribution. If it instead looks skewed, the underlying data distribution is compromising the validity of results. Plot 1: P-value distributions from simulated A/A tests. Clean normally-distributed estimated effects produce a uniform distribution (left), while the presence of extreme outliers results in skewed p-values (right), indicating a distorted false positive rate.Simulation Evidence: What Drives Failure? We ran AA permutation tests across a range of simulated data distributions to understand when they fail (i.e. not show a uniform p-value di ## The Causality Gap: Measuring the True Impact of Voluntary Adoption in Digital Marketplaces DevFeed: [The Causality Gap: Measuring the True Impact of Voluntary Adoption in Digital Marketplaces]() Original publisher: [Read original article]() Author: Lin Jia Published: 2026-05-22T08:05:43Z Content type: article Language: en Sources: [Booking.com Data Science]() Topics: [A/B Testing](), [doubleml](), [Data Science](), [Tool]() Tags: [a-b-testing](), [causal-inference](), [causal-machine-learning](), [causality](), [data-science](), [doubleml](), [e-commerce](), [experimentation](), [featured](), [measuring]() ### AI overview This article explains why standard A/B tests can mismeasure features that require voluntary adoption in digital marketplaces. It presents Randomized Encouragement Design combined with Double Machine Learning as a way to estimate both causal impact among adopters and overall rollout impact. ### Source excerpt by Lin Jia, Kexin Fei The Content of this post has been presented at Pydata Amsterdam 2026 and the slides can be found here TL; DR Whenever a feature requires voluntary adoption, standard A/B testing breaks: low adoption flattens the topline, and self-selection makes adopters incomparable to non-adopters. Combining Randomized Encouragement Design (RED) with Double Machine Learning (DoubleML) recovers two answers -- the causal lift for adopters and the rollout's overall impact. When they diverge, the gap turns an ambiguous topline into a sharp product decision -- build a better product, or build a better adoption funnel. 1. The Opt-In Barrier Across Demand and Supply Across the tech industry, many platform features rely on voluntary adoption. A customer chooses whether to claim a promotional discount. A traveller opts into a loyalty program. An e-commerce seller enables a smart-pricing tool. In every case, the platform cannot force adoption -- and the feature's true impact becomes hard to measure, on both demand and supply. At Booking.com this challenge spans both sides of the marketplace -- travellers choosing to log in, partners choosing to adopt new features. Unlike a search-ranking change that applies to 100% of traffic, opt-in features create a "trilemma" for Product Data Science: Voluntary Adoption (the "Opt-In" Barrier): Users must actively enable the feature. A standard A/B test cannot separate the product's effect from the motivation that drove users to adopt it. Extreme Heterogeneity: Travellers range from once-a-year holidaymakers to travel agencies booking thousands of nights; partners range from single-apartment hosts to hotel chains. This variance is noise on both sides. Finite Sample Sizes: Opt-in features target a finite sub-segment, so we cannot simply "run the test longer" to gain power. When these stack up, a flat topline can hide a strong product behind a weak adoption funnel. Genuinely strong features get killed, and resources flow into the wrong int ## Scaling Experimentation Quality at Booking.com DevFeed: [Scaling Experimentation Quality at Booking.com]() Original publisher: [Read original article]() Author: Edgar Cano Published: 2026-03-24T11:44:21Z Content type: article Language: en Sources: [Booking.com Data Science]() Topics: [experiments](), [Development](), [decision-making](), [human review]() Tags: [best-practices](), [data-science](), [decision-making](), [experimentation](), [featured](), [human-review](), [metrics](), [product-development](), [quality]() ### AI overview Booking.com describes how it addressed declining experimentation quality as experiment volume grew. The article discusses arbitrary test durations, significance-seeking, and the trade-offs between enforcing standards and educating product teams. ### Source excerpt Authors: Edgar Cano, Daisy Duursma, Nils Skotara, Melanie Mueller Figure 1. Three-pillar components of Booking.com's Experimentation Quality Experimentation is at the core of product development in Booking.com, powered by our in-house platform, "ET" (Experiment Tool). At any given moment, we run approximately 1,000 parallel experiments to evaluate product changes. These experiments or A/B tests allow teams to directly compare a new version of the website against the existing one, validating hypotheses about how specific changes impact important metrics. In our organization, these pitfalls became more evident as our experiment volume grew. We observed that experimenters might set an arbitrary "two-week" duration without thinking about sufficient power, or extend a test until results "became significant" or "trended positive." Knowing that this lack of consistency leads to flawed decision-making we dedicated significant effort to increasing the quality of our experimentation process, making Experimentation Quality a key KPI for our program. However, identifying the problem was only the start; the greater challenge is how to implement these standards across a large organization. Enforcement vs. Education When deciding how to scale quality, we faced a fundamental choice: Do we enforce strict controls or we rely on education. Ultimately, we left it to product teams to decide how to conduct their experiments. This choice entailed several trade-offs: Enforcement: Ensures comparability, consistency, and reliability. However, it comes at the cost of flexibility. There is a risk that people follow "rules" blindly without understanding the rationale. Education: Aims for a culture where experimenters understand the why behind best practices. This leads to better buy-in, allows teams to challenge methods, and highlights individual responsibility. It also prevents bottlenecking. If enforcement requires human review, it slows down development. However, education requires a massive ## Booking.com 2026 GenAI and ML PhD Research Internship in Amsterdam DevFeed: [Booking.com 2026 GenAI and ML PhD Research Internship in Amsterdam]() Original publisher: [Read original article]() Author: Yang Yang Published: 2026-02-05T10:39:26Z Content type: article Language: en Sources: [Booking.com Data Science]() Topics: [Machine Learning & Artificial Intelligence](), [genai](), [AI Research](), [Python](), [Large Language Model](), [Apache Spark](), [SQL](), [Hadoop](), [Reinforcement learning](), [Synthetic Data Generation](), [Transformer]() Tags: [2026](), [ai](), [big-data](), [blog-posts](), [featured](), [genai](), [internship](), [machine-learning](), [ml](), [python](), [research]() ### AI overview Booking.com is recruiting current PhD students in quantitative fields for a three-month GenAI and machine learning research internship in Amsterdam in 2026. Projects include LLM alignment, transformer explainability, embeddings, context engineering, and synthetic data generation. ### Source excerpt At Booking.com, we don't just use Machine Learning -- we use it to solve some of the most complex travel challenges in the world. We're looking for the next generation of researchers to join our Machine Learning community in Amsterdam for a 3-month deep dive into cutting-edge AI. The Program As a Research Intern, you'll be embedded in our teams, working alongside world-class mentors. Your mission? To tackle real-world problems and push the boundaries of the state-of-the-art. Are You the One? We're looking for current PhD students in quantitative fields (CS, Math, AI, Physics) who can conduct independent research and have a solid grip on Python and Big Data tech (SQL, Spark, Hadoop). What's in it for you? You won't just be "an intern". You'll be a contributor to our Machine Learning community. You'll have the opportunity to contribute to the existing efforts of the Machine Learning teams, participate in internal knowledge-sharing sessions, and enjoy the collaborative, high-energy environment of our Amsterdam HQ. Projects Regularized Target Encoding for large real-world datasets Multi-Agent Collaboration Aligning LLMs with user feedback via reinforcement learning Multi-level treatments Interpretable Foundations: Explainability Methods for Transformer Models on Sequential Event Data Scalable and generalisable ID embedding learning Improving property embeddings with better handling of rich and long-context data Utility-aware retrieval for context engineering in travel planning Synthetic Data Generation in Images Requirements We are looking for independent researchers with strong understanding of Machine Learning topics (see requirements for each project in the Linkedin ad), have a track record of peer-reviewed publications and a passion for solving complex problems. Why Booking.com? You'll join a vibrant, diverse community of data scientists and researchers who love to experiment. Beyond the code, you'll experience the unique culture of our Amsterdam headquarters -- a hub ## Beyond Prompt Engineering: How We Used Supervised Fine-Tuning for Travel Recommendations DevFeed: [Beyond Prompt Engineering: How We Used Supervised Fine-Tuning for Travel Recommendations]() Original publisher: [Read original article]() Author: Amit Meitin Published: 2026-01-29T10:53:21Z Content type: article Language: en Sources: [Booking.com Data Science]() Topics: [Fine-tuning](), [recommendation systems](), [Large Language Model](), [real-time](), [AI Development]() Tags: [ai](), [blog-posts](), [engineering](), [fine-tuning](), [inference](), [language-models](), [machine-learning](), [models](), [prompt](), [recommendations]() ### AI overview Booking.com describes how it used supervised fine-tuning to improve travel recommendations for its AI Trip Planner. The approach combines large language models' understanding of unstructured traveler requests with structured behavioral data from searches, clicks, and bookings, while the article reports superior recommendation metrics and 3x faster inference. ### Source excerpt How fine-tuning delivered superior recommendation metrics while achieving 3x faster inference Every traveler is unique, and so is every trip. At Booking.com, we're always looking for new ways to make trip planning feel less like a chore and more like an adventure. That's why we set out to build an AI Trip Planner that doesn't just answer questions, but actually understands what travelers want, expressed in ways they are most comfortable with. But here's the challenge: travelers today are expressing their needs in ways that are more unstructured and nuanced than ever before. Instead of ticking boxes or picking from drop-down menus, people describe their dream vacations in their own words. Sometimes these requests are vague, sometimes very specific, but always personal. Traditional machine learning models, which thrive on structured data and clear signals, can struggle to keep up with this new level of expressiveness. At the same time, we at Booking.com have years of valuable, structured data from searches, clicks, and bookings. This data captures what travelers actually do: the trips they plan and the vacation they go on. The big question for us became how to combine the best of both worlds -- How can we harness the power of Large Language Models (LLMs) to understand and respond to unstructured requests, and use the wealth of behavioral data that has driven our recommendations for years? This post is about how we tackled that question and what we learned along the way. What is the AI Trip Planner? The AI Trip Planner is Booking.com's conversational assistant that helps travelers find destinations, accommodations and other trip components through natural, real-time chat. Instead of searching with filters, users describe what they want in their own words. The planner suggests options, whether it is inspiration or specific places to stay. Integrated into the app, it makes planning and discovering seamless and is now available in multiple countries and languages. Bridging ## AI Agent Evaluation DevFeed: [AI Agent Evaluation]() Original publisher: [Read original article]() Author: Antonio Castelli Published: 2026-01-21T13:11:23Z Content type: tutorial Language: en Sources: [Booking.com Data Science]() Topics: [LLM evaluation / benchmarking](), [AI Agent](), [Artificial Intelligence](), [Tool]() Tags: [agent](), [ai](), [ai-agent](), [api](), [best-practices](), [blog-posts](), [evaluation](), [llm-agents](), [llm-evaluation](), [machine-learning](), [sql](), [tools]() ### AI overview Booking.com's article presents practical guidance for evaluating LLM agents. It explains how agents differ from single LLMs because they can use external tools and iterate through thoughts, actions, and observations, then introduces black-box and glass-box evaluation approaches. ### Source excerpt AI Agent Evaluation: practical tips at Booking.com Authors: Zeno Belligoli, Antonio Castelli, George Chouliaras This article is the 2nd part of our GenAI evaluation best practices series. You can read the first part, focusing on single LLM evaluation, here. 1. Overview LLM agents are advanced AI systems that leverage large language models to perform complex tasks autonomously. Unlike single LLMs that respond to direct prompts, agents can break down problems, use external tools, and iterate on solutions through a series of thoughts, actions, and observations. This allows them to achieve goals that require more intricate planning, reasoning, and interaction with various environments. Agents can autonomously determine if tool utilization is necessary to fulfill a user's request. For example, they might employ: a calculator for mathematical operations, a flights API to retrieve available flights for a certain destination, or execute a SQL query to fetch information about an hotel reservation from a database. Figure1: Schematic view of an agent. The user might interact with the agent within a multi-turn conversation, asking it to perform various tasks (recommend vacation destinations, ask questions about a hotel etc.). The agent should try to complete the requested tasks abiding by a set of rules (or constraints) provided by the product specification (e.g. do not recommend properties outside the inventory). In performing the tasks the agent has freedom to use a certain set of tools, and it can interact with them via API requests. Tool examples typically range from general (e.g. calculate travel distance) to use case specific functionalities (e.g. retrieving information from a database). While these enhanced capabilities are fundamental to solve tasks which would be difficult (or even impossible) to solve with text generation only, they require a more complex evaluation process compared to evaluating a single "prompt-response" LLM. The evaluation of an agent's performance ## Highlights of Booking.com's publication in 2025 DevFeed: [Highlights of Booking.com's publication in 2025]() Original publisher: [Read original article]() Author: Yang Yang Published: 2026-01-20T09:20:13Z Content type: article Language: en Sources: [Booking.com Data Science]() Topics: [Machine learning](), [Artificial Intelligence](), [Natural language processing](), [Language models](), [Fine-tuning](), [recommendation systems](), [Optimization](), [Compression](), [Inference]() Tags: [2025](), [ai](), [booking](), [compression](), [data-science](), [fine-tuning](), [inference](), [language-models](), [latency](), [neurips](), [nlp](), [optimization](), [publication]() ### AI overview Booking.com highlights its 2025 machine learning publications, including papers accepted at major conferences and research on applying Medusa speculative decoding and knowledge distillation to travel-related language model tasks. ### Source excerpt At Booking.com, our mission is to make experiencing the world easier for everyone. We are committed to investing in cutting-edge technology that removes the barriers to travel, enabling seamless connections between millions of travelers and unforgettable experiences, diverse transportation options, and exceptional accommodations. The intersection of academic rigor and industry application is where true transformation happens. In 2025, our ML community bridged this gap more effectively than ever, contributing vital new insights to the global scientific community. With 8 out of 13 papers accepted at premier conferences -- including NeurIPS, SIGIR, KDD, and ACL -- our colleagues have demonstrated world-class expertise in AI, NLP, recommendation systems, uplift modeling, etc. These aren't just theoretical wins; they are the engines of innovation that allow us to push technological boundaries, ensuring our platform remains the most sophisticated and intuitive guide in the ever-evolving travel industry. Below, we highlight some of the key achievements and insights from these groundbreaking works. Speed Without Sacrifice: Fine-Tuning Language Models with Medusa and Knowledge Distillation in Travel Applications By Daniel Zagyva, Emmanouil Stergiadis, Laurens Van Der Maas, Aleksandra Dokic, Eran Fainman, Ilya Gusev, Moran Beladev Best paper award of 2025 ACL Industry Track https://aclanthology.org/2025.acl-industry.48/ In high-stakes industrial NLP applications, balancing generation quality with speed and efficiency presents significant challenges. We address them by investigating two complementary optimization approaches: Medusa for speculative decoding and knowledge distillation (KD) for model compression. We demonstrate the practical application of these techniques in real-world travel domain tasks, including trip planning, smart filters, and generating accommodation descriptions. We introduce modifications to the Medusa implementation, starting with base pre-trained models