# From Traditional ML to AI Agents: How Booking.com Scales AI Observability With Arize AI

DevFeed: [From Traditional ML to AI Agents: How Booking.com Scales AI Observability With Arize AI](<https://devfeed.tech/articles/from-traditional-ml-to-ai-agents-how-booking-com-scales-ai-observability-with-arize-ai-30450.md>)

Original publisher: [Read original article](<https://booking.ai/from-traditional-ml-to-ai-agents-how-booking-com-scales-ai-observability-with-arize-ai-625ac3996c7e?source=rss----4d265f07defc---4>)

Author: Amir Bitaraf

Published: 2026-07-10T07:52:18Z

Content type: article

Language: en

Sources: [Booking.com Data Science](<https://devfeed.tech/sources/booking-com-data-science.md>)

Topics: [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [observability](<https://devfeed.tech/topics/observability.md>), [observability pipelines](<https://devfeed.tech/topics/observability-pipelines.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [AI for customer service automation](<https://devfeed.tech/topics/ai-for-customer-service-automation.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-native](<https://devfeed.tech/tags/ai-native.md>), [ai-native-observability](<https://devfeed.tech/tags/ai-native-observability.md>), [ai-observability](<https://devfeed.tech/tags/ai-observability.md>), [arize](<https://devfeed.tech/tags/arize.md>), [automated](<https://devfeed.tech/tags/automated.md>), [background](<https://devfeed.tech/tags/background.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [batch](<https://devfeed.tech/tags/batch.md>), [behaviour](<https://devfeed.tech/tags/behaviour.md>), [blog-posts](<https://devfeed.tech/tags/blog-posts.md>), [booking](<https://devfeed.tech/tags/booking.md>), [building](<https://devfeed.tech/tags/building.md>), [can](<https://devfeed.tech/tags/can.md>), [ml](<https://devfeed.tech/tags/ml.md>), [observability](<https://devfeed.tech/tags/observability.md>), [with](<https://devfeed.tech/tags/with.md>)

## AI overview

Booking.com uses Arize AI as a shared observability platform for traditional ML and agentic AI, with different telemetry paths for each. Agent workflows emit OpenTelemetry traces with OpenInference conventions for model calls, retrieval, tools, guardrails, and related context; a central collector samples traces, applies PII-aware redaction, and routes them onward. Traditional ML uses prepared tabular prediction logs with features, predictions, labels, and model metadata, excluding sensitive data and limiting fields or sampling when needed. Arize supports trace inspection, alerts and dashboards, evaluations, production-derived datasets, plus drift, cohort, and feature-importance analysis for ML. In one production case, a latency alert led the team to identify an experimental model running without a service tier; coordinating with the provider restored expected latency. In prototyping, elevated P90 token counts and lower later-turn evaluations exposed long attraction URLs accumulating in chatbot history, allowing a fix before production.

## Source excerpt

Building an AI-native observability stack for agentic AI and traditional ML at Booking.com Authors: Amir Bitaraf, Shahaf Veber Why AI Observability Matters at Booking.com At Booking.com, AI helps travellers and partners in every step of their journey, from how people discover destinations to the way we support them while they're on the road. Rather than a single flagship model, we rely on a large and growing collection of systems that each solve a specific problem at scale. To make this concrete, consider a few examples: Trip planning assistants that help travelers turn vague ideas ("somewhere warm in April with good hiking") into concrete, bookable itineraries. On-site helpers that turn property details, amenities, reviews, and options into plain-language guidance, so people can choose the right stay with confidence. Partner copilots that help accommodation partners and other suppliers respond to guest messages faster and more consistently, while still staying in control of the final reply. Ranking systems that decide which options to show first in search and recommendation to surfaces, balancing user relevance with experimentation needs. Fraud detection models that quietly protect customers and partners in the background by flagging suspicious activity before it turns into real harm. Each of these systems is built and iterated on by different teams, uses different data, and runs under different constraints such as real-time vs batch, strict latency budgets vs more relaxed ones, fully automated vs human-in-the-loop. As we scale this ecosystem, observability becomes a first-class requirement, not a nice-to-have as we need to: Know when something changes in the real world, a new travel pattern, a data quality issue, a misconfiguration and how that affects model behaviour and user experience. Detect regressions early: slower responses, more confusing answers, drops in relevance or conversion, or subtle shifts that only show up for specific geographies, devices, or use