# In-House LLM Serving at Netflix

DevFeed: [In-House LLM Serving at Netflix](<https://devfeed.tech/articles/in-house-llm-serving-at-netflix-140.md>)

Original publisher: [Read original article](<https://netflixtechblog.com/in-house-llm-serving-at-netflix-a5a8e799ea2c?source=rss----2615bd06b42e---4>)

Author: Netflix Technology Blog

Published: 2026-07-17T21:32:39Z

Content type: article

Language: en

Sources: [Netflix](<https://devfeed.tech/sources/netflix.md>), [Netflix TechBlog - Medium](<https://devfeed.tech/sources/netflix-techblog-medium.md>)

Topics: [LLMs](<https://devfeed.tech/topics/llms.md>), [Netflix](<https://devfeed.tech/topics/netflix.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [AI Platform](<https://devfeed.tech/topics/ai-platform.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [autoscaling](<https://devfeed.tech/topics/autoscaling.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [gRPC](<https://devfeed.tech/topics/grpc.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-platform](<https://devfeed.tech/tags/ai-platform.md>), [autoscaling](<https://devfeed.tech/tags/autoscaling.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [netflix](<https://devfeed.tech/tags/netflix.md>), [platform-engineering](<https://devfeed.tech/tags/platform-engineering.md>)

## AI overview

Netflix describes its in-house LLM serving stack, covering deployment, inference, API access paths, and production trade-offs.

## Source excerpt

By AI Platform's Model Runtime team and Inference team Introduction Most organizations consume LLMs through hosted APIs. Netflix went further -- we run the full stack ourselves, from model deployment through inference, inside our existing production environment rather than a separate ML silo. Some of those decisions weren't obvious, and a few revealed their trade-offs only under production load. This post focuses on the choices where alternatives were seriously considered: engine selection, model packaging, API surface design, deployment strategy, and output constraints enforcement. The goal is to share not just what was built, but why -- and what production revealed that the design phase didn't anticipate. Architecture Overview Member-scale ML at Netflix is fronted by a unified JVM-based serving system that handles the end-to-end flow for downstream consumers: routing and A/B test logic, candidate generation, feature fetching, inference, post-processing, and logging at each stage. Both real-time and cached batch paths are supported. Figure 1 shows the two ways callers reach inference today: the gRPC path through this serving system and a direct HTTP path used by newer LLM-driven applications. Where inference runs depends on the model. Small CPU models run in-process, avoiding remote-call overhead. Larger models need GPUs -- the serving system handles pre- and post-processing locally but delegates inference to a remote service, Model Scoring Service (MSS). MSS is the shared inference backend, supporting XGBoost, TensorFlow, PyTorch, and LLMs behind a unified interface, with NVIDIA Triton Inference Server underneath managing model loading, batching, and GPU scheduling. On top of Triton sits a Java control plane that handles deployment, versioning, health checking, autoscaling, and multi-region rollout. Model authors package their artifacts and configure the deployment; the control plane provisions GPU instances, configures Triton, and orchestrates zero-downtime upgrades