# model-serving Published articles for model-serving. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Introducing Amazon SageMaker HyperPod Inference Gateway DevFeed: [Introducing Amazon SageMaker HyperPod Inference Gateway]() Original publisher: [Read original article]() Author: Vinay Arora Published: 2026-09-18T13:08:34Z Content type: release Language: en Sources: [Artificial Intelligence]() Topics: [Amazon SageMaker HyperPod](), [Model Routing](), [model-serving](), [Inference Performance](), [Amazon Elastic Kubernetes Service](), [Kubernetes](), [AI Inference](), [GPU](), [AI Platform](), [Prometheus]() Tags: [amazon-eks](), [amazon-sagemaker-hyperpod](), [announcements](), [expert-400](), [gpu](), [inference](), [kubernetes](), [latency](), [metrics](), [model-serving](), [performance](), [routing]() ### AI overview Amazon SageMaker HyperPod Inference Gateway is a Kubernetes-native, GPU-aware routing add-on for Amazon EKS. It uses real-time GPU signals and model-serving metrics to route inference requests to suitable pods, aiming to reduce GPU waste and first-token latency without application changes. ### Source excerpt Amazon SageMaker HyperPod Inference Gateway is a Kubernetes-native, GPU-aware routing add-on for Amazon EKS. It uses real-time GPU signals to send each inference request to the best-suited pod, cutting first-token latency by up to 82% with no changes to your model servers or client applications. ## When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving DevFeed: [When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving]() Original publisher: [Read original article]() Author: Tanya Lenz Published: 2026-09-09T20:31:04Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Optimization](), [model-deployment](), [multimodal](), [Dynamo](), [Inference Performance](), [Embeddings](), [Mixture of Experts (MoE)](), [GPU](), [Open Source](), [Artificial Intelligence]() Tags: [agentic-ai-generative-ai](), [ai-agent](), [ai-inference](), [computer-vision-video-analytics](), [developer-tools-techniques](), [dynamo](), [embeddings](), [gpu](), [inference](), [low-latency-inference](), [mixture-of-experts-moe](), [model-serving](), [multimodal](), [nvfp4](), [open-source](), [optimization]() ### AI overview This article explains when encode-prefill-decode disaggregation improves multimodal model serving. Using NVIDIA Dynamo, it separates vision encoding from LLM prefill and decode so the stages can batch, schedule, and scale independently. The approach is most effective for image-heavy prompts, short-to-medium outputs, and quantized mixture-of-experts models, with reported gains of up to 5x in time to first token and 7x in end-to-end response time. ### Source excerpt Encode-prefill-decode (EPD) disaggregation is an inference optimization technique for multimodal models that separates the vision encoder stage from the prefill... ## Self-Hosted Platform Build Order: Dependencies from Bare Metal to Model Serving DevFeed: [Self-Hosted Platform Build Order: Dependencies from Bare Metal to Model Serving]() Original publisher: [Read original article]() Author: Philipp Lehmann (philipp.lehmann@gruppe.ai) Published: 2026-09-04T07:00:00Z Content type: article Language: en Sources: [Philipp Lehmann]() Topics: [Self-hosted](), [Ansible](), [Kubernetes](), [ceph](), [observability](), [model-serving](), [Compose]() Tags: [ansible](), [architecture](), [article](), [ceph](), [compose](), [infrastructure-as-code](), [kubernetes](), [model-serving](), [observability](), [self-hosted]() ### AI overview This article explains the dependency order for building a self-hosted platform. It covers consistent bare-metal inventory, Ansible configuration convergence, networking, Kubernetes, Ceph storage, identity, observability, and model serving. ### Source excerpt Self-hosted platform build order: why mesh, cluster, Ceph storage, identity and observability must precede model serving, shown with Compose depends_on. ## Kubeflow Has Graduated from CNCF DevFeed: [Kubeflow Has Graduated from CNCF]() Original publisher: [Read original article]() Author: Kubeflow Published: 2026-08-18T05:00:00Z Content type: release Language: en Sources: [Kubeflow]() Topics: [AI Infrastructure](), [AI Platform](), [Kubernetes](), [Cloud Native Ecosystem](), [distributed-training](), [Fine-tuning](), [model-serving]() Tags: [ai-infrastructure](), [announcement](), [cloud-native](), [cncf](), [distributed-training](), [fine-tuning](), [inference](), [kubernetes](), [model-serving](), [production]() ### AI overview Kubeflow has graduated from the Cloud Native Computing Foundation, recognizing its maturity and adoption as a Kubernetes-native platform for AI and machine learning workloads. The article highlights its community growth, enterprise use, security audit, governance work, and focus on scalable, portable infrastructure. ### Source excerpt Kubeflow is a CNCF Graduated Project ## Using ArgoCD to Manage AI Model Deployments with GitOps DevFeed: [Using ArgoCD to Manage AI Model Deployments with GitOps]() Original publisher: [Read original article]() Author: Pramodh Kumar M Published: 2026-07-28T18:24:57Z Content type: tutorial Language: en Sources: [Kubernetes - KodeKloud Blog | DevOps, Cloud, Kubernetes, AI Tutorials & More]() Topics: [GitOps](), [argocd](), [model-deployment](), [Artificial Intelligence](), [model-serving](), [Architecture & Design]() Tags: [ai](), [ai-models](), [argo-rollouts-model-promotion](), [argocd](), [argocd-ai-model-deployment](), [argocd-applicationset](), [argocd-sync-waves](), [automation](), [ci-cd](), [declarative-model-serving](), [deployment](), [devops](), [gitops](), [gitops-for-ai-model-deployments](), [gitops-mlops](), [kserve](), [kserve-canary-deployment](), [kubernetes](), [kubernetes-model-deployment-rollback](), [ml-model-deployment](), [ml-model-versioning](), [mlops](), [model-deployment](), [model-registry-to-git-automation](), [model-serving](), [pull-requests](), [standard](), [tutorial]() ### AI overview This tutorial explains how to use ArgoCD and GitOps to deploy AI models while accounting for model weights stored outside container images. It covers pinned model references, rollback and recovery, synchronization ordering, readiness probes, evaluation-based promotion gates, and KServe status handling. ### Source excerpt Your microservices deploy through pull requests with full audit trails. Your models deploy because someone ran a script. Here is how to close that gap with ArgoCD, and what changes when the artifact weighs four gigabytes. ## KubeCon + CloudNativeCon India 2026: Our Kubeflow Community Experience DevFeed: [KubeCon + CloudNativeCon India 2026: Our Kubeflow Community Experience]() Original publisher: [Read original article]() Author: Khushi Agrawal Published: 2026-07-27T05:00:00Z Content type: article Language: en Sources: [Kubeflow]() Topics: [Kubernetes](), [MLOps](), [distributed-training](), [model-serving](), [Large Language Model](), [Retrieval Augmented Generation (RAG)](), [GPU](), [Cloud]() Tags: [2026](), [ai](), [cloud-native](), [community](), [distributed-training](), [gpu](), [india](), [kubecon](), [kubernetes](), [llms](), [machine-learning](), [mlops](), [model-serving](), [model-training](), [rag]() ### AI overview A Kubeflow community recap of KubeCon + CloudNativeCon India 2026 in Mumbai. It describes Kubeflow's presence at the CNCF Project Pavilion, common architecture questions, featured talks, booth discussions about production machine learning workflows, distributed training, model serving, GPU resource allocation, LLMs, and RAG pipelines, and efforts to onboard contributors. ### Source excerpt Introduction ## In-House LLM Serving at Netflix DevFeed: [In-House LLM Serving at Netflix]() Original publisher: [Read original article]() Author: Netflix Technology Blog Published: 2026-07-17T21:32:39Z Content type: article Language: en Sources: [Netflix](), [Netflix TechBlog - Medium]() Topics: [LLMs](), [Netflix](), [Inference](), [AI Platform](), [model-deployment](), [autoscaling](), [GPU](), [gRPC]() Tags: [ai](), [ai-platform](), [autoscaling](), [deployment](), [gpu](), [inference](), [llm](), [model-serving](), [netflix](), [platform-engineering]() ### AI overview Netflix describes its in-house LLM serving stack, covering deployment, inference, API access paths, and production trade-offs. ### Source excerpt By AI Platform's Model Runtime team and Inference team Introduction Most organizations consume LLMs through hosted APIs. Netflix went further -- we run the full stack ourselves, from model deployment through inference, inside our existing production environment rather than a separate ML silo. Some of those decisions weren't obvious, and a few revealed their trade-offs only under production load. This post focuses on the choices where alternatives were seriously considered: engine selection, model packaging, API surface design, deployment strategy, and output constraints enforcement. The goal is to share not just what was built, but why -- and what production revealed that the design phase didn't anticipate. Architecture Overview Member-scale ML at Netflix is fronted by a unified JVM-based serving system that handles the end-to-end flow for downstream consumers: routing and A/B test logic, candidate generation, feature fetching, inference, post-processing, and logging at each stage. Both real-time and cached batch paths are supported. Figure 1 shows the two ways callers reach inference today: the gRPC path through this serving system and a direct HTTP path used by newer LLM-driven applications. Where inference runs depends on the model. Small CPU models run in-process, avoiding remote-call overhead. Larger models need GPUs -- the serving system handles pre- and post-processing locally but delegates inference to a remote service, Model Scoring Service (MSS). MSS is the shared inference backend, supporting XGBoost, TensorFlow, PyTorch, and LLMs behind a unified interface, with NVIDIA Triton Inference Server underneath managing model loading, batching, and GPU scheduling. On top of Triton sits a Java control plane that handles deployment, versioning, health checking, autoscaling, and multi-region rollout. Model authors package their artifacts and configure the deployment; the control plane provisions GPU instances, configures Triton, and orchestrates zero-downtime upgrades ## Building a Minimal Control Plane to Reconcile Docker Containers for Model Serving DevFeed: [Building a Minimal Control Plane to Reconcile Docker Containers for Model Serving]() Original publisher: [Read original article]() Published: 2026-07-02T00:00:00Z Content type: tutorial Language: en Sources: [Alex Korbonits]() Topics: [control-plane](), [model-serving](), [Docker Container](), [Amazon Machine Learning](), [Streaming]() Tags: [container](), [control-plane](), [inference](), [model-serving](), [python]() ### AI overview The author explains control planes by building a small Python-based system that maintains a declared number of Docker containers for model serving. The system repeatedly compares desired and actual state and reconciles differences, including restoring a container after it stops. ### Source excerpt I can design batch, real-time, and streaming inference -- the data plane. But I'd never built the control plane that manages it. So I built the smallest one I could, from scratch, to finally understand reconciliation: the one idea that separates a control plane from a deploy script. ## Fully Automated AI Inference on AWS, Azure, and Google Cloud with Pulumi DevFeed: [Fully Automated AI Inference on AWS, Azure, and Google Cloud with Pulumi]() Original publisher: [Read original article]() Author: Engin Diri Published: 2026-06-30T00:00:00Z Content type: tutorial Language: en Sources: [Pulumi]() Topics: [Ollama](), [Cloud](), [Infrastructure as code](), [model-serving](), [GPU](), [Artificial Intelligence](), [OpenID connect (OIDC)](), [Ubuntu]() Tags: [ai](), [aws](), [azure](), [cloud](), [gcp](), [google-cloud](), [gpu](), [inference](), [infrastructure-as-code](), [model-serving](), [oidc](), [ollama](), [pulumi-esc](), [tutorials](), [typescript](), [ubuntu]() ### AI overview This tutorial shows how to use Pulumi to provision GPU virtual machines on AWS, Azure, and Google Cloud, install drivers, run Ollama, and pull an open-source model without manual setup. It also uses Pulumi ESC with OIDC instead of a long-lived access key and avoids treating model-download completion as an infrastructure resource. ### Source excerpt Putting Ollama on a cloud GPU is something I keep coming back to. A while ago I wrote up running open-source LLMs on an AWS EC2 box with Ollama and Pulumi, and the shape never really changes: a GPU instance, a model server, and a firewall rule in front. Infrastructure as code earned its place by making that kind of setup predictable and repeatable, and AI infrastructure is no exception. A GPU box serving a model is still a VM, a disk, and a firewall rule, and it should be declared like one. Thorsten Hans made exactly that case in his Akamai post, Fully Automated AI Infrastructures with Terraform and Akamai Cloud, which stands up a single GPU instance on Linode, installs the drivers, runs Ollama, and pulls a model, with no manual steps after terraform apply. I liked the shape of it, so this post ports the same idea to Pulumi and runs it across AWS, Azure, and Google Cloud instead of one. The result is one program shape per cloud: a single pulumi up brings up a GPU box that installs its own driver, runs Ollama, and pulls a model with no manual steps, and a single pulumi destroy takes it back down. Along the way it drops the two imperative bits the Terraform version leans on: a static access token sitting in an environment variable, and a null_resource running a shell loop to wait for the model. The first becomes an OIDC login from a Pulumi ESC environment, so no long-lived key lives anywhere. The second turns out not to be a resource at all. What you are building Strip away the per-cloud naming and every version of this is the same three things: a GPU virtual machine, a firewall in front of it, and a cloud-init script that turns a bare Ubuntu box into a running inference server. The model serving runs on Ollama, which exposes an HTTP API on port 11434 and keeps the model resident in GPU memory between requests. flowchart LR Dev([Your machine / curl]) -->|"HTTP :11434"| FW["Firewall / security group (allow 11434, optional 22)"] FW --> VM["GPU VM (Ubuntu 24.04) ## I Built a 256GB Local AI Cluster on My Desk DevFeed: [I Built a 256GB Local AI Cluster on My Desk]() Original publisher: [Read original article]() Author: Techno Tim Published: 2026-05-18T13:00:00Z Content type: article Language: en Sources: [Techno Tim]() Topics: [Local AI](), [Artificial Intelligence](), [agentic-coding](), [Hardware](), [Homelab](), [coding](), [model-serving](), [vllm](), [networking](), [Docker](), [NCCL](), [Monitoring]() Tags: [agentic-coding](), [ai](), [docker](), [github](), [hardware](), [homelab](), [local-ai](), [model-serving](), [monitoring](), [nccl](), [networking](), [self-hosted](), [self-hosted-ai](), [server](), [vllm]() ### AI overview The article tests a two-node local AI cluster built from ASUS Ascent GX10 systems, using one machine for practical agentic coding and two machines to run a larger local model. It evaluates a real coding workflow involving model serving, storage, networking, memory, Docker, NCCL, monitoring, power draw, and the application being built, concluding that local AI is capable and cloud-independent but still operationally complex. ### Source excerpt I have been covering local and self-hosted AI for a few years now - from running models privately at home to what is still running in my homelab today. But to run the larger, more capable models, you need something more specialized than a general-purpose home server. I wanted to know how good local AI has actually gotten, so I built a mini AI cluster on my desk and used it for a real coding wo... ## DigitalOcean Dedicated Inference: A Technical Deep Dive DevFeed: [DigitalOcean Dedicated Inference: A Technical Deep Dive]() Original publisher: [Read original article]() Author: dgupta Published: 2026-04-25T02:51:09Z Content type: article Language: en Sources: [DigitalOcean]() Topics: [AI Platform](), [Digital Ocean](), [Large Language Model](), [model-serving](), [GPU](), [autoscaling](), [Kubernetes](), [VPC]() Tags: [ai-platform](), [architecture](), [autoscaling](), [deep-dive](), [digitalocean](), [engineering](), [gpu](), [inference](), [kubernetes](), [llm](), [model-serving](), [production](), [technical]() ### AI overview This technical deep dive explains DigitalOcean Dedicated Inference, a managed LLM hosting service on dedicated GPUs. It describes the Kubernetes-native inference stack, public and private VPC endpoints, OpenAI-compatible APIs, serving and routing components, autoscaling, observability, and cost considerations for sustained, high-volume workloads. ### Source excerpt Getting a model to answer 10 inference requests concurrently is tricky but simple enough; getting it to handle 2,000 engineers hitting a coding assistant with long contexts, all day, without runaway costs, is where teams stall. A working endpoint is only the beginning. Teams need to identify the supporting hardware and wire up the right components--serving, scaling, observability, and cost guardrails--so the deployment can support expected SLAs and SLOs under real, sustained load. DigitalOcean already offers Serverless Inference on the DigitalOcean AI Platform: a fast path to models from OpenAI, Anthropic, Meta, or other providers, with minimal setup and token-based pricing. This offering works well for many use cases. However, when you need your own weights, predictable performance on dedicated GPUs, and economics that favor sustained, high-volume token generation over pay-per-token bursts, a different approach makes sense Dedicated Inference, our managed LLM hosting service on the DigitalOcean AI Platform, fills that gap. Dedicated Inference deploys and operates an opinionated inference stack on dedicated GPUs, with Kubernetes-native orchestration under the hood. You interact through the control plane and APIs you already use in the DigitalOcean ecosystem; the data plane exposes public and private endpoints so applications inside, or outside, your VPC can call your models securely. The service is designed to collapse a vast combinatorial space--GPU SKUs, runtimes, routers, autoscaling policies--into guided defaults so teams hit production milestones faster than DIY stacks, while retaining knobs that matter for model serving: replicas, scaling behavior, and advanced optimizations as you roll out your product roadmap. What we manage vs. what you control Every managed product draws a line between operator-owned and customer-owned concerns. Dedicated Inference aims to put day-two operations--cluster lifecycle integration, ingress, core serving and routing components, and t ## Implementing Model Monitoring on Databricks DevFeed: [Implementing Model Monitoring on Databricks]() Original publisher: [Read original article]() Author: Başak Tuğçe Eskili Published: 2025-08-06T16:55:23Z Content type: tutorial Language: en Sources: [MarvelousMLOps]() Topics: [databricks](), [MLOps](), [Monitoring](), [model-serving](), [dashboards](), [Inference](), [Logging](), [Data Quality]() Tags: [databricks](), [metrics](), [mlops](), [model-monitoring](), [model-serving](), [monitoring](), [request](), [table]() ### AI overview Lecture 10 in an MLOps with Databricks course demonstrates model monitoring using inference tables and Lakehouse Monitoring. It covers collecting inference logs, creating a structured monitoring table, scheduling refreshes, and building a dashboard to visualize metrics and detect drift. ### Source excerpt Lecture 10 of MLOps with Databricks course ## Deploying a model serving endpoint DevFeed: [Deploying a model serving endpoint]() Original publisher: [Read original article]() Author: Başak Tuğçe Eskili Published: 2025-08-02T11:00:01Z Content type: tutorial Language: en Sources: [MarvelousMLOps]() Topics: [databricks](), [MLOps](), [model-serving](), [model-deployment](), [Deployment](), [autoscaling](), [API](), [A/B Testing]() Tags: [a-b-testing](), [api](), [article](), [autoscaling](), [databricks](), [deployment](), [latency](), [mlops](), [model-serving]() ### AI overview Lecture 6 in an MLOps with Databricks course explains how to deploy registered MLflow models as Databricks Model Serving REST APIs. It covers serverless deployment, automatic scaling, monitoring, Unity Catalog integration, workload sizing, and limitations such as restricted runtime and cluster control. ### Source excerpt Lecture 6 of MLOps with Databricks course ## Model serving architectures DevFeed: [Model serving architectures]() Original publisher: [Read original article]() Author: Maria Vechtomova Published: 2025-08-01T17:39:58Z Content type: tutorial Language: en Sources: [MarvelousMLOps]() Topics: [databricks](), [MLOps](), [model-serving](), [model-deployment](), [Deployment](), [Python](), [Low Latency](), [SDK](), [real-time]() Tags: [batch](), [databricks](), [deployment](), [latency](), [mlops](), [model-serving](), [python](), [real-time](), [sdk]() ### AI overview Lecture 5 in an MLOps with Databricks course explains model serving architectures on Databricks. It covers batch prediction and feature serving, model serving, and model serving with feature lookup, including serverless infrastructure and endpoint-based access. ### Source excerpt Lecture 5 of MLOps with Databricks course ## How to Test Databricks Model Deployments Locally DevFeed: [How to Test Databricks Model Deployments Locally]() Original publisher: [Read original article]() Author: Mehmet Acikgoz Published: 2025-05-01T16:56:46Z Content type: tutorial Language: en Sources: [MarvelousMLOps]() Topics: [model-serving](), [databricks](), [DevOps](), [Deployment](), [Testing](), [SDK](), [Command-line interface]() Tags: [cli](), [databricks](), [deployment](), [devops](), [ml](), [model-deployment](), [model-serving](), [sdk](), [testing]() ### AI overview This tutorial explains how machine learning teams can test Databricks model-serving deployments locally before deploying them remotely. It covers downloading a pyfunc model artifact tracked in MLflow, deploying the endpoint locally, and testing it with the Iris-Species dataset. ### Source excerpt Test Databricks model deployments locally