# model-serving Deployment of trained machine-learning models to serve inference requests. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Introducing Amazon SageMaker HyperPod Inference Gateway DevFeed: [Introducing Amazon SageMaker HyperPod Inference Gateway]() Original publisher: [Read original article]() Author: Vinay Arora Published: 2026-09-18T13:08:34Z Content type: release Language: en Sources: [Artificial Intelligence]() Topics: [Amazon SageMaker HyperPod](), [Model Routing](), [model-serving](), [Inference Performance](), [Amazon Elastic Kubernetes Service](), [Kubernetes](), [AI Inference](), [GPU](), [AI Platform](), [Prometheus]() Tags: [amazon-eks](), [amazon-sagemaker-hyperpod](), [announcements](), [expert-400](), [gpu](), [inference](), [kubernetes](), [latency](), [metrics](), [model-serving](), [performance](), [routing]() ### AI overview Amazon SageMaker HyperPod Inference Gateway is a Kubernetes-native, GPU-aware routing add-on for Amazon EKS. It uses real-time GPU signals and model-serving metrics to route inference requests to suitable pods, aiming to reduce GPU waste and first-token latency without application changes. ### Source excerpt Amazon SageMaker HyperPod Inference Gateway is a Kubernetes-native, GPU-aware routing add-on for Amazon EKS. It uses real-time GPU signals to send each inference request to the best-suited pod, cutting first-token latency by up to 82% with no changes to your model servers or client applications. ## Self-Hosted Platform Build Order: Dependencies from Bare Metal to Model Serving DevFeed: [Self-Hosted Platform Build Order: Dependencies from Bare Metal to Model Serving]() Original publisher: [Read original article]() Author: Philipp Lehmann (philipp.lehmann@gruppe.ai) Published: 2026-09-04T07:00:00Z Content type: article Language: en Sources: [Philipp Lehmann]() Topics: [Self-hosted](), [Ansible](), [Kubernetes](), [ceph](), [observability](), [model-serving](), [Compose]() Tags: [ansible](), [architecture](), [article](), [ceph](), [compose](), [infrastructure-as-code](), [kubernetes](), [model-serving](), [observability](), [self-hosted]() ### AI overview This article explains the dependency order for building a self-hosted platform. It covers consistent bare-metal inventory, Ansible configuration convergence, networking, Kubernetes, Ceph storage, identity, observability, and model serving. ### Source excerpt Self-hosted platform build order: why mesh, cluster, Ceph storage, identity and observability must precede model serving, shown with Compose depends_on. ## Equinix Inference Exchange Brings NVIDIA Compute and 200+ Open Models Closer to Enterprise Data DevFeed: [Equinix Inference Exchange Brings NVIDIA Compute and 200+ Open Models Closer to Enterprise Data]() Original publisher: [Read original article]() Author: Harold Fritts Published: 2026-09-03T16:22:15Z Content type: news Language: en Sources: [StorageReview.com]() Topics: [AI Inference](), [Inference Performance](), [AI Infrastructure](), [model-serving](), [Open Source Models & Datasets](), [data centers](), [networking]() Tags: [ai](), [ai-inference](), [ai-infrastructure](), [architectures](), [data-centers](), [enterprise](), [hardware](), [inference](), [latency](), [networking](), [open-source](), [production]() ### AI overview Equinix Inference Exchange is a distributed AI inference platform that places NVIDIA compute and Together AI's open-model serving closer to enterprise data, users, and applications. It combines Equinix's interconnection infrastructure, NVIDIA hardware, and support for more than 200 open-source models to address latency, data sovereignty, networking complexity, and inference costs. ### Source excerpt Equinix has expanded its partnership with NVIDIA and entered a new collaboration with Together AI to launch Equinix Inference Exchange. Designed as a distributed AI inference architecture for enterprise deployments, the platform aims to shift compute workloads closer to core data repositories, end users, and operational applications. Announced alongside Equinix Fabric One at the Equinix The post Equinix Inference Exchange Brings NVIDIA Compute and 200+ Open Models Closer to Enterprise Data appeared first on StorageReview.com. ## Self-Hosted LLM Inference: Serving, Benchmarking and Agent Guardrails DevFeed: [Self-Hosted LLM Inference: Serving, Benchmarking and Agent Guardrails]() Original publisher: [Read original article]() Author: Philipp Lehmann (philipp.lehmann@gruppe.ai) Published: 2026-08-07T07:00:00Z Content type: article Language: en Sources: [Philipp Lehmann]() Topics: [Self-hosted](), [Large Language Model](), [Inference](), [model-serving](), [benchmarking](), [LLM evaluation / benchmarking](), [API]() Tags: [agents](), [api](), [benchmarking](), [guardrails](), [inference](), [llm](), [ollama](), [security](), [self-hosted](), [testing]() ### AI overview This article examines self-hosted LLM inference for infrastructure work, focusing on data locality, local-versus-hosted model tradeoffs, repeatable benchmarking on personal tasks, model serving, and guardrails for agents that can propose or apply changes. ### Source excerpt Ollama OpenAI-compatible /v1/chat/completions: a repeatable benchmark on your own tasks and a dry-run plus --approve guardrail for infrastructure agents. ## Using ArgoCD to Manage AI Model Deployments with GitOps DevFeed: [Using ArgoCD to Manage AI Model Deployments with GitOps]() Original publisher: [Read original article]() Author: Pramodh Kumar M Published: 2026-07-28T18:24:57Z Content type: tutorial Language: en Sources: [Kubernetes - KodeKloud Blog | DevOps, Cloud, Kubernetes, AI Tutorials & More]() Topics: [GitOps](), [argocd](), [model-deployment](), [Artificial Intelligence](), [model-serving](), [Architecture & Design]() Tags: [ai](), [ai-models](), [argo-rollouts-model-promotion](), [argocd](), [argocd-ai-model-deployment](), [argocd-applicationset](), [argocd-sync-waves](), [automation](), [ci-cd](), [declarative-model-serving](), [deployment](), [devops](), [gitops](), [gitops-for-ai-model-deployments](), [gitops-mlops](), [kserve](), [kserve-canary-deployment](), [kubernetes](), [kubernetes-model-deployment-rollback](), [ml-model-deployment](), [ml-model-versioning](), [mlops](), [model-deployment](), [model-registry-to-git-automation](), [model-serving](), [pull-requests](), [standard](), [tutorial]() ### AI overview This tutorial explains how to use ArgoCD and GitOps to deploy AI models while accounting for model weights stored outside container images. It covers pinned model references, rollback and recovery, synchronization ordering, readiness probes, evaluation-based promotion gates, and KServe status handling. ### Source excerpt Your microservices deploy through pull requests with full audit trails. Your models deploy because someone ran a script. Here is how to close that gap with ArgoCD, and what changes when the artifact weighs four gigabytes. ## KubeCon + CloudNativeCon India 2026: Our Kubeflow Community Experience DevFeed: [KubeCon + CloudNativeCon India 2026: Our Kubeflow Community Experience]() Original publisher: [Read original article]() Author: Khushi Agrawal Published: 2026-07-27T05:00:00Z Content type: article Language: en Sources: [Kubeflow]() Topics: [Kubernetes](), [MLOps](), [distributed-training](), [model-serving](), [Large Language Model](), [Retrieval Augmented Generation (RAG)](), [GPU](), [Cloud]() Tags: [2026](), [ai](), [cloud-native](), [community](), [distributed-training](), [gpu](), [india](), [kubecon](), [kubernetes](), [llms](), [machine-learning](), [mlops](), [model-serving](), [model-training](), [rag]() ### AI overview A Kubeflow community recap of KubeCon + CloudNativeCon India 2026 in Mumbai. It describes Kubeflow's presence at the CNCF Project Pavilion, common architecture questions, featured talks, booth discussions about production machine learning workflows, distributed training, model serving, GPU resource allocation, LLMs, and RAG pipelines, and efforts to onboard contributors. ### Source excerpt Introduction ## Building a Minimal Control Plane to Reconcile Docker Containers for Model Serving DevFeed: [Building a Minimal Control Plane to Reconcile Docker Containers for Model Serving]() Original publisher: [Read original article]() Published: 2026-07-02T00:00:00Z Content type: tutorial Language: en Sources: [Alex Korbonits]() Topics: [control-plane](), [model-serving](), [Docker Container](), [Amazon Machine Learning](), [Streaming]() Tags: [container](), [control-plane](), [inference](), [model-serving](), [python]() ### AI overview The author explains control planes by building a small Python-based system that maintains a declared number of Docker containers for model serving. The system repeatedly compares desired and actual state and reconciles differences, including restoring a container after it stops. ### Source excerpt I can design batch, real-time, and streaming inference -- the data plane. But I'd never built the control plane that manages it. So I built the smallest one I could, from scratch, to finally understand reconciliation: the one idea that separates a control plane from a deploy script. ## Fully Automated AI Inference on AWS, Azure, and Google Cloud with Pulumi DevFeed: [Fully Automated AI Inference on AWS, Azure, and Google Cloud with Pulumi]() Original publisher: [Read original article]() Author: Engin Diri Published: 2026-06-30T00:00:00Z Content type: tutorial Language: en Sources: [Pulumi]() Topics: [Ollama](), [Cloud](), [Infrastructure as code](), [model-serving](), [GPU](), [Artificial Intelligence](), [OpenID connect (OIDC)](), [Ubuntu]() Tags: [ai](), [aws](), [azure](), [cloud](), [gcp](), [google-cloud](), [gpu](), [inference](), [infrastructure-as-code](), [model-serving](), [oidc](), [ollama](), [pulumi-esc](), [tutorials](), [typescript](), [ubuntu]() ### AI overview This tutorial shows how to use Pulumi to provision GPU virtual machines on AWS, Azure, and Google Cloud, install drivers, run Ollama, and pull an open-source model without manual setup. It also uses Pulumi ESC with OIDC instead of a long-lived access key and avoids treating model-download completion as an infrastructure resource. ### Source excerpt Putting Ollama on a cloud GPU is something I keep coming back to. A while ago I wrote up running open-source LLMs on an AWS EC2 box with Ollama and Pulumi, and the shape never really changes: a GPU instance, a model server, and a firewall rule in front. Infrastructure as code earned its place by making that kind of setup predictable and repeatable, and AI infrastructure is no exception. A GPU box serving a model is still a VM, a disk, and a firewall rule, and it should be declared like one. Thorsten Hans made exactly that case in his Akamai post, Fully Automated AI Infrastructures with Terraform and Akamai Cloud, which stands up a single GPU instance on Linode, installs the drivers, runs Ollama, and pulls a model, with no manual steps after terraform apply. I liked the shape of it, so this post ports the same idea to Pulumi and runs it across AWS, Azure, and Google Cloud instead of one. The result is one program shape per cloud: a single pulumi up brings up a GPU box that installs its own driver, runs Ollama, and pulls a model with no manual steps, and a single pulumi destroy takes it back down. Along the way it drops the two imperative bits the Terraform version leans on: a static access token sitting in an environment variable, and a null_resource running a shell loop to wait for the model. The first becomes an OIDC login from a Pulumi ESC environment, so no long-lived key lives anywhere. The second turns out not to be a resource at all. What you are building Strip away the per-cloud naming and every version of this is the same three things: a GPU virtual machine, a firewall in front of it, and a cloud-init script that turns a bare Ubuntu box into a running inference server. The model serving runs on Ollama, which exposes an HTTP API on port 11434 and keeps the model resident in GPU memory between requests. flowchart LR Dev([Your machine / curl]) -->|"HTTP :11434"| FW["Firewall / security group (allow 11434, optional 22)"] FW --> VM["GPU VM (Ubuntu 24.04) ## I Built a 256GB Local AI Cluster on My Desk DevFeed: [I Built a 256GB Local AI Cluster on My Desk]() Original publisher: [Read original article]() Author: Techno Tim Published: 2026-05-18T13:00:00Z Content type: article Language: en Sources: [Techno Tim]() Topics: [Local AI](), [Artificial Intelligence](), [agentic-coding](), [Hardware](), [Homelab](), [coding](), [model-serving](), [vllm](), [networking](), [Docker](), [NCCL](), [Monitoring]() Tags: [agentic-coding](), [ai](), [docker](), [github](), [hardware](), [homelab](), [local-ai](), [model-serving](), [monitoring](), [nccl](), [networking](), [self-hosted](), [self-hosted-ai](), [server](), [vllm]() ### AI overview The article tests a two-node local AI cluster built from ASUS Ascent GX10 systems, using one machine for practical agentic coding and two machines to run a larger local model. It evaluates a real coding workflow involving model serving, storage, networking, memory, Docker, NCCL, monitoring, power draw, and the application being built, concluding that local AI is capable and cloud-independent but still operationally complex. ### Source excerpt I have been covering local and self-hosted AI for a few years now - from running models privately at home to what is still running in my homelab today. But to run the larger, more capable models, you need something more specialized than a general-purpose home server. I wanted to know how good local AI has actually gotten, so I built a mini AI cluster on my desk and used it for a real coding wo... ## DigitalOcean Dedicated Inference: A Technical Deep Dive DevFeed: [DigitalOcean Dedicated Inference: A Technical Deep Dive]() Original publisher: [Read original article]() Author: dgupta Published: 2026-04-25T02:51:09Z Content type: article Language: en Sources: [DigitalOcean]() Topics: [AI Platform](), [Digital Ocean](), [Large Language Model](), [model-serving](), [GPU](), [autoscaling](), [Kubernetes](), [VPC]() Tags: [ai-platform](), [architecture](), [autoscaling](), [deep-dive](), [digitalocean](), [engineering](), [gpu](), [inference](), [kubernetes](), [llm](), [model-serving](), [production](), [technical]() ### AI overview This technical deep dive explains DigitalOcean Dedicated Inference, a managed LLM hosting service on dedicated GPUs. It describes the Kubernetes-native inference stack, public and private VPC endpoints, OpenAI-compatible APIs, serving and routing components, autoscaling, observability, and cost considerations for sustained, high-volume workloads. ### Source excerpt Getting a model to answer 10 inference requests concurrently is tricky but simple enough; getting it to handle 2,000 engineers hitting a coding assistant with long contexts, all day, without runaway costs, is where teams stall. A working endpoint is only the beginning. Teams need to identify the supporting hardware and wire up the right components--serving, scaling, observability, and cost guardrails--so the deployment can support expected SLAs and SLOs under real, sustained load. DigitalOcean already offers Serverless Inference on the DigitalOcean AI Platform: a fast path to models from OpenAI, Anthropic, Meta, or other providers, with minimal setup and token-based pricing. This offering works well for many use cases. However, when you need your own weights, predictable performance on dedicated GPUs, and economics that favor sustained, high-volume token generation over pay-per-token bursts, a different approach makes sense Dedicated Inference, our managed LLM hosting service on the DigitalOcean AI Platform, fills that gap. Dedicated Inference deploys and operates an opinionated inference stack on dedicated GPUs, with Kubernetes-native orchestration under the hood. You interact through the control plane and APIs you already use in the DigitalOcean ecosystem; the data plane exposes public and private endpoints so applications inside, or outside, your VPC can call your models securely. The service is designed to collapse a vast combinatorial space--GPU SKUs, runtimes, routers, autoscaling policies--into guided defaults so teams hit production milestones faster than DIY stacks, while retaining knobs that matter for model serving: replicas, scaling behavior, and advanced optimizations as you roll out your product roadmap. What we manage vs. what you control Every managed product draws a line between operator-owned and customer-owned concerns. Dedicated Inference aims to put day-two operations--cluster lifecycle integration, ingress, core serving and routing components, and t ## Migrating to Nvidia Triton: High-Throughput, Low-Cost Inference at Scale DevFeed: [Migrating to Nvidia Triton: High-Throughput, Low-Cost Inference at Scale]() Original publisher: [Read original article]() Author: Eric Luu Published: 2025-09-19T01:05:13Z Content type: article Language: en Sources: [GumGum]() Topics: [Inference](), [model-serving](), [Nvidia](), [Machine learning](), [Natural language processing](), [databricks](), [Docker](), [Kafka](), [Python]() Tags: [batching](), [databricks](), [docker](), [gpu](), [inference](), [kafka](), [latency](), [machine-learning](), [nvidia](), [optimization](), [performance](), [triton-inference-server]() ### AI overview This article describes GumGum's migration of NLP inference from Databricks-based jobs using Docker images, Kafka streaming, and UDFs toward NVIDIA Triton. It introduces Triton's model-serving features, including batching, ensembles, dynamic model loading, request scheduling, and reduced GPU-CPU data transfers, while outlining limitations of the previous approach such as latency, scalability, GPU-metrics, and PySpark performance issues. ### Source excerpt Photo by Igor Omilaev on Unsplash There are tons of guides on what Nvidia Triton is and how to set it up, so I'll only go over those very briefly, instead focusing on the migration we've had and the small quirks that come from working with Python Backends. Nvidia Triton is an open source inference serving software, capable of serving virtually any machine learning model. Real time processing, batching, dynamic batching, ensembles, specialized backends, all possible with enough knobs and dials to maximize hardware use and minimize cost. Core features include File based model repository to store your models. If wanted, can be dynamically loaded onto the server Ensemble capabilities, can define whole inference pipelines, even ones that branch, on triton Ability to define each step in the inference pipeline as its own model (e.g., tokenizer -> model -> decoder), which can get hit as its own endpoint Dynamic batching between models, even in ensembles Automatic scheduling of requests between models and ensembles Minimal data transfers between GPU & CPU Our Previous Solution At GumGum we've previously deployed the bulk of our NLP inference on Databricks. These would run 24 hour jobs loading custom Docker images, streaming inputs from a Kafka topic and relying on UDF's to contain our inference logic. This was a move to mitigate issues we had on ECS related to starvation, scalability, and parallelism. Workers being paired on consumers lead to starvation. As a result, scalability of the service suffered and maintaining threading code became a pain. We've also experienced other issues as well: GPU metrics were not easily available, as we need to use an ML runtime, which was not compatible with docker containers (Although modern runtime do support this more). Latency was high as with Kafka structured streaming, microbatches must wait for all workers to finish. So if one worker hangs, the entire batch hangs. Performance on PySpark was always tough. Serializing between Java to Pyth ## Implementing Model Monitoring on Databricks DevFeed: [Implementing Model Monitoring on Databricks]() Original publisher: [Read original article]() Author: Başak Tuğçe Eskili Published: 2025-08-06T16:55:23Z Content type: tutorial Language: en Sources: [MarvelousMLOps]() Topics: [databricks](), [MLOps](), [Monitoring](), [model-serving](), [dashboards](), [Inference](), [Logging](), [Data Quality]() Tags: [databricks](), [metrics](), [mlops](), [model-monitoring](), [model-serving](), [monitoring](), [request](), [table]() ### AI overview Lecture 10 in an MLOps with Databricks course demonstrates model monitoring using inference tables and Lakehouse Monitoring. It covers collecting inference logs, creating a structured monitoring table, scheduling refreshes, and building a dashboard to visualize metrics and detect drift. ### Source excerpt Lecture 10 of MLOps with Databricks course ## Deploying a model serving endpoint DevFeed: [Deploying a model serving endpoint]() Original publisher: [Read original article]() Author: Başak Tuğçe Eskili Published: 2025-08-02T11:00:01Z Content type: tutorial Language: en Sources: [MarvelousMLOps]() Topics: [databricks](), [MLOps](), [model-serving](), [model-deployment](), [Deployment](), [autoscaling](), [API](), [A/B Testing]() Tags: [a-b-testing](), [api](), [article](), [autoscaling](), [databricks](), [deployment](), [latency](), [mlops](), [model-serving]() ### AI overview Lecture 6 in an MLOps with Databricks course explains how to deploy registered MLflow models as Databricks Model Serving REST APIs. It covers serverless deployment, automatic scaling, monitoring, Unity Catalog integration, workload sizing, and limitations such as restricted runtime and cluster control. ### Source excerpt Lecture 6 of MLOps with Databricks course ## Model serving architectures DevFeed: [Model serving architectures]() Original publisher: [Read original article]() Author: Maria Vechtomova Published: 2025-08-01T17:39:58Z Content type: tutorial Language: en Sources: [MarvelousMLOps]() Topics: [databricks](), [MLOps](), [model-serving](), [model-deployment](), [Deployment](), [Python](), [Low Latency](), [SDK](), [real-time]() Tags: [batch](), [databricks](), [deployment](), [latency](), [mlops](), [model-serving](), [python](), [real-time](), [sdk]() ### AI overview Lecture 5 in an MLOps with Databricks course explains model serving architectures on Databricks. It covers batch prediction and feature serving, model serving, and model serving with feature lookup, including serverless infrastructure and endpoint-based access. ### Source excerpt Lecture 5 of MLOps with Databricks course ## How to Test Databricks Model Deployments Locally DevFeed: [How to Test Databricks Model Deployments Locally]() Original publisher: [Read original article]() Author: Mehmet Acikgoz Published: 2025-05-01T16:56:46Z Content type: tutorial Language: en Sources: [MarvelousMLOps]() Topics: [model-serving](), [databricks](), [DevOps](), [Deployment](), [Testing](), [SDK](), [Command-line interface]() Tags: [cli](), [databricks](), [deployment](), [devops](), [ml](), [model-deployment](), [model-serving](), [sdk](), [testing]() ### AI overview This tutorial explains how machine learning teams can test Databricks model-serving deployments locally before deploying them remotely. It covers downloading a pyfunc model artifact tracked in MLflow, deploying the endpoint locally, and testing it with the Iris-Species dataset. ### Source excerpt Test Databricks model deployments locally ## PipelineRL DevFeed: [PipelineRL]() Original publisher: [Read original article]() Author: Alex Piche; Rafael Pardinas; Ehsan Kamalloo; Bahdanau Published: 2025-04-25T22:37:16Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [grpo](), [model-serving](), [gradient\_accumulation](), [GPU](), [datasets](), [math](), [Optimization]() Tags: [batch](), [benchmarks](), [blog-post](), [gpu](), [grpo](), [optimization](), [performance](), [rl](), [training]() ### AI overview This blog post presents PipelineRL, a modular reinforcement-learning architecture for training language models. It uses inflight weight updates to preserve efficient inference batching while keeping collected data on-policy or near on-policy. The article reports competitive results against Open-Reasoner-Zero on AIME 2024 and MATH 500 using a simpler GRPO-based implementation. ### Source excerpt In this blog post, we show that 1) inflight weight updates do not harm the training process and 2) PipelineRL achieves competitive results compared to Open-Reasoner-Zero, while using a simpler RL algorithm. We also present the modular PipelineRL architecture that facilitates trying new inference / trainer combinations. In conventional RL approaches (Figure 1a), there is a trade-off between high throughput inference and on-policy data collection. ## Serverless Inference with Hugging Face and NVIDIA NIM DevFeed: [Serverless Inference with Hugging Face and NVIDIA NIM]() Original publisher: [Read original article]() Author: Philipp Schmid; Jeff Boudier Published: 2024-07-29T00:00:00Z Content type: release Language: en Sources: [Hugging Face - Blog]() Topics: [NVIDIA NIM](), [hugging face](), [Inference](), [model-serving](), [Serverless](), [DGX Cloud](), [Large Language Model](), [Generative AI](), [API](), [llama](), [Access Control](), [Claude]() Tags: [access-control](), [api](), [cloud](), [dgx-cloud](), [generative-ai](), [hardware](), [hugging-face](), [inference](), [llama](), [llm](), [nvidia](), [nvidia-dgx](), [nvidia-nim](), [partnerships](), [pricing](), [serverless]() ### AI overview Hugging Face announces a serverless NVIDIA NIM API on the Hugging Face Hub for Enterprise Hub organizations. The service provides API access to open generative AI models, including Llama and Mistral, on NVIDIA DGX Cloud infrastructure, with pay-as-you-go pricing and a guide for creating fine-grained organization tokens. The article notes that the service was deprecated and unavailable as of April 10, 2025. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Unlocking Real-time Predictions with Shopify's Machine Learning Platform DevFeed: [Unlocking Real-time Predictions with Shopify's Machine Learning Platform]() Original publisher: [Read original article]() Author: Isaac Vidas Published: 2023-03-10T16:16:27Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [Shopify](), [Machine Learning & Artificial Intelligence](), [model-serving](), [Low-Latency Inference](), [real-time](), [model-deployment](), [Deployment]() Tags: [deployment](), [inference](), [low-latency](), [machine-learning](), [machine-learning-pipelines](), [real-time](), [shopify]() ### AI overview Shopify describes how it expanded Merlin, its machine learning platform built on Ray, with online inference capabilities for deploying and serving machine learning models that generate real-time predictions at scale. The article explains the platform's serving layer, service deployment, low-latency requirements, and distinction between batch and online inference. ### Source excerpt Learn how Shopify Data built new online inference capabilities into its Machine Learning Platform to deploy and serve models for real-time prediction at scale.