# Low-Latency Inference The practice of minimizing the time between presenting input to a deployed machine-learning model and receiving its output. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## AI Infra Summit: NVIDIA Vera Rubin and DSX Platform Advancements Showcase Energy Efficiencies of Optimizing Tokens Per Watt for AI Factories DevFeed: [AI Infra Summit: NVIDIA Vera Rubin and DSX Platform Advancements Showcase Energy Efficiencies of Optimizing Tokens Per Watt for AI Factories]() Original publisher: [Read original article]() Author: NVIDIA Writers Published: 2026-09-15T16:55:40Z Content type: news Language: en Sources: [NVIDIA Blog]() Topics: [AI Infrastructure](), [Nvidia](), [DSX](), [Vera Rubin](), [Low-Latency Inference](), [NVLink](), [Conversational AI]() Tags: [ai](), [ai-infrastructure](), [dsx](), [inference](), [infra](), [latency](), [low-latency-inference](), [networking](), [nvidia](), [nvidia-blackwell](), [nvidia-dsx](), [nvidia-vera](), [nvidia-vera-rubin](), [nvlink]() ### AI overview NVIDIA's AI Infra Summit coverage describes collaborations and platform updates focused on improving AI factory efficiency. The article highlights Vera Rubin systems, DSX MaxLPS, Dynamo inference software, NVLink and networking technologies, including claims of up to 1.4x more tokens per megawatt through factory-wide power optimization. ### Source excerpt Ian Buck, vice president of hyperscale and high-performance computing at NVIDIA, Tuesday spoke on AI factory efficiency at the AI Infra Summit, the Santa Clara Convention Center event that has morphed into a Coachella of infrastructure tech. Before a packed audience -- with more than 8,000 attendees this year, up from 3,500 last year -- [...] ## Chip Huyen explains how to cut inference costs without new hardware DevFeed: [Chip Huyen explains how to cut inference costs without new hardware]() Original publisher: [Read original article]() Author: Tim Koopmans Published: 2026-09-13T15:00:00Z Content type: article Language: en Sources: [The New Stack]() Topics: [Inference Performance](), [Low-Latency Inference](), [model-deployment](), [Optimization](), [Frontier Model](), [AI Engineering](), [Latency](), [math]() Tags: [ai](), [ai-agents](), [ai-engineering](), [ai-infrastructure](), [frontier-model](), [inference](), [low-latency](), [metrics](), [optimization](), [performance](), [post-contributed](), [reasoning](), [scylladb](), [sponsor-scylladb](), [sponsored](), [sponsored-post-contributed](), [tokens]() ### AI overview Chip Huyen explains why inference costs can outweigh one-time frontier-model training costs and outlines ways to optimize inference without new hardware. The article emphasizes latency metrics such as time to first token, time per output token, end-to-end latency, and goodput, especially for reasoning models. ### Source excerpt Last October, the P99 conference -- the online gathering for developers focused on high-performance, low-latency applications -- featured a cracking The post Chip Huyen explains how to cut inference costs without new hardware appeared first on The New Stack. ## Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference DevFeed: [Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference]() Original publisher: [Read original article]() Author: Kareem Syed-Mohammed Published: 2026-09-10T21:58:09Z Content type: release Language: en Sources: [Artificial Intelligence]() Topics: [Large Language Model](), [Low-Latency Inference](), [Caching](), [Inference](), [Inference Performance]() Tags: [advanced-300](), [amazon-sagemaker](), [amazon-sagemaker-ai](), [announcements](), [benchmarks](), [caching](), [inference](), [latency](), [llama](), [llm](), [routing](), [tensorrt-llm](), [vllm]() ### AI overview Amazon SageMaker Inference introduces prefix-aware routing for LLM requests. By consistently sending requests with matching prompt prefixes to the same instance, it improves reuse of cached KV computations and can reduce time to first token. ### Source excerpt Amazon SageMaker Inference now offers prefix-aware routing, a routing strategy that sends requests sharing the same prompt prefix to the same instance so the KV cache stays warm. In benchmarks on Llama 3.1 70B, it reduced P50 time-to-first-token by up to 77% and raised KV cache hit rates from about 25% to over 80%. ## The full stack behind abundant intelligence DevFeed: [The full stack behind abundant intelligence]() Original publisher: [Read original article]() Published: 2026-08-25T07:05:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [OpenAI](), [Artificial Intelligence](), [Hardware](), [Low-Latency Inference](), [Inference Performance](), [model-deployment](), [gpt-oss](), [Benchmark](), [cloud-infrastructure](), [systems](), [Amazon Web Services](), [Microsoft]() Tags: [ai](), [aws](), [benchmark](), [company](), [compute](), [cost](), [efficiency](), [energy-efficiency](), [hardware](), [inference](), [low-latency-inference](), [microsoft](), [models](), [nvidia](), [openai]() ### AI overview OpenAI describes an integrated compute strategy spanning data centers, chips, models, software, products, and devices. It reports that its custom Jalapeño inference chip achieved higher peak throughput per kilowatt and lower token latency than commercial systems on the InferenceX benchmark using GPT-OSS 120B, while also performing strongly on DeepSeek R1 and Kimi K2. ### Source excerpt OpenAI CFO Sarah Friar explains how advances across chips, compute, models, and products compound to deliver more useful intelligence at greater scale and lower cost. ## With Groq 3 LPX in Full Production, NVIDIA Extends Vera Rubin Inference for Agents DevFeed: [With Groq 3 LPX in Full Production, NVIDIA Extends Vera Rubin Inference for Agents]() Original publisher: [Read original article]() Author: NVIDIA Writers Published: 2026-08-24T15:00:41Z Content type: news Language: en Sources: [NVIDIA Blog]() Topics: [Inference Performance](), [long-context](), [Low-Latency Inference](), [networking](), [gemma4]() Tags: [agentic](), [agents](), [ai-factory](), [ai-inference](), [ai-infrastructure](), [benchmark](), [bluefi](), [cloud](), [cpx](), [groq](), [groq-3-lpx](), [inference](), [infrastructure](), [latency](), [long-context](), [low-latency-inference](), [lpx](), [networking](), [nvidia](), [nvidia-spectrum-x-ethernet](), [nvidia-vera](), [nvidia-vera-rubin](), [nvl72](), [nvlink](), [vera-cpu](), [vera-rubin]() ### AI overview NVIDIA announces Groq 3 LPX production availability alongside Vera Rubin NVL72, positioning the integrated compute, networking, and inference stack for long-context, agentic AI workloads. The article highlights a benchmark result of 3,400 output tokens per second on Gemma 4 31B and describes partner adoption of Vera Rubin, Spectrum-X Multiplane, and Groq 3 LPX. ### Source excerpt The next era of AI inference won't be defined by a single breakthrough chip, network or system. It'll be defined by how every layer of the AI factory works together. That's why NVIDIA is extending Vera Rubin NVL72 with fast token generation for agentic systems. Announced today, the NVIDIA Vera Rubin rack-scale system NVIDIA Groq [...] ## Open-weight models are fast on Neon AI Gateway. Here's why DevFeed: [Open-weight models are fast on Neon AI Gateway. Here's why]() Original publisher: [Read original article]() Author: Carlota Soto Published: 2026-08-20T12:00:00Z Content type: article Language: en Sources: [Blog -- Neon Docs]() Topics: [databricks](), [Inference Performance](), [model-deployment](), [Low-Latency Inference](), [Caching](), [Mixture of Experts (MoE)](), [gpt-oss](), [Large Language Model]() Tags: [caching](), [databricks](), [gpt-oss](), [inference](), [mixture-of-experts-moe](), [optimization](), [performance](), [product](), [speed]() ### AI overview The article explains why open-weight models can deliver low latency on Neon AI Gateway when paired with Databricks Foundation Model APIs and an optimized serving stack. It highlights continuous batching, KV-cache paging, custom kernels, quantization, multi-GPU layouts, prompt caching, and Mixture-of-Experts architectures. ### Source excerpt Open-weight models are built for performance, but how they're served determines whether end users actually feel that speed. Neon AI Gateway calls Databricks Foundation Model APIs for Databricks-hosted open-weight models, a stack that has years of inference engineering behind it. ## Gisting: Compressing LLM Agent context to ↑ throughput and ↓ cost DevFeed: [Gisting: Compressing LLM Agent context to ↑ throughput and ↓ cost]() Original publisher: [Read original article]() Author: Cody Mazza-Anthony Published: 2026-08-19T14:32:58Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [Compression](), [Large Language Model](), [Embeddings](), [Latency](), [Inference](), [Inference Performance](), [Low-Latency Inference](), [Post-training optimization](), [GraphQL](), [model-deployment]() Tags: [compression](), [embedding](), [inference](), [latency](), [llm](), [model](), [tokens]() ### AI overview Gisting compresses an LLM agent's system prompt into learned gist tokens, preserving prediction quality while reducing inference latency, increasing throughput, and lowering GPU requirements. ### Source excerpt Gisting compresses context into a set of learned tokens, preserving its quality while making the model faster and cheaper. ## Inference latency: what it measures & why it varies DevFeed: [Inference latency: what it measures & why it varies]() Original publisher: [Read original article]() Author: Jeff Mills Published: 2026-08-06T00:00:00Z Content type: tutorial Language: en Sources: [Redis Blog]() Topics: [Inference Performance](), [Low-Latency Inference](), [Language models]() Tags: [agent](), [benchmarks](), [gpu](), [inference](), [latency](), [llm](), [memory](), [metrics](), [model](), [production](), [redis](), [retrieval](), [tech-de]() ### AI overview A guide to LLM inference latency, distinguishing first-token time, per-token time, total response time, and multi-step agent run time. It contrasts latency with throughput and explains how prompt prefill, token decoding, queueing, and tail latency affect user experience. ### Source excerpt Ask an engineer what their LLM app's inference latency is, and the honest answer is "which one?" The time to the first visible token, the time to the finished response, and the time an agent spends across a chain of calls are three different numbers. ... ## Как оптимизировать инференс LLM: кеширование, время ответа и GPU-ресурсы DevFeed: [Как оптимизировать инференс LLM: кеширование, время ответа и GPU-ресурсы]() Original publisher: [Read original article]() Author: hotckisss (Яндекс, Yandex Cloud & Yandex Infrastructure) Published: 2026-07-08T07:04:08Z Content type: tutorial Language: ru Sources: [Яндекс - Как мы делаем Яндекс / Статьи]() Topics: [Large Language Model](), [Inference Performance](), [Low-Latency Inference](), [GPU](), [Artificial Intelligence]() Tags: [ai](), [inference](), [llm](), [ml](), [mlops](), [tag-b92bf5906bbd](), [time](), [token]() ### AI overview This article explains how to optimize LLM inference in production under mixed workloads. It examines causes of increased Time to First Token, including scheduling, GPU memory allocation, streaming output queues, networking, long contexts, and KV-cache growth, and outlines optimizations such as separating prefill and decode, contextual parallelism, speculative decoding, cache-aware load balancing, and large-model delivery. ### Source excerpt Вы запустили LLM-инференс в продакшене. Поток запросов не менялся, нагрузка та же, что вчера, -- а Time to First Token внезапно вырос в три раза. Первая мысль: что-то с моделью. На деле причина почти никогда не в модели -- она прячется в планировщике, аллокаторе GPU-памяти, очереди стримингового вывода или сети. Чем длиннее контекст, тем больнее. Для классического attention вычислительная сложность растёт очень быстро. KV-кеш раздувается до десятков и сотен гигабайт, а в облаке всё это происходит на совершенно произвольном трафике: у одного клиента кодовый ассистент, у другого -- аналитика на миллион запросов в день, у третьего -- голосовой робот. На таких смешанных нагрузках всплывает то, чего не видно на референсных замерах вендора. Читать далее ## For whom the door-bell tolls DevFeed: [For whom the door-bell tolls]() Original publisher: [Read original article]() Author: Kyle Bader Published: 2026-06-11T00:00:00Z Content type: article Language: en Sources: [Ceph Blog]() Topics: [Caching](), [Large Language Model](), [Inference Performance](), [Low-Latency Inference](), [GPU](), [cpu]() Tags: [2025](), [ai](), [architecture](), [blog-post](), [cache](), [caching](), [ceph](), [cpu](), [en-article](), [en-blog-post](), [gpu](), [gpu-initated](), [inference](), [llm](), [nvme](), [rados](), [rados-nkv](), [storage]() ### AI overview The article explores using Ceph RADOS through NVMe key-value commands to support GPU-initiated storage access for LLM KV caching. It connects content-addressable caching with the efficiency of GPU-initiated NVMe access and discusses a possible vendor-specific extension for executing classes against objects. ### Source excerpt For whom the door-bell tolls In a previous post we extolled the benefits of KV caching, a technique to save the KV states from the prefill step of LLM-based inference to reduce time to first token (TTFT) and skip redundant computation. I co-presented this with Tushar Gohad at Cephalocon. Since then I've been thinking a lot about how to improve the state of the art. Really move the needle. We've made strides in a lot of areas in Ceph, especially the work going into Fast EC -- if you don't know what I'm talking about you should check it out, it promises huge benefits to a large category of workloads. That's not what we're here for today, though. Last year there was one paper that stuck in my mind, which is fairly remarkable because I read on the order of 130. That paper was GPU-Initiated On-Demand High-Throughput Storage Access in the BaM System Architecture. I struggled with this. It describes a system where a CPU loads a kernel into the GPU that allows the GPU to serve as an NVMe initiator. The struggle was rooted in the fact that block just felt like the wrong interface for KV caching. If you use block, then you need a lookup table that maps the hash of the sequences representing a cache block to a particular (device, offset, length) tuple. It begs for a content-addressable approach with no centralized lookup or coordination. On the other hand, what was described in the paper was just flat-out more electrically efficient. I couldn't have my cake and eat it too. The idea: RADOS, spoken as NVMe key-value ¶ In 2025 we also saw the first ratified version of the NVMe key-value command set specification. We already have a Ceph implementation of NVMe/TCP that leverages SPDK, and it seemed like we could add support fairly easily for the key-value command set. For those who don't know the internal mechanics of Ceph, the native API operates against RADOS objects. RADOS is richer than most object stores: it supports reads and writes to arbitrary offsets, deletes, key-value sto ## Introducing Mellum2: A 12B Mixture-of-Experts Model by JetBrains DevFeed: [Introducing Mellum2: A 12B Mixture-of-Experts Model by JetBrains]() Original publisher: [Read original article]() Author: Nikita Pavlichenko Published: 2026-06-01T15:45:17Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Mixture of Experts (MoE)](), [Low-Latency Inference](), [Inference](), [Retrieval Augmented Generation (RAG)](), [Routing (disambiguation)](), [Benchmark](), [code-completion](), [hugging face]() Tags: [ai](), [benchmark](), [code](), [code-generation](), [coding](), [inference](), [latency](), [low-latency-inference](), [mixture-of-experts](), [model](), [rag](), [routing](), [software]() ### AI overview JetBrains introduces Mellum2, a 12B-parameter Mixture-of-Experts model trained from scratch on natural language and code. It activates 2.5B parameters per token to provide efficient, low-latency inference for routing, retrieval-augmented generation, summarization, sub-agent tasks, and high-throughput coding features. The model is released under the Apache 2.0 license and is reported to deliver competitive benchmark performance with more than twice the inference speed of similarly sized models. ### Source excerpt - Mellum2 is a 12B-parameter Mixture-of-Experts model trained from scratch on natural language and code. - The model activates only 2.5B parameters per token, making it efficient for high-throughput, low-latency inference. Mellum2 is can be used for routing, RAG, summarization, sub-agents, high-throughput coding features, and private deployments. - It is released under the Apache 2.0 license. ## How DigitalOcean optimized DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B for Serverless Inference DevFeed: [How DigitalOcean optimized DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B for Serverless Inference]() Original publisher: [Read original article]() Author: Bhaskar Dutt Published: 2026-04-28T09:00:00Z Content type: article Language: en Sources: [DigitalOcean]() Topics: [Digital Ocean](), [Inference](), [Inference Performance](), [Low-Latency Inference](), [qwen](), [Serverless](), [AI Development](), [GPU](), [Optimization]() Tags: [deepseek](), [digitalocean](), [engineering](), [gpu](), [inference](), [low-latency-inference](), [optimization](), [performance](), [qwen](), [serverless]() ### AI overview DigitalOcean announces general availability of DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B on DigitalOcean Serverless Inference. The article describes GPU-level optimization and serving-stack tuning, reporting 230 output tokens per second and sub-one-second time to first token for DeepSeek V3.2, plus top output-speed results in Artificial Analysis testing for DeepSeek V3.2 and Qwen 3.5 397B. ### Source excerpt Today at Deploy, we are announcing the general availability of DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B on DigitalOcean Serverless Inference. On DeepSeek V3.2 and Qwen 3.5 397B, we deliver #1 output speed across all providers Artificial Analysis tested. On DeepSeek V3.2 specifically, that translates to 230 output tokens per second and sub-1-second Time-to-First-Token (TTFT) for 10,000 input tokens. This post covers how we got there: the GPU-level work, the serving stack tuning, and the specific technical tradeoffs we made along the way. Why fast inference matters The focus in AI development has fundamentally shifted from the training of models to the efficiency of inference. This shift is driven by the proliferation of agentic workloads, copilots, and real-time systems that form the core of next-generation AI applications. For these applications, speed is no longer just a performance metric; it is the critical differentiator between an engaging product and one that users abandon. Specifically, low-latency inference is essential for a seamless end-user experience. For highly interactive applications like conversational agents and voice interfaces, any delay beyond a sub-1-second TTFT is perceived as sluggish. The importance of fast inference is compounded by the complexity of modern AI workflows. An agentic task, for instance, often involves dozens of sequential model calls, where even minute Time-Per-Output-Token (TPOT) delays can accumulate into several seconds of user-visible latency. Quick inference also helps businesses by providing reliable performance and lower costs. Optimization in this area, such as that provided by DigitalOcean's inference engine, allows enterprises to achieve superior token economics, sustained throughput, and predictable latency, which are essential for scaling their AI-native applications reliably and affordably. Leading the Artificial Analysis benchmarks on speed The benchmarks we're publishing today reflect this. On DeepSeek V3. ## OpenAI partners with Cerebras DevFeed: [OpenAI partners with Cerebras]() Original publisher: [Read original article]() Published: 2026-01-14T14:00:00Z Content type: news Language: en Sources: [OpenAI News]() Topics: [OpenAI](), [Inference Performance](), [Low-Latency Inference](), [real-time](), [AI Models](), [Hardware](), [Artificial Intelligence](), [ChatGPT]() Tags: [ai-models](), [compute](), [global-affairs](), [hardware](), [inference](), [latency](), [low-latency](), [low-latency-inference](), [models](), [openai](), [platform](), [real-time](), [scale]() ### AI overview OpenAI partners with Cerebras to add high-speed AI compute, targeting lower inference latency and faster responses for real-time AI workloads. The capacity will be integrated into OpenAI's inference stack in phases and come online through 2028. ### Source excerpt OpenAI partners with Cerebras to add 750MW of high-speed AI compute, reducing inference latency and making ChatGPT faster for real-time AI workloads. ## Building for an Open Future - our new partnership with Google Cloud DevFeed: [Building for an Open Future - our new partnership with Google Cloud]() Original publisher: [Read original article]() Author: Jeff Boudier; Simon Pagezy Published: 2025-11-13T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [hugging face](), [Cloud](), [Open Source](), [Cloud Run](), [inference-endpoints](), [Low-Latency Inference](), [Cache](), [xet](), [data](), [Deployment]() Tags: [announcement](), [artificial-intelligence](), [cache](), [cloud](), [data](), [datasets](), [deployment](), [google](), [google-cloud](), [inference-endpoints](), [networking](), [open](), [open-source](), [partnership](), [partnerships](), [storage](), [vertex](), [vertex-ai](), [xet]() ### AI overview Hugging Face and Google Cloud announce a strategic partnership focused on making open AI models easier to use, customize, deploy, and govern. The article describes integrations across Vertex AI, GKE AI/ML, Cloud Run GPUs, and other Google Cloud infrastructure, plus a planned CDN Gateway using Hugging Face Xet and Google Cloud storage and networking to accelerate model and dataset downloads and improve supply-chain robustness. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## State of open video generation models in Diffusers DevFeed: [State of open video generation models in Diffusers]() Original publisher: [Read original article]() Author: Sayak Paul; Aryan V S; Dhruv Nair Published: 2025-01-27T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [diffusers](), [Artificial Intelligence](), [Open Source](), [AI Development](), [Fine-tuning](), [Low-Latency Inference]() Tags: [artificial-intelligence](), [diffusers](), [diffusion](), [fine-tuning](), [guide](), [inference](), [latency](), [open-source](), [quantization](), [video](), [video-gen]() ### AI overview An overview of open video generation models, their capabilities and limitations, and the Diffusers team's plans to support their adoption at scale. It covers inference optimization, quantization, latency, and fine-tuning. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Unlocking Real-time Predictions with Shopify's Machine Learning Platform DevFeed: [Unlocking Real-time Predictions with Shopify's Machine Learning Platform]() Original publisher: [Read original article]() Author: Isaac Vidas Published: 2023-03-10T16:16:27Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [Shopify](), [Machine Learning & Artificial Intelligence](), [model-serving](), [Low-Latency Inference](), [real-time](), [model-deployment](), [Deployment]() Tags: [deployment](), [inference](), [low-latency](), [machine-learning](), [machine-learning-pipelines](), [real-time](), [shopify]() ### AI overview Shopify describes how it expanded Merlin, its machine learning platform built on Ray, with online inference capabilities for deploying and serving machine learning models that generate real-time predictions at scale. The article explains the platform's serving layer, service deployment, low-latency requirements, and distinction between batch and online inference. ### Source excerpt Learn how Shopify Data built new online inference capabilities into its Machine Learning Platform to deploy and serve models for real-time prediction at scale.