# Low-Latency Inference Published articles for Low-Latency Inference. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## AI Infra Summit: NVIDIA Vera Rubin and DSX Platform Advancements Showcase Energy Efficiencies of Optimizing Tokens Per Watt for AI Factories DevFeed: [AI Infra Summit: NVIDIA Vera Rubin and DSX Platform Advancements Showcase Energy Efficiencies of Optimizing Tokens Per Watt for AI Factories]() Original publisher: [Read original article]() Author: NVIDIA Writers Published: 2026-09-15T16:55:40Z Content type: news Language: en Sources: [NVIDIA Blog]() Topics: [AI Infrastructure](), [Nvidia](), [DSX](), [Vera Rubin](), [Low-Latency Inference](), [NVLink](), [Conversational AI]() Tags: [ai](), [ai-infrastructure](), [dsx](), [inference](), [infra](), [latency](), [low-latency-inference](), [networking](), [nvidia](), [nvidia-blackwell](), [nvidia-dsx](), [nvidia-vera](), [nvidia-vera-rubin](), [nvlink]() ### AI overview NVIDIA's AI Infra Summit coverage describes collaborations and platform updates focused on improving AI factory efficiency. The article highlights Vera Rubin systems, DSX MaxLPS, Dynamo inference software, NVLink and networking technologies, including claims of up to 1.4x more tokens per megawatt through factory-wide power optimization. ### Source excerpt Ian Buck, vice president of hyperscale and high-performance computing at NVIDIA, Tuesday spoke on AI factory efficiency at the AI Infra Summit, the Santa Clara Convention Center event that has morphed into a Coachella of infrastructure tech. Before a packed audience -- with more than 8,000 attendees this year, up from 3,500 last year -- [...] ## When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving DevFeed: [When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving]() Original publisher: [Read original article]() Author: Tanya Lenz Published: 2026-09-09T20:31:04Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Optimization](), [model-deployment](), [multimodal](), [Dynamo](), [Inference Performance](), [Embeddings](), [Mixture of Experts (MoE)](), [GPU](), [Open Source](), [Artificial Intelligence]() Tags: [agentic-ai-generative-ai](), [ai-agent](), [ai-inference](), [computer-vision-video-analytics](), [developer-tools-techniques](), [dynamo](), [embeddings](), [gpu](), [inference](), [low-latency-inference](), [mixture-of-experts-moe](), [model-serving](), [multimodal](), [nvfp4](), [open-source](), [optimization]() ### AI overview This article explains when encode-prefill-decode disaggregation improves multimodal model serving. Using NVIDIA Dynamo, it separates vision encoding from LLM prefill and decode so the stages can batch, schedule, and scale independently. The approach is most effective for image-heavy prompts, short-to-medium outputs, and quantized mixture-of-experts models, with reported gains of up to 5x in time to first token and 7x in end-to-end response time. ### Source excerpt Encode-prefill-decode (EPD) disaggregation is an inference optimization technique for multimodal models that separates the vision encoder stage from the prefill... ## The full stack behind abundant intelligence DevFeed: [The full stack behind abundant intelligence]() Original publisher: [Read original article]() Published: 2026-08-25T07:05:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [OpenAI](), [Artificial Intelligence](), [Hardware](), [Low-Latency Inference](), [Inference Performance](), [model-deployment](), [gpt-oss](), [Benchmark](), [cloud-infrastructure](), [systems](), [Amazon Web Services](), [Microsoft]() Tags: [ai](), [aws](), [benchmark](), [company](), [compute](), [cost](), [efficiency](), [energy-efficiency](), [hardware](), [inference](), [low-latency-inference](), [microsoft](), [models](), [nvidia](), [openai]() ### AI overview OpenAI describes an integrated compute strategy spanning data centers, chips, models, software, products, and devices. It reports that its custom Jalapeño inference chip achieved higher peak throughput per kilowatt and lower token latency than commercial systems on the InferenceX benchmark using GPT-OSS 120B, while also performing strongly on DeepSeek R1 and Kimi K2. ### Source excerpt OpenAI CFO Sarah Friar explains how advances across chips, compute, models, and products compound to deliver more useful intelligence at greater scale and lower cost. ## With Groq 3 LPX in Full Production, NVIDIA Extends Vera Rubin Inference for Agents DevFeed: [With Groq 3 LPX in Full Production, NVIDIA Extends Vera Rubin Inference for Agents]() Original publisher: [Read original article]() Author: NVIDIA Writers Published: 2026-08-24T15:00:41Z Content type: news Language: en Sources: [NVIDIA Blog]() Topics: [Inference Performance](), [long-context](), [Low-Latency Inference](), [networking](), [gemma4]() Tags: [agentic](), [agents](), [ai-factory](), [ai-inference](), [ai-infrastructure](), [benchmark](), [bluefi](), [cloud](), [cpx](), [groq](), [groq-3-lpx](), [inference](), [infrastructure](), [latency](), [long-context](), [low-latency-inference](), [lpx](), [networking](), [nvidia](), [nvidia-spectrum-x-ethernet](), [nvidia-vera](), [nvidia-vera-rubin](), [nvl72](), [nvlink](), [vera-cpu](), [vera-rubin]() ### AI overview NVIDIA announces Groq 3 LPX production availability alongside Vera Rubin NVL72, positioning the integrated compute, networking, and inference stack for long-context, agentic AI workloads. The article highlights a benchmark result of 3,400 output tokens per second on Gemma 4 31B and describes partner adoption of Vera Rubin, Spectrum-X Multiplane, and Groq 3 LPX. ### Source excerpt The next era of AI inference won't be defined by a single breakthrough chip, network or system. It'll be defined by how every layer of the AI factory works together. That's why NVIDIA is extending Vera Rubin NVL72 with fast token generation for agentic systems. Announced today, the NVIDIA Vera Rubin rack-scale system NVIDIA Groq [...] ## How NVIDIA Groq 3 LPX Unlocks Ultrafast Interactivity at Long Context on NVIDIA Vera Rubin DevFeed: [How NVIDIA Groq 3 LPX Unlocks Ultrafast Interactivity at Long Context on NVIDIA Vera Rubin]() Original publisher: [Read original article]() Author: Tanya Lenz Published: 2026-08-24T15:00:00Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [d-matrix](), [Vera Rubin](), [Inference Performance](), [long-context](), [Benchmark](), [Artificial Intelligence](), [gemma4](), [systems](), [Cache]() Tags: [agent](), [agentic](), [agentic-ai-generative-ai](), [ai](), [ai-factory](), [ai-inference](), [benchmark](), [cache](), [collective](), [data-center-cloud](), [developer-tools-techniques](), [groq](), [groq-3-lpx](), [inference-performance](), [long-context](), [low-latency-inference](), [lpx](), [nvidia](), [nvidia-vera](), [performance](), [rubin-gpu](), [speed](), [systems](), [tokens](), [training-ai-models](), [vera-rubin](), [vera-rubin-nvl72]() ### AI overview NVIDIA Groq 3 LPX, paired with Vera Rubin NVL72, delivers high-interactivity AI inference for long-context workloads. A reported benchmark measured 3,431 output tokens per second on Gemma 4 31B with a 100K context. ### Source excerpt NVIDIA Groq 3 LPX is the interactive AI inference accelerator for the NVIDIA Vera Rubin platform. At the core of the platform is NVIDIA Vera Rubin NVL72, the... ## Introducing Mellum2: A 12B Mixture-of-Experts Model by JetBrains DevFeed: [Introducing Mellum2: A 12B Mixture-of-Experts Model by JetBrains]() Original publisher: [Read original article]() Author: Nikita Pavlichenko Published: 2026-06-01T15:45:17Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Mixture of Experts (MoE)](), [Low-Latency Inference](), [Inference](), [Retrieval Augmented Generation (RAG)](), [Routing (disambiguation)](), [Benchmark](), [code-completion](), [hugging face]() Tags: [ai](), [benchmark](), [code](), [code-generation](), [coding](), [inference](), [latency](), [low-latency-inference](), [mixture-of-experts](), [model](), [rag](), [routing](), [software]() ### AI overview JetBrains introduces Mellum2, a 12B-parameter Mixture-of-Experts model trained from scratch on natural language and code. It activates 2.5B parameters per token to provide efficient, low-latency inference for routing, retrieval-augmented generation, summarization, sub-agent tasks, and high-throughput coding features. The model is released under the Apache 2.0 license and is reported to deliver competitive benchmark performance with more than twice the inference speed of similarly sized models. ### Source excerpt - Mellum2 is a 12B-parameter Mixture-of-Experts model trained from scratch on natural language and code. - The model activates only 2.5B parameters per token, making it efficient for high-throughput, low-latency inference. Mellum2 is can be used for routing, RAG, summarization, sub-agents, high-throughput coding features, and private deployments. - It is released under the Apache 2.0 license. ## How DigitalOcean optimized DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B for Serverless Inference DevFeed: [How DigitalOcean optimized DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B for Serverless Inference]() Original publisher: [Read original article]() Author: Bhaskar Dutt Published: 2026-04-28T09:00:00Z Content type: article Language: en Sources: [DigitalOcean]() Topics: [Digital Ocean](), [Inference](), [Inference Performance](), [Low-Latency Inference](), [qwen](), [Serverless](), [AI Development](), [GPU](), [Optimization]() Tags: [deepseek](), [digitalocean](), [engineering](), [gpu](), [inference](), [low-latency-inference](), [optimization](), [performance](), [qwen](), [serverless]() ### AI overview DigitalOcean announces general availability of DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B on DigitalOcean Serverless Inference. The article describes GPU-level optimization and serving-stack tuning, reporting 230 output tokens per second and sub-one-second time to first token for DeepSeek V3.2, plus top output-speed results in Artificial Analysis testing for DeepSeek V3.2 and Qwen 3.5 397B. ### Source excerpt Today at Deploy, we are announcing the general availability of DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B on DigitalOcean Serverless Inference. On DeepSeek V3.2 and Qwen 3.5 397B, we deliver #1 output speed across all providers Artificial Analysis tested. On DeepSeek V3.2 specifically, that translates to 230 output tokens per second and sub-1-second Time-to-First-Token (TTFT) for 10,000 input tokens. This post covers how we got there: the GPU-level work, the serving stack tuning, and the specific technical tradeoffs we made along the way. Why fast inference matters The focus in AI development has fundamentally shifted from the training of models to the efficiency of inference. This shift is driven by the proliferation of agentic workloads, copilots, and real-time systems that form the core of next-generation AI applications. For these applications, speed is no longer just a performance metric; it is the critical differentiator between an engaging product and one that users abandon. Specifically, low-latency inference is essential for a seamless end-user experience. For highly interactive applications like conversational agents and voice interfaces, any delay beyond a sub-1-second TTFT is perceived as sluggish. The importance of fast inference is compounded by the complexity of modern AI workflows. An agentic task, for instance, often involves dozens of sequential model calls, where even minute Time-Per-Output-Token (TPOT) delays can accumulate into several seconds of user-visible latency. Quick inference also helps businesses by providing reliable performance and lower costs. Optimization in this area, such as that provided by DigitalOcean's inference engine, allows enterprises to achieve superior token economics, sustained throughput, and predictable latency, which are essential for scaling their AI-native applications reliably and affordably. Leading the Artificial Analysis benchmarks on speed The benchmarks we're publishing today reflect this. On DeepSeek V3. ## OpenAI partners with Cerebras DevFeed: [OpenAI partners with Cerebras]() Original publisher: [Read original article]() Published: 2026-01-14T14:00:00Z Content type: news Language: en Sources: [OpenAI News]() Topics: [OpenAI](), [Inference Performance](), [Low-Latency Inference](), [real-time](), [AI Models](), [Hardware](), [Artificial Intelligence](), [ChatGPT]() Tags: [ai-models](), [compute](), [global-affairs](), [hardware](), [inference](), [latency](), [low-latency](), [low-latency-inference](), [models](), [openai](), [platform](), [real-time](), [scale]() ### AI overview OpenAI partners with Cerebras to add high-speed AI compute, targeting lower inference latency and faster responses for real-time AI workloads. The capacity will be integrated into OpenAI's inference stack in phases and come online through 2028. ### Source excerpt OpenAI partners with Cerebras to add 750MW of high-speed AI compute, reducing inference latency and making ChatGPT faster for real-time AI workloads.