# TensorRT-LLM TensorRT-LLM is a toolkit with Python and C++ APIs for building optimized TensorRT engines and efficiently performing large language model inference on NVIDIA GPUs. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## NVIDIA Vera Rubin NVL72 Delivers Leading Performance in MLPerf Inference v6.1 Debut DevFeed: [NVIDIA Vera Rubin NVL72 Delivers Leading Performance in MLPerf Inference v6.1 Debut]() Original publisher: [Read original article]() Author: Zhihan Jiang Published: 2026-09-16T15:00:48Z Content type: article Language: en Sources: [NVIDIA Blog]() Topics: [NVIDIA Vera Rubin](), [Vera Rubin NVL72](), [benchmarking](), [Inference Performance](), [Dynamo](), [vllm](), [Nvidia](), [TensorRT-LLM]() Tags: [ai-infrastructure](), [benchmarks](), [dynamo](), [efficiency](), [hardware](), [inference](), [mlperf](), [networking](), [nvidia](), [nvidia-vera-rubin](), [nvl72](), [optimization](), [performance](), [software](), [tensorrt](), [tensorrt-llm](), [vera-rubin-nvl72](), [vllm]() ### AI overview NVIDIA reports MLPerf Inference v6.1 preview results for Vera Rubin NVL72 and GB300 NVL72 systems. Vera Rubin NVL72 delivered up to 3.7x higher throughput than GB300 NVL72 on Qwen3-VL and up to 2.5x higher throughput on DeepSeek-R1, while a four-rack GB300 NVL72 submission achieved 99% scaling efficiency. The results used vLLM, NVIDIA Dynamo, and TensorRT-LLM. ### Source excerpt System performance, efficient infrastructure scaling and continuous software optimization are key levers that determine AI inference economics. Higher system performance means more tokens generated, resulting in higher revenue. Efficient scaling means throughput grows proportionally as hardware gets added, requiring fewer resources to serve users at scale. Continuous optimization means generating more value from infrastructure investments. [...] ## Experiment with Qwen3.8-Flash-Next on NVIDIA GB300 NVL72 for Agentic Coding DevFeed: [Experiment with Qwen3.8-Flash-Next on NVIDIA GB300 NVL72 for Agentic Coding]() Original publisher: [Read original article]() Author: Michelle Horton Published: 2026-08-26T17:07:12Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [qwen](), [long-context](), [Mixture of Experts (MoE)](), [agentic-coding](), [Inference](), [GPU](), [multimodal](), [NeMo](), [sglang](), [TensorRT-LLM](), [vllm](), [rust-ai]() Tags: [agentic-ai-generative-ai](), [agentic-coding](), [benchmarks](), [context-window](), [developer-tools-techniques](), [gb300-nvl72](), [inference](), [mixture-of-experts-moe](), [multimodal](), [nemo](), [nvlink](), [qwen](), [sglang](), [tensorrt-llm](), [top-stories](), [vllm]() ### AI overview This NVIDIA developer article introduces Qwen3.8-Flash-Next, a multimodal mixture-of-experts model released by Alibaba for experimentation and evaluation. It explains the model's long-context hybrid architecture, including Gated DeltaNet and Qwen Sparse Attention, and discusses reported efficiency improvements for million-token workloads. The article also covers inference support through SGLang, vLLM, TensorRT-LLM, and NVIDIA NeMo, plus performance on the NVIDIA GB300 NVL72 platform. ### Source excerpt Alibaba released the model weights for Qwen3.8-Flash-Next as a preview of the upcoming Qwen4 architecture for developers to experiment with and evaluate. It's... ## Advanced Prompt Caching at Scale DevFeed: [Advanced Prompt Caching at Scale]() Original publisher: [Read original article]() Author: Andrew Dugan Published: 2026-04-07T19:11:40Z Content type: tutorial Language: en Sources: [DigitalOcean]() Topics: [Caching](), [Inference Performance](), [Large Language Model](), [Inference](), [Latency](), [Load Balancing](), [round robin](), [sglang](), [TensorRT-LLM](), [vllm](), [model architecture]() Tags: [ai-ml](), [architecture](), [caching](), [decoding](), [efficiency](), [engineering](), [inference](), [latency](), [llm](), [load-balancing](), [prompt](), [round-robin](), [sglang](), [tensorrt-llm](), [token](), [tokens](), [vllm]() ### AI overview This tutorial explains how prompt caching works across multiple LLM replicas. It describes how round-robin load balancing reduces cache-hit rates and presents session affinity, tiered routing, and prefix-aware load balancing as architectural strategies for preserving KV-cache reuse while reducing latency and inference costs. ### Source excerpt Introduction Prompt caching is the process of reusing already computed KV states across inference requests in order to save money and reduce latency. Within a single replica, modern inference engines like vLLM, SGLang, and TensorRT-LLM handle it automatically. Incoming prompts are matched against cached prefixes and recomputed only where necessary, without requiring user configurations The problem nobody talks about is what happens when you scale to many replicas. Under round-robin load balancing, a request with an identical prefix has only a 1/N chance of hitting the replica where that prefix is already cached. The cache hit rate that made prompt caching so attractive at one replica degrades almost linearly as your fleet grows, unless you architect around it deliberately. Done right, prompt caching at scale offers 50-90% discounts on cached input tokens and can reduce time-to-first-token (TTFT) latency by up to 80%. This article covers the architectural strategies that make that possible. The Single-Replica Ceiling Refer to our previous prompt caching article for a detailed explanation of how KV caching works under the hood. Every transformer-based LLM uses KV caching to store key and value vectors from the attention layers in GPU VRAM during decoding. This intra-request caching is baked into the model architecture to increase throughput and maximize efficiency. Within a single replica, modern open-source engines like vLLM, SGLang (via RadixAttention), and TensorRT-LLM support automatic prefix caching out of the box, matching incoming prompts against previously cached prefixes to maximize KV reuse without any user configuration. Reusing KV states across requests from many users and replicas is where inference frameworks differ significantly. In the simplest architecture, the cache lives on individual replicas in VRAM. It is not shared across model instances at all. When a user makes an inference request, the prompt from their request is cached on a single replica. ## NVIDIA Dynamo 1.0 Is Available to DigitalOcean Customers for Inference Performance and Cost Efficiency DevFeed: [NVIDIA Dynamo 1.0 Is Available to DigitalOcean Customers for Inference Performance and Cost Efficiency]() Original publisher: [Read original article]() Author: Waverly Swinton Published: 2026-03-19T22:13:37Z Content type: release Language: en Sources: [DigitalOcean]() Topics: [Dynamo](), [Digital Ocean](), [Inference Performance](), [Artificial Intelligence](), [Generative AI](), [Inference](), [Cloud](), [GB200](), [sglang](), [TensorRT-LLM](), [vllm](), [Kubernetes]() Tags: [ai-ml](), [cloud](), [cost](), [deployment](), [digitalocean](), [dynamo](), [gb200](), [inference](), [inference-performance](), [kubernetes](), [latency](), [llm](), [load-balancing](), [nvidia](), [product-updates](), [routing](), [sglang](), [tensorrt-llm](), [vllm]() ### AI overview NVIDIA Dynamo 1.0 is now available to DigitalOcean customers as an inference service framework for large-scale generative AI workloads. The article describes claimed performance improvements on NVIDIA GB200 NVL systems, cost-efficiency benefits, deployment options, and features including GPU-aware routing and disaggregated serving. ### Source excerpt NVIDIA Dynamo 1.0, which was released on Monday at NVIDIA GTC, is now available to DigitalOcean customers to help drive performance enhancements and cost efficiency. NVIDIA Dynamo 1.0 offers a 7x inference performance increase on NVIDIA GB200 NVL systems, and by pairing it with DigitalOcean's Agentic Inference Cloud, customers can achieve higher performance at lower costs while benefiting from seamless deployment. Working together, DigitalOcean's optimizations with NVIDIA have already achieved a 67% cost savings for customers like Workato, and this new generation of Dynamo can unlock even greater gains for businesses who run production-grade agentic workflows. DigitalOcean customers can get access to NVIDIA Dynamo 1.0 as a container image that can be run on a Droplet or can deploy directly on DigitalOcean Kubernetes with an inference runtime (vLLM, SGlang, TensorRT). What is NVIDIA Dynamo 1.0? NVIDIA Dynamo is a cutting-edge, high-performance inference service framework specifically designed to accelerate and optimize large-scale generative AI and inference models. Dynamo is an orchestration layer that sits above engines like vLLM, SGLang, and NVIDIA TensorRT-LLM. Think of it as the distributed traffic controller for your GPU fleet, seamlessly orchestrating GPU and memory resources across a cluster and reducing bottleneck by intelligently routing requests Key technical breakthroughs offered by Dynamo 1.0 include: 7x Performance Boost: When paired with NVIDIA Blackwell Ultra GPUs, Dynamo can increase inference performance by up to 7x, significantly lowering your cost per token. KV-Aware Routing: Instead of simple round-robin load balancing, Dynamo routes requests to the specific GPUs that already have the relevant "memory" from previous turns of a conversation. Disaggregated Serving: Dynamo splits the "prefill" (reading the prompt) and "decode" (generating the answer) phases across different GPUs to maximize utilization and reduce latency. Memory Offloading: The KV B ## Accelerate a World of LLMs on Hugging Face with NVIDIA NIM DevFeed: [Accelerate a World of LLMs on Hugging Face with NVIDIA NIM]() Original publisher: [Read original article]() Author: Neal Vaidya Published: 2025-07-21T18:01:30Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [NVIDIA NIM](), [hugging face](), [Deployment](), [Inference](), [Docker Container](), [TensorRT-LLM](), [sglang](), [vllm]() Tags: [cuda](), [deployment](), [docker](), [hugging-face](), [inference](), [llms](), [nim](), [nvidia-nim](), [sglang](), [tensorrt-llm](), [vllm]() ### AI overview This tutorial explains how NVIDIA NIM can deploy a broad range of LLMs from Hugging Face using a single Docker container. It covers supported checkpoint formats, inference frameworks, environment prerequisites, authentication, caching, permissions, and local model deployment. ### Source excerpt NVIDIA AI customers and ecosystem partners leverage NVIDIA NIM inference microservices to streamline deployment of the latest AI models on NVIDIA accelerated infrastructure, including LLMs, multi-modal and domain-specific models from NVIDIA, Meta, Mistral AI, Google and hundreds more innovative model builders. ## Introducing multi-backends (TRT-LLM, vLLM) support for Text Generation Inference DevFeed: [Introducing multi-backends (TRT-LLM, vLLM) support for Text Generation Inference]() Original publisher: [Read original article]() Author: Morgan Funtowicz; Hugo Larcher Published: 2025-01-16T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [tgi](), [model-deployment](), [Rust](), [vllm](), [TensorRT-LLM](), [Concurrency](), [Memory Safety](), [servers](), [llama.cpp](), [sglang](), [Python](), [Code]() Tags: [backend](), [backends](), [community](), [concurrency](), [http](), [inference](), [jetstream](), [llama-cpp](), [memory-safety](), [neuron](), [open-source](), [performance](), [production](), [python](), [rust](), [sglang](), [tensorrt-llm](), [tgi](), [vllm]() ### AI overview Hugging Face introduces TGI Backends, a unified frontend architecture for integrating inference solutions such as vLLM, SGLang, llama.cpp, and TensorRT-LLM. The approach lets users switch backends based on model, hardware, and performance requirements while supporting production deployment across diverse accelerators. The article also describes TGI's Rust and Python components, including Rust-based HTTP and scheduling layers designed for memory safety and concurrency. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Codestral Mamba DevFeed: [Codestral Mamba]() Original publisher: [Read original article]() Published: 2024-07-16T08:00:00Z Content type: news Language: en Sources: [Mistral AI Blog]() Topics: [Mamba](), [code productivity](), [Inference](), [Local AI](), [SDK](), [TensorRT-LLM](), [llama.cpp]() Tags: [efficiency](), [inference](), [llama-cpp](), [mamba](), [mixtral](), [open](), [sdk](), [tensorrt-llm]() ### AI overview Mistral introduces Codestral Mamba, a freely usable and distributable code-focused model based on the Mamba architecture. It offers linear-time inference, supports very long sequences, and is intended for efficient local code-assistant use. The article describes deployment through the mistral-inference SDK and TensorRT-LLM, with anticipated llama.cpp support, and notes its Apache 2.0 license. ### Source excerpt The most powerful AI platform for enterprises. Customize, fine-tune, and deploy AI assistants, autonomous agents, and multimodal AI with open models.