# TensorRT Published articles for TensorRT. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor DevFeed: [TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor]() Original publisher: [Read original article]() Author: Elizabeth Goodman Published: 2026-09-16T20:37:07Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Jetson AGX Thor Developer Kit](), [Inference Performance](), [benchmarking](), [TensorRT](), [Large Language Model](), [quantization](), [Artificial Intelligence]() Tags: [agentic-ai-generative-ai](), [ai-agent](), [ai-inference](), [evaluation](), [jetson](), [jetson-agx-thor-developer-kit](), [llm](), [llm-benchmarking](), [mlperf](), [nvidia](), [performance](), [quantization](), [tensorrt](), [tensorrt-llm](), [thor]() ### AI overview This article reports that NVIDIA TensorRT Edge-LLM ran Qwen3.6-27B on a single NVIDIA Jetson AGX Thor Developer Kit for the MLPerf Inference v6.1 Edge Agentic benchmark. Using NVFP4 quantization, tree-based multi-token prediction, and KV cache reuse, it achieved 52.33 tokens per second and completed 1,007 turns in 24 minutes and 36 seconds, 6.4 times faster than the llama.cpp reference submission. ### Source excerpt AI agents are moving from cloud data centers to vehicles, robots, and other edge devices. Unlike a chatbot that answers a single prompt, an agent works through... ## NVIDIA Vera Rubin NVL72 Delivers Leading Performance in MLPerf Inference v6.1 Debut DevFeed: [NVIDIA Vera Rubin NVL72 Delivers Leading Performance in MLPerf Inference v6.1 Debut]() Original publisher: [Read original article]() Author: Zhihan Jiang Published: 2026-09-16T15:00:48Z Content type: article Language: en Sources: [NVIDIA Blog]() Topics: [NVIDIA Vera Rubin](), [Vera Rubin NVL72](), [benchmarking](), [Inference Performance](), [Dynamo](), [vllm](), [Nvidia](), [TensorRT-LLM]() Tags: [ai-infrastructure](), [benchmarks](), [dynamo](), [efficiency](), [hardware](), [inference](), [mlperf](), [networking](), [nvidia](), [nvidia-vera-rubin](), [nvl72](), [optimization](), [performance](), [software](), [tensorrt](), [tensorrt-llm](), [vera-rubin-nvl72](), [vllm]() ### AI overview NVIDIA reports MLPerf Inference v6.1 preview results for Vera Rubin NVL72 and GB300 NVL72 systems. Vera Rubin NVL72 delivered up to 3.7x higher throughput than GB300 NVL72 on Qwen3-VL and up to 2.5x higher throughput on DeepSeek-R1, while a four-rack GB300 NVL72 submission achieved 99% scaling efficiency. The results used vLLM, NVIDIA Dynamo, and TensorRT-LLM. ### Source excerpt System performance, efficient infrastructure scaling and continuous software optimization are key levers that determine AI inference economics. Higher system performance means more tokens generated, resulting in higher revenue. Efficient scaling means throughput grows proportionally as hardware gets added, requiring fewer resources to serve users at scale. Continuous optimization means generating more value from infrastructure investments. [...] ## Skild AI Taps NVIDIA Physical AI to Teach Robots New Tasks From a Single Video DevFeed: [Skild AI Taps NVIDIA Physical AI to Teach Robots New Tasks From a Single Video]() Original publisher: [Read original article]() Author: Sasa Docca Published: 2026-09-10T16:30:35Z Content type: news Language: en Sources: [NVIDIA Blog]() Topics: [foundation-models](), [Cosmos](), [Simulation and Design](), [synthetic-data]() Tags: [ai](), [artificial-intelligence](), [cosmos](), [customer-stories](), [industrial-and-manufacturing](), [isaac](), [model](), [nvidia](), [nvidia-blackwell](), [omniverse](), [physical-ai](), [robotics](), [robots](), [simulation-and-design](), [synthetic-data-generation](), [tensorrt](), [video]() ### AI overview Skild AI's S1 robot foundation model learns new long-horizon physical tasks from a single video demonstration through in-context learning, without task-specific retraining. The article describes its development on NVIDIA AI infrastructure and use of NVIDIA Isaac Lab and Cosmos technologies. ### Source excerpt Manufacturing floors, warehouses and production lines rarely stay fixed -- tasks change, layouts shift and new products arrive, and most robots can't keep up without significant reprogramming. Skild AI's new S1 robot foundation model helps address this, designed to learn previously unseen, long-horizon tasks from a single video demonstration. The model, launched last week, uses [...] ## Deploy an Open Model from Checkpoint to Inference in Two Commands with NVIDIA TensorRT Model Connect DevFeed: [Deploy an Open Model from Checkpoint to Inference in Two Commands with NVIDIA TensorRT Model Connect]() Original publisher: [Read original article]() Author: Tanya Lenz Published: 2026-08-28T17:06:28Z Content type: tutorial Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [AI, ML & Data Engineering](), [C++](), [Orchestration]() Tags: [ai](), [ai-inference](), [ai-models](), [api](), [applications](), [c-plus-plus](), [data-center-cloud](), [deployment](), [developer-tools-techniques](), [edge-computing](), [hugging-face](), [inference](), [integration](), [kernel](), [model](), [nvidia](), [open](), [open-source](), [orchestration](), [python](), [pytorch](), [tensorrt]() ### AI overview The article explains NVIDIA TensorRT Model Connect, a collection of modifiable reference implementations for deploying supported open models from a Hugging Face ID or local checkpoint to native C++ inference. It describes a two-phase deployment bundle workflow, semantic and module-level C++ APIs, and custom GPU-kernel integration. ### Source excerpt Open AI models are evolving faster than ever, but bringing them into native applications can still require model-specific conversion, preprocessing,... ## Make Long-Running NVIDIA TensorRT Engine Builds Observable and Cancelable in Python or C++ DevFeed: [Make Long-Running NVIDIA TensorRT Engine Builds Observable and Cancelable in Python or C++]() Original publisher: [Read original article]() Author: Michelle Horton Published: 2026-07-22T16:35:04Z Content type: tutorial Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [TensorRT](), [C++](), [Python](), [GPU](), [Cache]() Tags: [agentic-ai-generative-ai](), [build-ai-agents](), [c-plus-plus](), [cache](), [code](), [developer-tools-techniques](), [developers](), [featured](), [gpu](), [python](), [tensorrt](), [tutorial]() ### AI overview This tutorial explains how to make long-running NVIDIA TensorRT engine builds observable and cancelable in Python and C++. It presents a thread-safe progress monitor with nested build phases, shows how to connect it to the TensorRT builder, and demonstrates cancellation through Ctrl-C or a programmatic stop signal. Progress can be routed to a terminal, IDE, service, or agent runtime. ### Source excerpt A TensorRT engine build can take seconds to many minutes. Large strongly typed models, deep tactic search, and a cold timing cache on a brand-new GPU SKU can... ## Hardware-Rooted AI Security That Won't Slow You Down DevFeed: [Hardware-Rooted AI Security That Won't Slow You Down]() Original publisher: [Read original article]() Author: Elizabeth Goodman Published: 2026-07-02T21:25:42Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Cryptography](), [Inference Performance]() Tags: [agentic-ai-generative-ai](), [ai](), [ai-agent](), [ai-inference](), [ai-security](), [benchmarks](), [blackwell](), [cloud-services](), [code-software-generation](), [data-center-cloud](), [dynamo-triton](), [encryption](), [featured](), [gpu](), [hardware](), [inference-performance](), [latency](), [nvlink](), [security](), [security-for-ai](), [software-defined-data-center](), [tensorrt](), [tensorrt-llm](), [trustworthy-ai-cybersecurity]() ### AI overview An overview of NVIDIA Confidential Computing for protecting AI inference workloads, including hardware-rooted attestation and encryption. It reports benchmark results of up to 98% of the inference performance of configurations without confidential-computing security. ### Source excerpt AI has transformed how organizations operate, driving unprecedented levels of productivity and innovation. However, AI adoption can be impeded by concerns... ## A Practical Loop for Optimizing GPU Inference Throughput DevFeed: [A Practical Loop for Optimizing GPU Inference Throughput]() Original publisher: [Read original article]() Author: Andrei Potapkin Published: 2026-05-01T17:57:59Z Content type: article Language: en Sources: [Bumble Tech]() Topics: [Inference](), [Optimization](), [GPU](), [Benchmark](), [TensorRT](), [onnx](), [Cloud](), [migration]() Tags: [benchmark](), [cloud](), [cloud-migration](), [gpu](), [inference](), [nvidia-nsight-systems](), [onnx](), [optimization](), [python](), [tensorrt]() ### AI overview This practical article describes a production-focused loop for optimizing GPU inference: benchmark under representative load, identify the current throughput bottleneck, fix it, and verify the operational outcome. In the described ML moderation service, TensorRT, BF16, and an attention-subgraph rewrite increased throughput from roughly 150 to roughly 1100 requests per second in the same saturated benchmark. ### Source excerpt A practical loop for optimizing GPU inference without guessing It wasn't an incident. That made it interesting. A few days after a cloud migration, one of our highest-traffic ML services looked healthy. Traffic was flowing. Latency was fine. The rollout had gone the way you want these things to go: boringly. And yet a Slack thread started anyway. Not because anything was broken. Because something had become easier to see. On-prem, GPU efficiency tends to get folded into bigger conversations: capacity planning, hardware timelines, headroom for the next model. In the cloud, the same question sharpens. Throughput, utilization, and cost sit much closer together. A service can be perfectly healthy and still look expensive. That was our situation. We had a frontend text moderation model based on mT5, served through NVIDIA Triton as a two-component ensemble: a Python tokenizer and a static ONNX model. The service was stable, but throughput per GPU was much lower than our back-of-the-envelope estimates suggested Moments like that attract the usual optimization advice: Batch more aggressively. Tune Triton. Rewrite the Python stages. Enable TensorRT. Drop precision. None of that advice is silly. Some of it is very good. The problem is that it usually arrives before the diagnosis. GPU systems are awkward that way. By the time you go hunting for "the optimization", you are often already asking the wrong question. There usually isn't one magical fix. There is a bottleneck somewhere in the stack, and then another one behind it, and usually another one behind that. The loop that kept us honest was simple: TL;DR Benchmark the system under production-shaped load. Find the bottleneck limiting throughput right now. Fix that bottleneck. Check whether the operational outcome changed. In our case, that loop took us from roughly 150 requests per second to roughly 1100 in the same saturated benchmark: a bit over 7x baseline. TensorRT delivered the first big jump. BF16 moved the ceiling aga ## Load Balancing and Scaling LLM Serving DevFeed: [Load Balancing and Scaling LLM Serving]() Original publisher: [Read original article]() Author: Mohammad Ashar Khan Published: 2026-04-15T19:03:31Z Content type: article Language: en Sources: [DigitalOcean]() Topics: [Load Balancing](), [Caching](), [Large Language Model](), [Routing (disambiguation)](), [Latency](), [sglang](), [vllm](), [Concurrency](), [TensorRT]() Tags: [caching](), [concurrency](), [efficiency](), [engineering](), [http](), [inference](), [latency](), [llm](), [load-balancing](), [performance](), [round-robin](), [sglang](), [tensorrt](), [vllm]() ### AI overview This article explains why scaling LLM serving requires load-balancing strategies that account for prompt-prefix caching. It describes how naive round-robin routing reduces cache-hit probability as the number of replicas grows, and introduces cache-aware routing approaches intended to preserve cache efficiency, control costs, and improve performance. It also discusses inference engines including vLLM, SGLang, and TensorRT, along with prefill and decode processing. ### Source excerpt Load balancing for LLMs is fundamentally different from load balancing for traditional services like web servers, APIs, or databases. Prompt caching is the reason. Prompt caching typically cuts input token costs by 50-90% and can reduce Time to First Token (TTFT) latency by up to 80%, but those gains assume your request lands on the replica that already has the relevant prefix cached. Under naive round-robin load balancing across N replicas, that probability is 1/N. The cache hit rate that made caching so attractive at one replica degrades almost linearly as your fleet grows. Solving this requires rethinking how requests are routed at the infrastructure level. This article covers the load balancing strategies and specialized routers that preserve cache efficiency at scale, starting with why standard approaches fall short and progressing to precise, cache-aware routing techniques. Inferencing engines To achieve large-scale inferencing, we use inference engines. These engines simplify the complexities of serving LLMs and offer improved resource utilization on the underlying GPUs. They also enable higher concurrency and allow for customization to suit diverse inference workloads, such as real-time chat completions and long-form document summarization. Noteworthy engine options include vLLM, SGLang, and TensorRT. The inferencing process is largely consistent across different engines. Sending an HTTP request to an engine initiates a standard sequence of steps. Prefill Phase: The input prompt is first converted into token IDs using the model's tokenizer. Requests are grouped into batches for efficient concurrent processing by the engine. During this initial processing, special Key (K) and Value (V) tensors are computed. This phase concludes after the first forward pass, resulting in the generation of the first output token. Decode Phase: This phase involves an auto-regressive loop, continuing until an end-of-sequence token is generated or the maximum sequence length is re ## Upcoming Livestream: GPUs for AI (Shaped by You) DevFeed: [Upcoming Livestream: GPUs for AI (Shaped by You)]() Original publisher: [Read original article]() Author: Alex Razvant Published: 2026-01-27T09:30:49Z Content type: article Language: en Sources: [Neural Bits]() Topics: [GPU](), [Artificial Intelligence](), [Hardware](), [CUDA](), [pcie](), [Nvidia](), [rocm](), [MLX](), [TensorRT](), [Google](), [groq]() Tags: [ai](), [cuda](), [google](), [gpu](), [groq](), [hardware](), [mlx](), [nvidia](), [pcie](), [rocm](), [tensorrt]() ### AI overview An upcoming livestream will discuss how GPUs and other accelerators support AI workloads. The author invites audience feedback to shape coverage of GPU hardware, PCIe, CUDA, ASICs, TPUs, LPUs, model optimization, and related technologies. ### Source excerpt You can choose the topics for a Live Session on GPUs in AI ## What's new in TensorFlow 2.17 DevFeed: [What's new in TensorFlow 2.17]() Original publisher: [Read original article]() Author: TensorFlow Blog (noreply@blogger.com) Published: 2024-07-18T16:00:00Z Content type: release Language: en Sources: [The TensorFlow Blog]() Topics: [releases]() Tags: [announcement](), [cuda](), [gpu](), [keras](), [python](), [tensorflow](), [tensorflow-core](), [tensorrt]() ### AI overview TensorFlow 2.17 adds dedicated CUDA kernels for compute capability 8.9 GPUs and ends precompiled Python-package support for compute capability 5.0. The article also previews NumPy 2.0 support in TensorFlow 2.18 and the removal of TensorRT support after 2.17. ### Source excerpt Posted by the TensorFlow team TensorFlow 2.17 has been released! Highlights of this release (and 2.16) include CUDA update, upcoming Numpy 2.0, and more. For the full release notes, please click here. Note: Release updates on the new multi-backend Keras will be published on keras.io, starting with Keras 3.0. For more information, please see https://keras.io/keras_3/. TensorFlow Core CUDA Update TensorFlow binary distributions now ship with dedicated CUDA kernels for GPUs with a compute capability of 8.9. This improves the performance on the popular Ada-Generation GPUs like NVIDIA RTX 40**, L4 and L40. To keep Python wheel sizes in check, we made the decision to no longer ship CUDA kernels for compute capability 5.0. That means the oldest NVIDIA GPU generation supported by the precompiled Python packages is now the Pascal generation (compute capability 6.0). For Maxwell support, we either recommend sticking with TensorFlow version 2.16, or compiling TensorFlow from source. The latter will be possible as long as the used CUDA version still supports Maxwell GPUs. Numpy 2.0 Upcoming TensorFlow 2.18 release will include support for Numpy 2.0. This may break some edge cases of TensorFlow API usage. Drop TensorRT support Starting with TensorFlow 2.18, support for TensorRT will be dropped. TensorFlow 2.17 will be the last version to include it. ## TensorFlow 2.15 update: hot-fix for Linux installation issue DevFeed: [TensorFlow 2.15 update: hot-fix for Linux installation issue]() Original publisher: [Read original article]() Author: TensorFlow Blog (noreply@blogger.com) Published: 2023-12-05T22:00:00Z Content type: release Language: en Sources: [The TensorFlow Blog]() Topics: [pip](), [Linux](), [TensorRT](), [CUDA](), [Python]() Tags: [announcement](), [dependencies](), [explore](), [installation](), [keras](), [linux](), [nvdia](), [nvidia](), [packages](), [release](), [tensorflow](), [tensorflow-core](), [tensorrt](), [update](), [x86-64]() ### AI overview TensorFlow released version 2.15.0.post1 to fix Linux x86_64 installation problems caused by missing TensorRT-related Python dependencies when installing TensorFlow 2.15 with NVIDIA CUDA dependencies via pip. The hotfix restores the intended installation behavior, while users pinning versions must specify 2.15.0.post1 explicitly. ### Source excerpt Posted by the TensorFlow team We are releasing a hot-fix for an installation issue affecting the TensorFlow installation process. The TensorFlow 2.15.0 Python package was released such that it requested tensorrt-related packages that cannot be found unless the user installs them beforehand or provides additional installation flags. This dependency affected anyone installing TensorFlow 2.15 alongside NVIDIA CUDA dependencies via pip install tensorflow[and-cuda]. Depending on the installation method, TensorFlow 2.14 would be installed instead of 2.15, or users could receive an installation error due to those missing dependencies. To solve this issue as quickly as possible, we have released TensorFlow 2.15.0.post1 for the Linux x86_64 platform. This version removes the tensorrt Python package dependencies from the tensorflow[and-cuda] installation method. Support for TensorRT is otherwise unaffected as long as TensorRT is already installed on the system. Now, pip install tensorflow[and-cuda] works as originally intended for TensorFlow 2.15. Using .post1 instead of a full minor release allowed us to push this release out quickly. However, please be aware of the following caveat: for users wishing to pin their Python dependency in a requirements file or other situation, under Python's version specification rules, tensorflow[and-cuda]==2.15.0 will not install this fixed version. Please use ==2.15.0.post1 to specify this exact version on Linux platforms, or a fuzzy version specification, such as ==2.15.*, to specify the most recent compatible version of TensorFlow 2.15 on all platforms.