# TensorRT

NVIDIA TensorRT is an SDK for optimizing deep learning inference on NVIDIA GPUs, compiling trained models into deployment engines.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor

DevFeed: [TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor](<https://devfeed.tech/articles/tensorrt-edge-llm-completes-the-mlperf-edge-agentic-benchmark-6-4x-faster-on-jetson-agx-thor-31485.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/tensorrt-edge-llm-completes-the-mlperf-edge-agentic-benchmark-6-4x-faster-on-jetson-agx-thor/>)

Author: Elizabeth Goodman

Published: 2026-09-16T20:37:07Z

Content type: article

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [Jetson AGX Thor Developer Kit](<https://devfeed.tech/topics/jetson-agx-thor-developer-kit.md>), [Inference Performance](<https://devfeed.tech/topics/inference-performance.md>), [benchmarking](<https://devfeed.tech/topics/benchmarking.md>), [TensorRT](<https://devfeed.tech/topics/tensorrt.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [agentic-ai-generative-ai](<https://devfeed.tech/tags/agentic-ai-generative-ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [jetson](<https://devfeed.tech/tags/jetson.md>), [jetson-agx-thor-developer-kit](<https://devfeed.tech/tags/jetson-agx-thor-developer-kit.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-benchmarking](<https://devfeed.tech/tags/llm-benchmarking.md>), [mlperf](<https://devfeed.tech/tags/mlperf.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [performance](<https://devfeed.tech/tags/performance.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [tensorrt](<https://devfeed.tech/tags/tensorrt.md>), [tensorrt-llm](<https://devfeed.tech/tags/tensorrt-llm.md>), [thor](<https://devfeed.tech/tags/thor.md>)

### AI overview

This article reports that NVIDIA TensorRT Edge-LLM ran Qwen3.6-27B on a single NVIDIA Jetson AGX Thor Developer Kit for the MLPerf Inference v6.1 Edge Agentic benchmark. Using NVFP4 quantization, tree-based multi-token prediction, and KV cache reuse, it achieved 52.33 tokens per second and completed 1,007 turns in 24 minutes and 36 seconds, 6.4 times faster than the llama.cpp reference submission.

### Source excerpt

AI agents are moving from cloud data centers to vehicles, robots, and other edge devices. Unlike a chatbot that answers a single prompt, an agent works through...

## Make Long-Running NVIDIA TensorRT Engine Builds Observable and Cancelable in Python or C++

DevFeed: [Make Long-Running NVIDIA TensorRT Engine Builds Observable and Cancelable in Python or C++](<https://devfeed.tech/articles/make-long-running-nvidia-tensorrt-engine-builds-observable-and-cancelable-in-python-or-c-6877.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/make-long-running-nvidia-tensorrt-engine-builds-observable-and-cancelable-in-python-or-c/>)

Author: Michelle Horton

Published: 2026-07-22T16:35:04Z

Content type: tutorial

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [TensorRT](<https://devfeed.tech/topics/tensorrt.md>), [C++](<https://devfeed.tech/topics/c-plus-plus.md>), [Python](<https://devfeed.tech/topics/python.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Cache](<https://devfeed.tech/topics/cache.md>)

Tags: [agentic-ai-generative-ai](<https://devfeed.tech/tags/agentic-ai-generative-ai.md>), [build-ai-agents](<https://devfeed.tech/tags/build-ai-agents.md>), [c-plus-plus](<https://devfeed.tech/tags/c-plus-plus.md>), [cache](<https://devfeed.tech/tags/cache.md>), [code](<https://devfeed.tech/tags/code.md>), [developer-tools-techniques](<https://devfeed.tech/tags/developer-tools-techniques.md>), [developers](<https://devfeed.tech/tags/developers.md>), [featured](<https://devfeed.tech/tags/featured.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [python](<https://devfeed.tech/tags/python.md>), [tensorrt](<https://devfeed.tech/tags/tensorrt.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>)

### AI overview

This tutorial explains how to make long-running NVIDIA TensorRT engine builds observable and cancelable in Python and C++. It presents a thread-safe progress monitor with nested build phases, shows how to connect it to the TensorRT builder, and demonstrates cancellation through Ctrl-C or a programmatic stop signal. Progress can be routed to a terminal, IDE, service, or agent runtime.

### Source excerpt

A TensorRT engine build can take seconds to many minutes. Large strongly typed models, deep tactic search, and a cold timing cache on a brand-new GPU SKU can...

## A Practical Loop for Optimizing GPU Inference Throughput

DevFeed: [A Practical Loop for Optimizing GPU Inference Throughput](<https://devfeed.tech/articles/why-100-gpu-util-wasn-t-enough-22606.md>)

Original publisher: [Read original article](<https://medium.com/bumble-tech/why-100-gpu-util-wasnt-enough-d386d3979cb9?source=rss----6353b5325b1a---4>)

Author: Andrei Potapkin

Published: 2026-05-01T17:57:59Z

Content type: article

Language: en

Sources: [Bumble Tech](<https://devfeed.tech/sources/bumble-tech.md>)

Topics: [Inference](<https://devfeed.tech/topics/inference.md>), [Optimization](<https://devfeed.tech/topics/optimization.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [TensorRT](<https://devfeed.tech/topics/tensorrt.md>), [onnx](<https://devfeed.tech/topics/onnx.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [migration](<https://devfeed.tech/topics/migration.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-migration](<https://devfeed.tech/tags/cloud-migration.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [nvidia-nsight-systems](<https://devfeed.tech/tags/nvidia-nsight-systems.md>), [onnx](<https://devfeed.tech/tags/onnx.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [python](<https://devfeed.tech/tags/python.md>), [tensorrt](<https://devfeed.tech/tags/tensorrt.md>)

### AI overview

This practical article describes a production-focused loop for optimizing GPU inference: benchmark under representative load, identify the current throughput bottleneck, fix it, and verify the operational outcome. In the described ML moderation service, TensorRT, BF16, and an attention-subgraph rewrite increased throughput from roughly 150 to roughly 1100 requests per second in the same saturated benchmark.

### Source excerpt

A practical loop for optimizing GPU inference without guessing It wasn't an incident. That made it interesting. A few days after a cloud migration, one of our highest-traffic ML services looked healthy. Traffic was flowing. Latency was fine. The rollout had gone the way you want these things to go: boringly. And yet a Slack thread started anyway. Not because anything was broken. Because something had become easier to see. On-prem, GPU efficiency tends to get folded into bigger conversations: capacity planning, hardware timelines, headroom for the next model. In the cloud, the same question sharpens. Throughput, utilization, and cost sit much closer together. A service can be perfectly healthy and still look expensive. That was our situation. We had a frontend text moderation model based on mT5, served through NVIDIA Triton as a two-component ensemble: a Python tokenizer and a static ONNX model. The service was stable, but throughput per GPU was much lower than our back-of-the-envelope estimates suggested Moments like that attract the usual optimization advice: Batch more aggressively. Tune Triton. Rewrite the Python stages. Enable TensorRT. Drop precision. None of that advice is silly. Some of it is very good. The problem is that it usually arrives before the diagnosis. GPU systems are awkward that way. By the time you go hunting for "the optimization", you are often already asking the wrong question. There usually isn't one magical fix. There is a bottleneck somewhere in the stack, and then another one behind it, and usually another one behind that. The loop that kept us honest was simple: TL;DR Benchmark the system under production-shaped load. Find the bottleneck limiting throughput right now. Fix that bottleneck. Check whether the operational outcome changed. In our case, that loop took us from roughly 150 requests per second to roughly 1100 in the same saturated benchmark: a bit over 7x baseline. TensorRT delivered the first big jump. BF16 moved the ceiling aga

## Load Balancing and Scaling LLM Serving

DevFeed: [Load Balancing and Scaling LLM Serving](<https://devfeed.tech/articles/load-balancing-and-scaling-llm-serving-19903.md>)

Original publisher: [Read original article](<https://www.digitalocean.com/blog/load-balancing-scaling-llm-serving>)

Author: Mohammad Ashar Khan

Published: 2026-04-15T19:03:31Z

Content type: article

Language: en

Sources: [DigitalOcean](<https://devfeed.tech/sources/digitalocean.md>)

Topics: [Load Balancing](<https://devfeed.tech/topics/load-balancing.md>), [Caching](<https://devfeed.tech/topics/caching.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Routing (disambiguation)](<https://devfeed.tech/topics/routing.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [sglang](<https://devfeed.tech/topics/sglang.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Concurrency](<https://devfeed.tech/topics/concurrency.md>), [TensorRT](<https://devfeed.tech/topics/tensorrt.md>)

Tags: [caching](<https://devfeed.tech/tags/caching.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [http](<https://devfeed.tech/tags/http.md>), [inference](<https://devfeed.tech/tags/inference.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm](<https://devfeed.tech/tags/llm.md>), [load-balancing](<https://devfeed.tech/tags/load-balancing.md>), [performance](<https://devfeed.tech/tags/performance.md>), [round-robin](<https://devfeed.tech/tags/round-robin.md>), [sglang](<https://devfeed.tech/tags/sglang.md>), [tensorrt](<https://devfeed.tech/tags/tensorrt.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

This article explains why scaling LLM serving requires load-balancing strategies that account for prompt-prefix caching. It describes how naive round-robin routing reduces cache-hit probability as the number of replicas grows, and introduces cache-aware routing approaches intended to preserve cache efficiency, control costs, and improve performance. It also discusses inference engines including vLLM, SGLang, and TensorRT, along with prefill and decode processing.

### Source excerpt

Load balancing for LLMs is fundamentally different from load balancing for traditional services like web servers, APIs, or databases. Prompt caching is the reason. Prompt caching typically cuts input token costs by 50-90% and can reduce Time to First Token (TTFT) latency by up to 80%, but those gains assume your request lands on the replica that already has the relevant prefix cached. Under naive round-robin load balancing across N replicas, that probability is 1/N. The cache hit rate that made caching so attractive at one replica degrades almost linearly as your fleet grows. Solving this requires rethinking how requests are routed at the infrastructure level. This article covers the load balancing strategies and specialized routers that preserve cache efficiency at scale, starting with why standard approaches fall short and progressing to precise, cache-aware routing techniques. Inferencing engines To achieve large-scale inferencing, we use inference engines. These engines simplify the complexities of serving LLMs and offer improved resource utilization on the underlying GPUs. They also enable higher concurrency and allow for customization to suit diverse inference workloads, such as real-time chat completions and long-form document summarization. Noteworthy engine options include vLLM, SGLang, and TensorRT. The inferencing process is largely consistent across different engines. Sending an HTTP request to an engine initiates a standard sequence of steps. Prefill Phase: The input prompt is first converted into token IDs using the model's tokenizer. Requests are grouped into batches for efficient concurrent processing by the engine. During this initial processing, special Key (K) and Value (V) tensors are computed. This phase concludes after the first forward pass, resulting in the generation of the first output token. Decode Phase: This phase involves an auto-regressive loop, continuing until an end-of-sequence token is generated or the maximum sequence length is re

## Build a Domain-Specific Embedding Model in Under a Day

DevFeed: [Build a Domain-Specific Embedding Model in Under a Day](<https://devfeed.tech/articles/build-a-domain-specific-embedding-model-in-under-a-day-7379.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/nvidia/domain-specific-embedding-finetune>)

Author: Steve Han; Rucha Apte; Sean Sodha; Oliver Holworthy

Published: 2026-03-20T19:38:16Z

Content type: tutorial

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [synthetic-data](<https://devfeed.tech/topics/synthetic-data.md>), [Synthetic Data Generation](<https://devfeed.tech/topics/synthetic-data-generation.md>), [Model Development](<https://devfeed.tech/topics/model-development.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [NeMo](<https://devfeed.tech/topics/nemo.md>), [Nemotron](<https://devfeed.tech/topics/nemotron.md>), [NVIDIA NIM](<https://devfeed.tech/topics/nvidia-nim.md>), [TensorRT](<https://devfeed.tech/topics/tensorrt.md>), [Claude](<https://devfeed.tech/topics/claude.md>), [API keys](<https://devfeed.tech/topics/api-keys.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [nemo](<https://devfeed.tech/tags/nemo.md>), [nemotron](<https://devfeed.tech/tags/nemotron.md>), [nim](<https://devfeed.tech/tags/nim.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvidia-nim](<https://devfeed.tech/tags/nvidia-nim.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [synthetic-data](<https://devfeed.tech/tags/synthetic-data.md>), [synthetic-data-generation](<https://devfeed.tech/tags/synthetic-data-generation.md>)

### AI overview

A tutorial showing how to fine-tune a general-purpose embedding model for a specific domain in less than a day using synthetic question-answer pairs generated from domain documents. It covers data generation, contrastive training, retrieval evaluation, and deployment, using NVIDIA NeMo components and a Llama-Nemotron embedding model.

### Source excerpt

With a single GPU and less than a day of training time, you can transform a general-purpose embedding model into one that truly understands your domain, no manual labeling required. To help you hit the ground running, we are also releasing a ready-to-use synthetic training dataset generated from NVIDIA's public documentation using this exact pipeline.

## TensorFlow 2.15 update: hot-fix for Linux installation issue

DevFeed: [TensorFlow 2.15 update: hot-fix for Linux installation issue](<https://devfeed.tech/articles/tensorflow-2-15-update-hot-fix-for-linux-installation-issue-7403.md>)

Original publisher: [Read original article](<https://blog.tensorflow.org/2023/12/tensorflow-215-update-hot-fix-linux-installation-issue.html>)

Author: TensorFlow Blog (noreply@blogger.com)

Published: 2023-12-05T22:00:00Z

Content type: release

Language: en

Sources: [The TensorFlow Blog](<https://devfeed.tech/sources/the-tensorflow-blog.md>)

Topics: [pip](<https://devfeed.tech/topics/pip.md>), [Linux](<https://devfeed.tech/topics/linux.md>), [TensorRT](<https://devfeed.tech/topics/tensorrt.md>), [CUDA](<https://devfeed.tech/topics/cuda.md>), [Python](<https://devfeed.tech/topics/python.md>)

Tags: [announcement](<https://devfeed.tech/tags/announcement.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [explore](<https://devfeed.tech/tags/explore.md>), [installation](<https://devfeed.tech/tags/installation.md>), [keras](<https://devfeed.tech/tags/keras.md>), [linux](<https://devfeed.tech/tags/linux.md>), [nvdia](<https://devfeed.tech/tags/nvdia.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [packages](<https://devfeed.tech/tags/packages.md>), [release](<https://devfeed.tech/tags/release.md>), [tensorflow](<https://devfeed.tech/tags/tensorflow.md>), [tensorflow-core](<https://devfeed.tech/tags/tensorflow-core.md>), [tensorrt](<https://devfeed.tech/tags/tensorrt.md>), [update](<https://devfeed.tech/tags/update.md>), [x86-64](<https://devfeed.tech/tags/x86-64.md>)

### AI overview

TensorFlow released version 2.15.0.post1 to fix Linux x86_64 installation problems caused by missing TensorRT-related Python dependencies when installing TensorFlow 2.15 with NVIDIA CUDA dependencies via pip. The hotfix restores the intended installation behavior, while users pinning versions must specify 2.15.0.post1 explicitly.

### Source excerpt

Posted by the TensorFlow team We are releasing a hot-fix for an installation issue affecting the TensorFlow installation process. The TensorFlow 2.15.0 Python package was released such that it requested tensorrt-related packages that cannot be found unless the user installs them beforehand or provides additional installation flags. This dependency affected anyone installing TensorFlow 2.15 alongside NVIDIA CUDA dependencies via pip install tensorflow[and-cuda]. Depending on the installation method, TensorFlow 2.14 would be installed instead of 2.15, or users could receive an installation error due to those missing dependencies. To solve this issue as quickly as possible, we have released TensorFlow 2.15.0.post1 for the Linux x86_64 platform. This version removes the tensorrt Python package dependencies from the tensorflow[and-cuda] installation method. Support for TensorRT is otherwise unaffected as long as TensorRT is already installed on the system. Now, pip install tensorflow[and-cuda] works as originally intended for TensorFlow 2.15. Using .post1 instead of a full minor release allowed us to push this release out quickly. However, please be aware of the following caveat: for users wishing to pin their Python dependency in a requirements file or other situation, under Python's version specification rules, tensorflow[and-cuda]==2.15.0 will not install this fixed version. Please use ==2.15.0.post1 to specify this exact version on Linux platforms, or a fuzzy version specification, such as ==2.15.*, to specify the most recent compatible version of TensorFlow 2.15 on all platforms.