# nvidia-nsight-systems

Published articles for nvidia-nsight-systems.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## A Practical Loop for Optimizing GPU Inference Throughput

DevFeed: [A Practical Loop for Optimizing GPU Inference Throughput](<https://devfeed.tech/articles/why-100-gpu-util-wasn-t-enough-22606.md>)

Original publisher: [Read original article](<https://medium.com/bumble-tech/why-100-gpu-util-wasnt-enough-d386d3979cb9?source=rss----6353b5325b1a---4>)

Author: Andrei Potapkin

Published: 2026-05-01T17:57:59Z

Content type: article

Language: en

Sources: [Bumble Tech](<https://devfeed.tech/sources/bumble-tech.md>)

Topics: [Inference](<https://devfeed.tech/topics/inference.md>), [Optimization](<https://devfeed.tech/topics/optimization.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [TensorRT](<https://devfeed.tech/topics/tensorrt.md>), [onnx](<https://devfeed.tech/topics/onnx.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [migration](<https://devfeed.tech/topics/migration.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-migration](<https://devfeed.tech/tags/cloud-migration.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [nvidia-nsight-systems](<https://devfeed.tech/tags/nvidia-nsight-systems.md>), [onnx](<https://devfeed.tech/tags/onnx.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [python](<https://devfeed.tech/tags/python.md>), [tensorrt](<https://devfeed.tech/tags/tensorrt.md>)

### AI overview

This practical article describes a production-focused loop for optimizing GPU inference: benchmark under representative load, identify the current throughput bottleneck, fix it, and verify the operational outcome. In the described ML moderation service, TensorRT, BF16, and an attention-subgraph rewrite increased throughput from roughly 150 to roughly 1100 requests per second in the same saturated benchmark.

### Source excerpt

A practical loop for optimizing GPU inference without guessing It wasn't an incident. That made it interesting. A few days after a cloud migration, one of our highest-traffic ML services looked healthy. Traffic was flowing. Latency was fine. The rollout had gone the way you want these things to go: boringly. And yet a Slack thread started anyway. Not because anything was broken. Because something had become easier to see. On-prem, GPU efficiency tends to get folded into bigger conversations: capacity planning, hardware timelines, headroom for the next model. In the cloud, the same question sharpens. Throughput, utilization, and cost sit much closer together. A service can be perfectly healthy and still look expensive. That was our situation. We had a frontend text moderation model based on mT5, served through NVIDIA Triton as a two-component ensemble: a Python tokenizer and a static ONNX model. The service was stable, but throughput per GPU was much lower than our back-of-the-envelope estimates suggested Moments like that attract the usual optimization advice: Batch more aggressively. Tune Triton. Rewrite the Python stages. Enable TensorRT. Drop precision. None of that advice is silly. Some of it is very good. The problem is that it usually arrives before the diagnosis. GPU systems are awkward that way. By the time you go hunting for "the optimization", you are often already asking the wrong question. There usually isn't one magical fix. There is a bottleneck somewhere in the stack, and then another one behind it, and usually another one behind that. The loop that kept us honest was simple: TL;DR Benchmark the system under production-shaped load. Find the bottleneck limiting throughput right now. Fix that bottleneck. Check whether the operational outcome changed. In our case, that loop took us from roughly 150 requests per second to roughly 1100 in the same saturated benchmark: a bit over 7x baseline. TensorRT delivered the first big jump. BF16 moved the ceiling aga