# Migrating to Nvidia Triton: High-Throughput, Low-Cost Inference at Scale

DevFeed: [Migrating to Nvidia Triton: High-Throughput, Low-Cost Inference at Scale](<https://devfeed.tech/articles/migrating-to-nvidia-triton-high-throughput-low-cost-inference-at-scale-35070.md>)

Original publisher: [Read original article](<https://medium.com/gumgum-tech/migrating-to-nvidia-triton-high-throughput-low-cost-inference-at-scale-24af3ada4210?source=rss----d4c1dee0f87b---4>)

Author: Eric Luu

Published: 2025-09-19T01:05:13Z

Content type: article

Language: en

Sources: [GumGum](<https://devfeed.tech/sources/gumgum.md>)

Topics: [Inference](<https://devfeed.tech/topics/inference.md>), [model-serving](<https://devfeed.tech/topics/model-serving.md>), [Nvidia](<https://devfeed.tech/topics/nvidia.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Natural language processing](<https://devfeed.tech/topics/nlp.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [Docker](<https://devfeed.tech/topics/docker.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Python](<https://devfeed.tech/topics/python.md>)

Tags: [batching](<https://devfeed.tech/tags/batching.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [docker](<https://devfeed.tech/tags/docker.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [latency](<https://devfeed.tech/tags/latency.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [triton-inference-server](<https://devfeed.tech/tags/triton-inference-server.md>)

## AI overview

This article describes GumGum's migration of NLP inference from Databricks-based jobs using Docker images, Kafka streaming, and UDFs toward NVIDIA Triton. It introduces Triton's model-serving features, including batching, ensembles, dynamic model loading, request scheduling, and reduced GPU-CPU data transfers, while outlining limitations of the previous approach such as latency, scalability, GPU-metrics, and PySpark performance issues.

## Source excerpt

Photo by Igor Omilaev on Unsplash There are tons of guides on what Nvidia Triton is and how to set it up, so I'll only go over those very briefly, instead focusing on the migration we've had and the small quirks that come from working with Python Backends. Nvidia Triton is an open source inference serving software, capable of serving virtually any machine learning model. Real time processing, batching, dynamic batching, ensembles, specialized backends, all possible with enough knobs and dials to maximize hardware use and minimize cost. Core features include File based model repository to store your models. If wanted, can be dynamically loaded onto the server Ensemble capabilities, can define whole inference pipelines, even ones that branch, on triton Ability to define each step in the inference pipeline as its own model (e.g., tokenizer -> model -> decoder), which can get hit as its own endpoint Dynamic batching between models, even in ensembles Automatic scheduling of requests between models and ensembles Minimal data transfers between GPU & CPU Our Previous Solution At GumGum we've previously deployed the bulk of our NLP inference on Databricks. These would run 24 hour jobs loading custom Docker images, streaming inputs from a Kafka topic and relying on UDF's to contain our inference logic. This was a move to mitigate issues we had on ECS related to starvation, scalability, and parallelism. Workers being paired on consumers lead to starvation. As a result, scalability of the service suffered and maintaining threading code became a pain. We've also experienced other issues as well: GPU metrics were not easily available, as we need to use an ML runtime, which was not compatible with docker containers (Although modern runtime do support this more). Latency was high as with Kafka structured streaming, microbatches must wait for all workers to finish. So if one worker hangs, the entire batch hangs. Performance on PySpark was always tough. Serializing between Java to Pyth