# Dynamo-Triton

Dynamo-Triton is open-source software for deploying AI models across major frameworks and platforms, providing production inference features such as dynamic batching and concurrent execution.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Doubling Query Embedding Throughput with Dynamic Batching in Vespa Cloud

DevFeed: [Doubling Query Embedding Throughput with Dynamic Batching in Vespa Cloud](<https://devfeed.tech/articles/doubling-query-embedding-throughput-with-dynamic-batching-in-vespa-cloud-144297.md>)

Original publisher: [Read original article](<https://blog.vespa.ai/doubling-query-embedding-throughput-with-dynamic-batching/>)

Author: gleb ingrid

Published: 2026-11-18T00:00:00Z

Content type: article

Language: en

Sources: [Vespa Blog](<https://devfeed.tech/sources/vespa-blog.md>)

Topics: [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [Dynamo-Triton](<https://devfeed.tech/topics/dynamo-triton.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>)

Tags: [batching](<https://devfeed.tech/tags/batching.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [efficiently](<https://devfeed.tech/tags/efficiently.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [embedding-model](<https://devfeed.tech/tags/embedding-model.md>), [embeddinggemma-2](<https://devfeed.tech/tags/embeddinggemma-2.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [latency](<https://devfeed.tech/tags/latency.md>), [local](<https://devfeed.tech/tags/local.md>), [models](<https://devfeed.tech/tags/models.md>), [onnx](<https://devfeed.tech/tags/onnx.md>), [performance](<https://devfeed.tech/tags/performance.md>), [queries](<https://devfeed.tech/tags/queries.md>), [query](<https://devfeed.tech/tags/query.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [vespa-cloud](<https://devfeed.tech/tags/vespa-cloud.md>)

### AI overview

Vespa Cloud's inference runtime (Vespa 8.784.16) uses NVIDIA Triton to queue concurrent embedding requests and combine waiting requests into GPU batches. A concurrency limit creates the queue that makes batching possible; maximum batch size and token padding buckets are configurable, and the runtime does not wait for batches to fill. In tests on an NVIDIA L4 with short queries, batch size 8, buckets 32 and 64, and concurrency 1 served twice the query load at the same median latency; peak throughput was about 2.5 times higher, with lower p99 latency at all but low loads. The benchmark included queueing, embedding, search, and result fetching, but excluded rendering and network round trips. The article recommends leaving batching defaults on CPU nodes, where cores already process requests in parallel and padding adds work; long inputs may also gain little from batching. The feature is for Vespa Cloud container nodes.

### Source excerpt

Vespa Cloud now supports dynamic batching of concurrent requests for local embedding models. In our benchmarks, this more than doubles the number of queries embedded per second on a GPU, while adding just a few milliseconds of latency at low loads.

## Leading European Financial Services Organizations Embrace Open Models to Build Proprietary Intelligence

DevFeed: [Leading European Financial Services Organizations Embrace Open Models to Build Proprietary Intelligence](<https://devfeed.tech/articles/leading-european-financial-services-organizations-embrace-open-models-to-build-proprietary-intelligence-95045.md>)

Original publisher: [Read original article](<https://blogs.nvidia.com/blog/european-financial-services-organizations-open-models-proprietary-intelligence/>)

Author: Jason Renders

Published: 2026-10-22T05:00:56Z

Content type: news

Language: en

Sources: [NVIDIA](<https://devfeed.tech/sources/nvidia-blog.md>)

Topics: [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>), [ai security](<https://devfeed.tech/topics/ai-security.md>), [Nvidia](<https://devfeed.tech/topics/nvidia.md>), [Payment Fraud](<https://devfeed.tech/topics/payment-fraud.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [digital sovereignty](<https://devfeed.tech/topics/digital-sovereignty.md>), [RAPIDS](<https://devfeed.tech/topics/rapids.md>), [Nemotron](<https://devfeed.tech/topics/nemotron.md>), [Dynamo-Triton](<https://devfeed.tech/topics/dynamo-triton.md>)

Tags: [accelerated-computing](<https://devfeed.tech/tags/accelerated-computing.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [banking](<https://devfeed.tech/tags/banking.md>), [banks](<https://devfeed.tech/tags/banks.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [blackwell-ultra](<https://devfeed.tech/tags/blackwell-ultra.md>), [civo](<https://devfeed.tech/tags/civo.md>), [cudf](<https://devfeed.tech/tags/cudf.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [detection](<https://devfeed.tech/tags/detection.md>), [dynamo-triton](<https://devfeed.tech/tags/dynamo-triton.md>), [financial](<https://devfeed.tech/tags/financial.md>), [financial-services](<https://devfeed.tech/tags/financial-services.md>), [gtc-2026](<https://devfeed.tech/tags/gtc-2026.md>), [nvidia-blackwell](<https://devfeed.tech/tags/nvidia-blackwell.md>), [open](<https://devfeed.tech/tags/open.md>), [open-weight-models](<https://devfeed.tech/tags/open-weight-models.md>), [revolut](<https://devfeed.tech/tags/revolut.md>)

### AI overview

The excerpt presents European financial organizations using open or specialized models and accelerated computing on their own or sovereign infrastructure to build financial intelligence. Revolut is scaling its PRAGMA foundation model with NVIDIA cuDF and Dynamo-Triton on Blackwell Ultra GPUs for an initial 80 million customers, with engineering for 3x scale. Starling trained Starling Intelligence on its transaction data in U.K. sovereign cloud; Lloyds built ATUM from banking transactions using RTX PRO 6000 Blackwell GPUs; SBS offers a platform for models that keep data within banks. Other examples include bunq's Finn, reported to match leading closed-model quality at 60% of the cost per answer on Nebius, and Santander's use of NVIDIA Kumo models for structured financial data. For research, LSEG reports up to 10x faster historical tick-data analysis with Polars GPU and cuDF. For fraud pipelines, HSBC cites 5x faster data processing, while Stripe reports 42x faster pipelines and 92% cost savings in offline testing with GPU-accelerated Ray Data.

### Source excerpt

At GTC Berlin, banks, payment companies and financial data providers are showing what production-scale intelligence looks like -- built on NVIDIA accelerated computing, powered by open weight models and running on their own infrastructure. Revolut is rebuilding its technology stack on a single foundation model PRAGMA, powered by NVIDIA accelerated computing. Using the NVIDIA cuDF [...]

## CoreWeave targets AI inference bottlenecks with full-stack optimization

DevFeed: [CoreWeave targets AI inference bottlenecks with full-stack optimization](<https://devfeed.tech/articles/coreweave-targets-ai-inference-bottlenecks-with-full-stack-optimization-77535.md>)

Original publisher: [Read original article](<https://siliconangle.com/2026/10/08/ai-inference-gets-full-stack-coreweave-launches-forge-fullyconnected/>)

Author: Jonathan Anthony

Published: 2026-10-09T02:30:18Z

Content type: news

Language: en

Sources: [SiliconANGLE](<https://devfeed.tech/sources/siliconangle.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Managed Inference and Agents](<https://devfeed.tech/topics/managed-inference-and-agents.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Dynamo-Triton](<https://devfeed.tech/topics/dynamo-triton.md>), [AWS Trainium](<https://devfeed.tech/topics/aws-trainium.md>)

Tags: [accelerated-computing](<https://devfeed.tech/tags/accelerated-computing.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-cloud](<https://devfeed.tech/tags/ai-cloud.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [ai-networking](<https://devfeed.tech/tags/ai-networking.md>), [ai-operationalization](<https://devfeed.tech/tags/ai-operationalization.md>), [ai-storage](<https://devfeed.tech/tags/ai-storage.md>), [cloud-computing](<https://devfeed.tech/tags/cloud-computing.md>), [coreweave](<https://devfeed.tech/tags/coreweave.md>), [coreweave-forge](<https://devfeed.tech/tags/coreweave-forge.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cube-event-coverage](<https://devfeed.tech/tags/cube-event-coverage.md>), [data-center-cooling](<https://devfeed.tech/tags/data-center-cooling.md>), [data-centers](<https://devfeed.tech/tags/data-centers.md>), [dave-vellante](<https://devfeed.tech/tags/dave-vellante.md>), [developer](<https://devfeed.tech/tags/developer.md>), [developer-experience](<https://devfeed.tech/tags/developer-experience.md>), [development](<https://devfeed.tech/tags/development.md>), [dynamo](<https://devfeed.tech/tags/dynamo.md>), [forge](<https://devfeed.tech/tags/forge.md>), [fully-connected-2026](<https://devfeed.tech/tags/fully-connected-2026.md>), [fullyconnected](<https://devfeed.tech/tags/fullyconnected.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-infrastructure](<https://devfeed.tech/tags/gpu-infrastructure.md>), [gpu-utilization](<https://devfeed.tech/tags/gpu-utilization.md>), [john-furrier](<https://devfeed.tech/tags/john-furrier.md>), [managed-services](<https://devfeed.tech/tags/managed-services.md>), [model-training](<https://devfeed.tech/tags/model-training.md>), [models](<https://devfeed.tech/tags/models.md>), [news](<https://devfeed.tech/tags/news.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [observability](<https://devfeed.tech/tags/observability.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [production-ai](<https://devfeed.tech/tags/production-ai.md>), [software](<https://devfeed.tech/tags/software.md>), [thecube](<https://devfeed.tech/tags/thecube.md>), [urvashi-chowdhary](<https://devfeed.tech/tags/urvashi-chowdhary.md>)

### AI overview

CoreWeave is expanding beyond GPU capacity with managed services for training, post-training, and inference. The article describes layer-by-layer inference optimization using vLLM, quantized models, and custom speculative decoders, and reports that its preview RL Rollouts capability improved model reload latency by 15x in testing. CoreWeave Forge brings serving, observability, post-training, and evaluation together, with a free entry tier and paid tiers offering additional capabilities.

### Source excerpt

AI inference is fast becoming the workload that decides the economics of the AI boom. Training built the first wave of GPU clouds, but serving models faster and cheaper will define the next. That shift is pushing specialized cloud providers beyond raw GPU capacity into storage, networking and software. One provider is layering managed services [...] The post CoreWeave targets AI inference bottlenecks with full-stack optimization appeared first on SiliconANGLE.

## Scality AI Inference Factory Serves KV Cache From Object Storage Over RDMA, 14x Faster Than Recompute

DevFeed: [Scality AI Inference Factory Serves KV Cache From Object Storage Over RDMA, 14x Faster Than Recompute](<https://devfeed.tech/articles/scality-ai-inference-factory-serves-kv-cache-from-object-storage-over-rdma-14x-faster-than-recompute-77324.md>)

Original publisher: [Read original article](<https://www.storagereview.com/news/scality-ai-inference-factory-kv-cache-object-storage-rdma>)

Author: Harold Fritts

Published: 2026-10-08T16:32:44Z

Content type: news

Language: en

Sources: [StorageReview.com](<https://devfeed.tech/sources/storagereview-com.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [rdma](<https://devfeed.tech/topics/rdma.md>), [gpudirect rdma](<https://devfeed.tech/topics/gpudirect-rdma.md>), [Dynamo-Triton](<https://devfeed.tech/topics/dynamo-triton.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [versioning](<https://devfeed.tech/topics/versioning.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-data-infrastructure](<https://devfeed.tech/tags/ai-data-infrastructure.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [code](<https://devfeed.tech/tags/code.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [stack](<https://devfeed.tech/tags/stack.md>)

### AI overview

Scality launched AI Inference Factory, an open-code stack for running open-weight models on customer-owned infrastructure. It uses ADI object storage as a shared KV cache, with vLLM and Dynamo separating prefill and decode, and an RDMA path for moving cache data to GPUs. In a small lab, Scality reported that restoring a 14K-token context took 166 ms, compared with 2.3 seconds to recompute it; it also reported serving 8 TB of KV cache with first-token latency between 355 and 373 ms across the tested cache sizes.

### Source excerpt

Scality today launched Scality AI Inference Factory, an open-code software stack for running open-weight models on infrastructure the customer owns. Scality validates, ships, and maintains the whole stack, which places its AI Data Infrastructure (ADI) object storage under the inference layer as a shared key-value (KV) cache, and it is aimed at enterprises, government agencies, The post Scality AI Inference Factory Serves KV Cache From Object Storage Over RDMA, 14x Faster Than Recompute appeared first on StorageReview.com.

## Deploying an HSTU Generative Recommender with NVIDIA Dynamo-Triton

DevFeed: [Deploying an HSTU Generative Recommender with NVIDIA Dynamo-Triton](<https://devfeed.tech/articles/deploying-an-hstu-generative-recommender-with-nvidia-dynamo-triton-62868.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/deploying-an-hstu-generative-recommender-with-nvidia-dynamo-triton/>)

Author: Tanya Lenz

Published: 2026-09-30T20:54:59Z

Content type: tutorial

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [Dynamo-Triton](<https://devfeed.tech/topics/dynamo-triton.md>), [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>)

Tags: [agentic-ai-generative-ai](<https://devfeed.tech/tags/agentic-ai-generative-ai.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [batch](<https://devfeed.tech/tags/batch.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [developer-tools-techniques](<https://devfeed.tech/tags/developer-tools-techniques.md>), [dynamo-triton](<https://devfeed.tech/tags/dynamo-triton.md>), [edge-computing](<https://devfeed.tech/tags/edge-computing.md>), [featured](<https://devfeed.tech/tags/featured.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [latency](<https://devfeed.tech/tags/latency.md>), [low-latency-inference](<https://devfeed.tech/tags/low-latency-inference.md>), [model](<https://devfeed.tech/tags/model.md>), [modern](<https://devfeed.tech/tags/modern.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [performance](<https://devfeed.tech/tags/performance.md>), [practical](<https://devfeed.tech/tags/practical.md>), [production](<https://devfeed.tech/tags/production.md>), [python](<https://devfeed.tech/tags/python.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [recommender-system](<https://devfeed.tech/tags/recommender-system.md>), [recommenders-personalization](<https://devfeed.tech/tags/recommenders-personalization.md>)

### AI overview

This tutorial explains how to deploy an HSTU generative recommender from PyTorch development to production inference using NVIDIA Dynamo-Triton, PyTorch AOTI, and FlexKV. It covers model export and compilation, validation in Python and C++, and GPU-backed KV caching. In the stated benchmark, caching achieved best-case speedups of up to 4.47x for a three-layer model and 5.93x for an eight-layer model at a 100% GPU KV-cache hit rate.

### Source excerpt

Generative recommender (GR) systems are emerging as a powerful new approach for large-scale personalization. Instead of treating recommendation as a set of...

## Simplifying Model Serving Across Multiple GPUs with NVIDIA TensorRT Multi-Device Integration in NVIDIA Dynamo-Triton

DevFeed: [Simplifying Model Serving Across Multiple GPUs with NVIDIA TensorRT Multi-Device Integration in NVIDIA Dynamo-Triton](<https://devfeed.tech/articles/simplifying-model-serving-across-multiple-gpus-with-nvidia-tensorrt-multi-device-integration-in-nvidia-dynamo-triton-57601.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/simplifying-model-serving-across-multiple-gpus-with-nvidia-tensorrt-multi-device-integration-in-nvidia-dynamo-triton/>)

Author: Tanya Lenz

Published: 2026-09-21T21:51:06Z

Content type: tutorial

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [TensorRT](<https://devfeed.tech/topics/tensorrt.md>), [Dynamo-Triton](<https://devfeed.tech/topics/dynamo-triton.md>), [model-serving](<https://devfeed.tech/topics/model-serving.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>), [CUDA](<https://devfeed.tech/topics/cuda.md>), [gRPC](<https://devfeed.tech/topics/grpc.md>), [Parallelism](<https://devfeed.tech/topics/parallelism.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [Deployment](<https://devfeed.tech/topics/deployment.md>)

Tags: [agentic-ai-generative-ai](<https://devfeed.tech/tags/agentic-ai-generative-ai.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [backend](<https://devfeed.tech/tags/backend.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [developer-tools-techniques](<https://devfeed.tech/tags/developer-tools-techniques.md>), [dynamo-triton](<https://devfeed.tech/tags/dynamo-triton.md>), [featured](<https://devfeed.tech/tags/featured.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [grpc](<https://devfeed.tech/tags/grpc.md>), [inference](<https://devfeed.tech/tags/inference.md>), [inference-performance](<https://devfeed.tech/tags/inference-performance.md>), [latency](<https://devfeed.tech/tags/latency.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [multi-gpu](<https://devfeed.tech/tags/multi-gpu.md>), [nccl](<https://devfeed.tech/tags/nccl.md>), [networking-communications](<https://devfeed.tech/tags/networking-communications.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [tensorrt](<https://devfeed.tech/tags/tensorrt.md>)

### AI overview

This tutorial explains how TensorRT multi-device inference runs a single network across multiple GPUs using NCCL-backed collectives, while Dynamo-Triton exposes one gRPC model endpoint. It demonstrates the integration with NVIDIA Cosmos 3 Nano video generation and describes distributing the denoising transformer across up to eight GPUs.

### Source excerpt

The compute and memory demands of generative AI increasingly exceed what a single GPU can provide. NVIDIA TensorRT multi-device inference is a new capability...

## How to Connect Snowflake to Estuary Using PrivateLink and BYOC

DevFeed: [How to Connect Snowflake to Estuary Using PrivateLink and BYOC](<https://devfeed.tech/articles/secure-snowflake-deploying-industry-grade-snowflake-integrations-82368.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/secure-snowflake-integration-privatelink-byoc>)

Author: Jaume Boguñá

Published: 2026-04-27T17:14:22Z

Content type: tutorial

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Cloud Architecture](<https://devfeed.tech/topics/cloud-architecture.md>), [Cloud Networking](<https://devfeed.tech/topics/cloud-networking.md>), [Dynamo-Triton](<https://devfeed.tech/topics/dynamo-triton.md>)

Tags: [aws-privatelink](<https://devfeed.tech/tags/aws-privatelink.md>), [byoc-deployment](<https://devfeed.tech/tags/byoc-deployment.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-infrastructure](<https://devfeed.tech/tags/cloud-infrastructure.md>), [cloud-security](<https://devfeed.tech/tags/cloud-security.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [private-cloud](<https://devfeed.tech/tags/private-cloud.md>), [security](<https://devfeed.tech/tags/security.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [snowflake-integration](<https://devfeed.tech/tags/snowflake-integration.md>)

### AI overview

This tutorial explains how to connect Estuary to Snowflake through PrivateLink in a private or Bring Your Own Cloud (BYOC) deployment. It covers enabling private connectivity with Snowflake, sharing endpoint details with Estuary, configuring the Snowflake destination, and verifying that traffic uses the private endpoint. The guide presents this approach as a way to keep data off public networks and support security and compliance requirements.

### Source excerpt

Learn how to securely connect Snowflake using PrivateLink and BYOC in order to keep data off the public internet while meeting security and compliance requirements.

## Together AI announces inference, agent, and voice AI offerings at NVIDIA GTC 2026

DevFeed: [Together AI announces inference, agent, and voice AI offerings at NVIDIA GTC 2026](<https://devfeed.tech/articles/together-ai-at-nvidia-gtc-2026-explore-our-latest-innovations-across-research-and-products-80220.md>)

Original publisher: [Read original article](<https://www.together.ai/blog/together-ai-at-nvidia-gtc-2026>)

Author: Hiral Jasani

Published: 2026-03-16T00:00:00Z

Content type: news

Language: en

Sources: [Together.ai](<https://devfeed.tech/sources/together-ai.md>)

Topics: [Dynamo-Triton](<https://devfeed.tech/topics/dynamo-triton.md>), [OpenShell](<https://devfeed.tech/topics/openshell.md>), [Nemotron](<https://devfeed.tech/topics/nemotron.md>), [NemoClaw](<https://devfeed.tech/topics/nemoclaw.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [voice-ai](<https://devfeed.tech/tags/voice-ai.md>)

### AI overview

At NVIDIA GTC 2026, Together AI highlights work with NVIDIA on Dynamo 1.0 and NemoClaw, access to Nemotron 3 Super and Parakeet TDT 0.6B V3, and sessions on production inference and AI applications. The announcements focus on tools and infrastructure for developers building and deploying AI systems.

### Source excerpt

Together AI arrives at NVIDIA GTC 2026 with new launches in inference, agents, voice AI, and open models -- plus technical sessions from its research and engineering leaders.

## Migrating to Nvidia Triton: High-Throughput, Low-Cost Inference at Scale

DevFeed: [Migrating to Nvidia Triton: High-Throughput, Low-Cost Inference at Scale](<https://devfeed.tech/articles/migrating-to-nvidia-triton-high-throughput-low-cost-inference-at-scale-35070.md>)

Original publisher: [Read original article](<https://medium.com/gumgum-tech/migrating-to-nvidia-triton-high-throughput-low-cost-inference-at-scale-24af3ada4210?source=rss----d4c1dee0f87b---4>)

Author: Eric Luu

Published: 2025-09-19T01:05:13Z

Content type: tutorial

Language: en

Sources: [GumGum](<https://devfeed.tech/sources/gumgum.md>)

Topics: [Dynamo-Triton](<https://devfeed.tech/topics/dynamo-triton.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [dataloaders](<https://devfeed.tech/topics/dataloaders.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>)

Tags: [automatic](<https://devfeed.tech/tags/automatic.md>), [backends](<https://devfeed.tech/tags/backends.md>), [batch](<https://devfeed.tech/tags/batch.md>), [batching](<https://devfeed.tech/tags/batching.md>), [code](<https://devfeed.tech/tags/code.md>), [containers](<https://devfeed.tech/tags/containers.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [data](<https://devfeed.tech/tags/data.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [decoder](<https://devfeed.tech/tags/decoder.md>), [docker](<https://devfeed.tech/tags/docker.md>), [docker-images](<https://devfeed.tech/tags/docker-images.md>), [ecs](<https://devfeed.tech/tags/ecs.md>), [file](<https://devfeed.tech/tags/file.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [scale](<https://devfeed.tech/tags/scale.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [triton-inference-server](<https://devfeed.tech/tags/triton-inference-server.md>)

### AI overview

GumGum reports that migrating NLP inference from Databricks streaming jobs to Nvidia Triton reduced costs by 90% while processing about 750,000 requests per hour on 2-4 g4dn.xlarge servers. Triton's model ensembles, dynamic batching, scheduling and built-in Prometheus metrics supported serving and tuning the pipeline. The migration split CPU tokenization and GPU inference into separate models; the team used queue, compute and utilization metrics to identify a GPU tokenizer as the bottleneck, then adjusted batch sizes to shift GPU capacity toward it. The article's Python-backend examples show that dynamic batching combines request objects rather than merging their tensors: `execute` must return one response per request, with output shapes matching each input batch. Combining client-side micro-batches with dynamic batching therefore requires flattening and reassembling data, which the author advises avoiding. The reported gains required tuning; specialized backends and shared-memory optimizations remained unexplored.

### Source excerpt

Photo by Igor Omilaev on Unsplash There are tons of guides on what Nvidia Triton is and how to set it up, so I'll only go over those very briefly, instead focusing on the migration we've had and the small quirks that come from working with Python Backends. Nvidia Triton is an open source inference serving software, capable of serving virtually any machine learning model. Real time processing, batching, dynamic batching, ensembles, specialized backends, all possible with enough knobs and dials to maximize hardware use and minimize cost. Core features include File based model repository to store your models. If wanted, can be dynamically loaded onto the server Ensemble capabilities, can define whole inference pipelines, even ones that branch, on triton Ability to define each step in the inference pipeline as its own model (e.g., tokenizer -> model -> decoder), which can get hit as its own endpoint Dynamic batching between models, even in ensembles Automatic scheduling of requests between models and ensembles Minimal data transfers between GPU & CPU Our Previous Solution At GumGum we've previously deployed the bulk of our NLP inference on Databricks. These would run 24 hour jobs loading custom Docker images, streaming inputs from a Kafka topic and relying on UDF's to contain our inference logic. This was a move to mitigate issues we had on ECS related to starvation, scalability, and parallelism. Workers being paired on consumers lead to starvation. As a result, scalability of the service suffered and maintaining threading code became a pain. We've also experienced other issues as well: GPU metrics were not easily available, as we need to use an ML runtime, which was not compatible with docker containers (Although modern runtime do support this more). Latency was high as with Kafka structured streaming, microbatches must wait for all workers to finish. So if one worker hangs, the entire batch hangs. Performance on PySpark was always tough. Serializing between Java to Pyth