# Doubling Query Embedding Throughput with Dynamic Batching in Vespa Cloud

DevFeed: [Doubling Query Embedding Throughput with Dynamic Batching in Vespa Cloud](<https://devfeed.tech/articles/doubling-query-embedding-throughput-with-dynamic-batching-in-vespa-cloud-144297.md>)

Original publisher: [Read original article](<https://blog.vespa.ai/doubling-query-embedding-throughput-with-dynamic-batching/>)

Author: gleb ingrid

Published: 2026-11-18T00:00:00Z

Content type: article

Language: en

Sources: [Vespa Blog](<https://devfeed.tech/sources/vespa-blog.md>)

Topics: [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [Dynamo-Triton](<https://devfeed.tech/topics/dynamo-triton.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>)

Tags: [batching](<https://devfeed.tech/tags/batching.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [efficiently](<https://devfeed.tech/tags/efficiently.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [embedding-model](<https://devfeed.tech/tags/embedding-model.md>), [embeddinggemma-2](<https://devfeed.tech/tags/embeddinggemma-2.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [latency](<https://devfeed.tech/tags/latency.md>), [local](<https://devfeed.tech/tags/local.md>), [models](<https://devfeed.tech/tags/models.md>), [onnx](<https://devfeed.tech/tags/onnx.md>), [performance](<https://devfeed.tech/tags/performance.md>), [queries](<https://devfeed.tech/tags/queries.md>), [query](<https://devfeed.tech/tags/query.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [vespa-cloud](<https://devfeed.tech/tags/vespa-cloud.md>)

## AI overview

Vespa Cloud's inference runtime (Vespa 8.784.16) uses NVIDIA Triton to queue concurrent embedding requests and combine waiting requests into GPU batches. A concurrency limit creates the queue that makes batching possible; maximum batch size and token padding buckets are configurable, and the runtime does not wait for batches to fill. In tests on an NVIDIA L4 with short queries, batch size 8, buckets 32 and 64, and concurrency 1 served twice the query load at the same median latency; peak throughput was about 2.5 times higher, with lower p99 latency at all but low loads. The benchmark included queueing, embedding, search, and result fetching, but excluded rendering and network round trips. The article recommends leaving batching defaults on CPU nodes, where cores already process requests in parallel and padding adds work; long inputs may also gain little from batching. The feature is for Vespa Cloud container nodes.

## Source excerpt

Vespa Cloud now supports dynamic batching of concurrent requests for local embedding models. In our benchmarks, this more than doubles the number of queries embedded per second on a GPU, while adding just a few milliseconds of latency at low loads.