# VoxServe: Streaming-Centric Serving System for Speech Language Models

DevFeed: [VoxServe: Streaming-Centric Serving System for Speech Language Models](<https://devfeed.tech/articles/voxserve-streaming-centric-serving-system-for-speech-language-models-113752.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.00269>)

Author: Keisuke Kamahori, Rohan Kadekodi, Garv Ghai, Wei-Tzu Lee, Atindra Jha, Stephanie Wang, Arvind Krishnamurthy, Baris Kasikci

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Streaming](<https://devfeed.tech/topics/streaming.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [audio](<https://devfeed.tech/tags/audio.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [codec](<https://devfeed.tech/tags/codec.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-sd](<https://devfeed.tech/tags/cs-sd.md>), [eess-as](<https://devfeed.tech/tags/eess-as.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm](<https://devfeed.tech/tags/llm.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [scheduler](<https://devfeed.tech/tags/scheduler.md>), [serving](<https://devfeed.tech/tags/serving.md>), [speech](<https://devfeed.tech/tags/speech.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [tts](<https://devfeed.tech/tags/tts.md>)

## AI overview

VoxServe improves streaming SpeechLM throughput by overlapping the autoregressive backbone and audio codec on separate GPU partitions, accepting higher codec latency when it does not delay first audio or cause playback stalls. Its scheduler uses three asynchronous lanes: a large LM partition, a small background codec partition for batched nonurgent chunks, and a whole-GPU urgent lane for each request's first chunk and chunks near their playback deadlines. On Qwen3-TTS, it sustains 1.5x the request rate of the best existing system at a 200 ms time-to-first-audio target without playback stalls.

## Source excerpt

arXiv:2602.00269v2 Announce Type: replace-cross Abstract: Speech Language Models (SpeechLMs) are increasingly deployed for streaming speech generation, where audio playback must start quickly and never stall. Serving them requires running two very different model components on the same system. An autoregressive LLM backbone emits audio tokens, and a codec converts chunks of those tokens into waveforms. Existing systems run the two as separately scheduled stages, and therefore either serialize them, let them interfere, or strand capacity by placing them on separate devices. We observe that overlapping the backbone and the codec on separate parts of the GPU raises throughput. Although this increases codec latency, most chunks can tolerate it, as long as it neither lies on the critical path of time-to-first-audio (TTFA) nor causes a playback stall. Based on this insight, we present VoxServe, a SpeechLM serving system that allocates GPU resources according to the urgency of chunks. VoxServe executes work on three asynchronous lanes. The LM lane runs the backbone on a large partition of the GPU's streaming multiprocessors, the background codec lane runs on a small partition beside it, and the urgent lane uses the whole GPU. The scheduler queues each request to the LM lane and routes every chunk the backbone completes by its urgency. A request's first chunk and chunks close to their playback deadline go to the urgent lane, and the rest are batched on the background lane. On Qwen3-TTS, VoxServe sustains 1.5x the request rate of the best existing system at a 200ms TTFA target without playback stalls. The code of VoxServe is available at https://github.com/vox-serve/vox-serve.