# NanoCP: Taming Load Imbalance in Data-Expert Parallel Decoding under Variable-Length Workloads

DevFeed: [NanoCP: Taming Load Imbalance in Data-Expert Parallel Decoding under Variable-Length Workloads](<https://devfeed.tech/articles/nanocp-taming-load-imbalance-in-data-expert-parallel-decoding-under-variable-length-workloads-95145.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.21100>)

Author: Jiefei Chen, Binbin Lin, Jinming Ma, Jiangfei Duan, Haojie Duanmu, Hao Liu, Qinxiu Cheng, Xiuhong Li, Zhilin Pei, Hui Wang, Xingcheng Zhang, Dahua Lin

Published: 2026-10-22T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [Load Balancing](<https://devfeed.tech/topics/load-balancing.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [aot](<https://devfeed.tech/tags/aot.md>), [attention](<https://devfeed.tech/tags/attention.md>), [backend](<https://devfeed.tech/tags/backend.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [batch](<https://devfeed.tech/tags/batch.md>), [cache](<https://devfeed.tech/tags/cache.md>), [communication](<https://devfeed.tech/tags/communication.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [data](<https://devfeed.tech/tags/data.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [graph](<https://devfeed.tech/tags/graph.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [latency](<https://devfeed.tech/tags/latency.md>), [load](<https://devfeed.tech/tags/load.md>), [load-balancing](<https://devfeed.tech/tags/load-balancing.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [variable](<https://devfeed.tech/tags/variable.md>)

## AI overview

NanoCP addresses load imbalance in MoE decoding by separating MoE communication from KV-cache placement and dynamically assigning each request a context-parallel degree based on its KV footprint. Long requests spread attention across multiple instances, while short requests stay local, balancing cache occupancy and batch sizes within each node. An ahead-of-time graph engine and custom routing-based communication backend connect this dynamic scheme to static execution. On the reported DeepSeek-V3 Issue 1% and Issue 5% workloads, at 99% TPOT-SLO attainment, NanoCP achieved 3.27x and 2.11x the maximum sustainable request rate of the best-performing vLLM baseline, and reduced P99 TPOT by 1.94x and 2.01x, respectively.

## Source excerpt

arXiv:2605.21100v2 Announce Type: replace Abstract: Modern serving systems for Mixture-of-Experts (MoE) models adopt hybrid data-expert parallelism: expert parallelism (EP) shards experts across GPUs to scale capacity, while data parallelism (DP) replicates attention layers across instances to process independent requests. Existing systems bind each request's attention, MoE communication, and KV cache to a single instance. Because attention latency scales with KV cache size while MoE communication latency scales with batch size, this binding cannot balance both simultaneously, producing EP stragglers; it also fragments KV memory across instances, inflating tail latency under long contexts. While existing context parallelism (CP) mitigates these constraints, existing group-level and uniform CP designs incur prohibitive communication and attention-side overheads. We present NanoCP, which decouples MoE communication from KV cache placement and achieves dual balance through dynamic context parallelism (DCP). DCP assigns each request a context-parallel degree sized to its KV footprint: long requests distribute attention across multiple instances; short requests remain local. This dynamic parallelism effectively liquefies the KV cache across DP instances within each node, balancing both the per-instance KV cache occupancy and batch sizes without unnecessary load-balancing costs. To bridge DCP with static execution, NanoCP introduces an ahead-of-time (AOT) graph engine paired with a custom routing-based communication backend. Experimental results show that, under 99% TPOT-SLO attainment, NanoCP achieves $3.27\times$ and $2.11\times$ the maximum sustainable request rate of the best-performing vLLM baseline on the DeepSeek-V3 Issue 1% and Issue 5% workloads, respectively, while reducing P99 TPOT by $1.94\times$ and $2.01\times$ on the same two workloads, respectively.