# ZeRO-Prefill: Zero Redundancy Overheads in MoE Prefill Serving

DevFeed: [ZeRO-Prefill: Zero Redundancy Overheads in MoE Prefill Serving](<https://devfeed.tech/articles/zero-prefill-zero-redundancy-overheads-in-moe-prefill-serving-103768.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.02960>)

Author: Zhaoyuan Su, Olatunji Ruwase, Karthik Ganesan, Aurick Qiao, Samyam Rajbhandari, Juncheng Yang, Yue Cheng, Yuxiong He

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [GPU](<https://devfeed.tech/topics/gpu.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [classification](<https://devfeed.tech/tags/classification.md>), [communication](<https://devfeed.tech/tags/communication.md>), [compute](<https://devfeed.tech/tags/compute.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [execution](<https://devfeed.tech/tags/execution.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [frontend](<https://devfeed.tech/tags/frontend.md>), [layer](<https://devfeed.tech/tags/layer.md>), [llm](<https://devfeed.tech/tags/llm.md>), [load](<https://devfeed.tech/tags/load.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [moe](<https://devfeed.tech/tags/moe.md>), [zero](<https://devfeed.tech/tags/zero.md>)

## AI overview

ZeRO-Prefill targets MoE workloads answered from a single prefill pass, where distributed execution can add communication and synchronization overhead. Its AsyncEP backend gathers expert weights asynchronously, overlapping weight AllGather with computation in place of per-layer activation AllToAll. A co-designed frontend uses prefix-aware routing and true-FLOPs load tracking to enforce a saturation threshold. On Qwen3-235B-A22B across four hardware and precision configurations, the abstract reports 1.34-1.37x the throughput of the strongest distributed baseline on real-world workloads and up to 1.59x on long-context synthetic workloads, with 29.8-36.2% per-GPU model FLOPs utilization.

## Source excerpt

arXiv:2605.02960v3 Announce Type: replace Abstract: Production LLM workloads increasingly serve discriminative tasks, such as classification, recommendation, and verification, whose answers are read from the logits of a single prefill pass with no autoregressive decoding. Serving these prefill-only workloads on mixture-of-experts (MoE) models is bottlenecked not by compute but by the distributed execution required to fit the model: existing parallel strategies (tensor, expert, and pipeline parallelism) trade memory pressure for redundant computation, communication, and synchronization, severely degrading MoE prefill serving efficiency. We observe that these overheads stem from coupling expert placement with synchronous activation routing -- a design inherited from the decoding era. The long, compute-bound forward passes of large-batch prefill open a per-layer window wide enough to stream expert weights in the background, replacing per-layer activation AllToAll with asynchronous weight AllGather fully overlapped with computation. We propose ZeRO-Prefill, a prefill-only serving system built on this idea. Its backend, AsyncEP (Asynchronous Expert Parallelism), gathers experts by weight rather than by activation; its co-designed frontend enforces a physically derived saturation threshold through prefix-aware routing and true-FLOPs load tracking. On Qwen3-235B-A22B across four hardware/precision configurations, ZeRO-Prefill delivers 1.34-1.37x throughput over the strongest distributed baseline on real-world workloads and up to 1.59x on long-context synthetic workloads, sustaining 29.8-36.2% per-GPU model FLOPs utilization.