# expert-parallelism

Published articles for expert-parallelism.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Harvesting Expert-Parallel Bubbles for Fine-Tuning

DevFeed: [Harvesting Expert-Parallel Bubbles for Fine-Tuning](<https://devfeed.tech/articles/harvesting-expert-parallel-bubbles-for-fine-tuning-142803.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24496>)

Author: Oana Balmau, Saurav Banka, Jiaxuan Chen, Anne-Marie Kermarrec, Ramya Narasimha Prabhu, Rafael Pires, Mathis Randl, Martijn de Vos

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [a100](<https://devfeed.tech/tags/a100.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [bubble](<https://devfeed.tech/tags/bubble.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [colocation](<https://devfeed.tech/tags/colocation.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-os](<https://devfeed.tech/tags/cs-os.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [engine](<https://devfeed.tech/tags/engine.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [expert-parallel](<https://devfeed.tech/tags/expert-parallel.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>)

### AI overview

BubbleTea uses per-GPU idle periods created by uneven expert-parallel token routing to run fine-tuning alongside LLM serving. It predicts each GPU's available bubble at every layer and schedules short fine-tuning chunks that reuse serving weights; compiler passes derive the chunks from the served model without porting its architecture to a separate training engine. On three MoE models served across four A100 GPUs, BubbleTea delivered 1.9-6.2x the fine-tuning throughput of LLMStation and kept p90 inference latency within 10% of serving alone in 50 of 63 workloads, versus 44 for LLMStation. On bursty production traces, it trained 3.3x faster than LLMStation and 14x faster than a stock trainer under CUDA MPS, with p90 latency within 7% of serving alone.

### Source excerpt

arXiv:2611.24496v1 Announce Type: cross Abstract: GPUs are expensive, yet latency-critical large language model (LLM) inference leaves part of them idle. Colocation reclaims this capacity by running best-effort work, such as fine-tuning, alongside serving. Many leading LLMs now use a Mixture-of-Experts (MoE) architecture, which adds a new source of idle time. Under expert parallelism, MoE experts are spread across GPUs, and because the router sends uneven numbers of tokens to each GPU, lightly loaded GPUs wait for the most loaded one. Even after applying load balancing solutions, such bubbles still take 8--13% of prefill time. We present BubbleTea, the first colocation system that leverages expert-parallel bubbles for fine-tuning. Unlike existing colocation systems, which make one scheduling decision for all GPUs per forward pass, BubbleTea predicts each GPU's bubble at every layer from the router's token assignment, and fills it with short chunks of fine-tuning work that reuse the weights already loaded for serving. We also introduce compiler passes that automatically derive these chunks from the served LLM as-is, without porting each model architecture to a separate training engine. Our evaluation on three MoE models, served on four A100 GPUs, shows that fine-tuning throughput with BubbleTea is 1.9--6.2$\times$ faster than with LLMStation, a state-of-the-art colocation system. BubbleTea keeps p90 inference latency within 10% of serving alone in 50 of 63 workloads, compared to 44 for LLMStation. On bursty production traces, BubbleTea trains 3.3$\times$ faster than LLMStation and 14$\times$ faster than a stock trainer colocated under CUDA MPS, with p90 inference latency staying within 7% of serving alone.

## Communicate the Change: Efficient Expert Parallelism via Activation Compression

DevFeed: [Communicate the Change: Efficient Expert Parallelism via Activation Compression](<https://devfeed.tech/articles/communicate-the-change-efficient-expert-parallelism-via-activation-compression-139983.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22215>)

Author: Fengyuan Liu, Jay Gala, Siva Reddy, Dzmitry Bahdanau

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [quantization](<https://devfeed.tech/topics/quantization.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [collective](<https://devfeed.tech/tags/collective.md>), [communication](<https://devfeed.tech/tags/communication.md>), [compression](<https://devfeed.tech/tags/compression.md>), [compute](<https://devfeed.tech/tags/compute.md>), [critical](<https://devfeed.tech/tags/critical.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-inference](<https://devfeed.tech/tags/distributed-inference.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [ethernet](<https://devfeed.tech/tags/ethernet.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [fewer](<https://devfeed.tech/tags/fewer.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [hidden-states](<https://devfeed.tech/tags/hidden-states.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>)

### AI overview

EchoMoE reduces communication overhead in expert-parallel MoE inference by sending a compressed difference between the current hidden state and a periodically refreshed earlier-layer reference, rather than sending each hidden state in full. The abstract reports 58% lower inter-node communication volume and 22.6% higher throughput, with average performance within 3.8 percentage points across 18 benchmarks. On bandwidth-constrained Ethernet, throughput improves by up to 2.38x. The modification is designed to retrofit pretrained MoE models, requiring about 8,500x fewer training tokens than full pretraining; the communicated representations can also be quantized without measurable quality loss. The authors report an implementation for pretrained models and an inference integration for vLLM.

### Source excerpt

arXiv:2611.22215v1 Announce Type: cross Abstract: Scaling has driven the success of large language models (LLM), with sparse mixture-of-experts (MoE) architectures enabling greater parameter capacity without a proportional increase in computation. At large scales, MoE models rely on expert parallelism to distribute experts across devices. Each MoE layer requires two synchronous collective communication steps during inference: dispatching hidden states to the activated experts and combining the resulting expert outputs. As compute throughput has outpaced interconnect bandwidth, these synchronous collectives have become a major bottleneck to MoE scaling and efficiency. We address this challenge with EchoMoE, a lightweight architecture modification that retrofits existing MoE models, requiring approximately 8,500x fewer training tokens than full pretraining. Motivated by the observation that transformer blocks update the residual stream incrementally, we use a hidden state from an earlier layer as a reference and communicate only the compressed difference between it and the current state, rather than communicating each hidden state in full. EchoMoE periodically refreshes these reference states and transmits only the compressed differences synchronously, reducing the traffic on latency-critical paths without significant computational overhead. EchoMoE reduces inter-node communication volume by 58% and improves throughput by 22.6%, while preserving average performance within 3.8 percentage points across 18 benchmarks. EchoMoE further improves inference throughput by up to 2.38x over bandwidth-constrained Ethernet, paving the way for wide-area distributed inference. EchoMoE also tolerates quantization of the communicated representations without measurable quality loss. We release an efficient implementation for applying EchoMoE to pretrained MoE models, together with an inference integration for vLLM.

## Supercharging Olmo-core for Efficient and Scalable MoE Training

DevFeed: [Supercharging Olmo-core for Efficient and Scalable MoE Training](<https://devfeed.tech/articles/supercharging-olmo-core-for-efficient-and-scalable-moe-training-136978.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18956>)

Author: Tianhua Tao, Akshita Bhagia, Dirk Groeneveld, Pete Walsh, Tyler Romero, Yashas Samaga, Jacob Morrison, Iz Beltagy, Taira Anderson, Noah A. Smith, Hannaneh Hajishirzi

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [b300](<https://devfeed.tech/tags/b300.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [ddp](<https://devfeed.tech/tags/ddp.md>), [deepep](<https://devfeed.tech/tags/deepep.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [fsdp](<https://devfeed.tech/tags/fsdp.md>), [gerrymandering](<https://devfeed.tech/tags/gerrymandering.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [load-balancing](<https://devfeed.tech/tags/load-balancing.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [moe](<https://devfeed.tech/tags/moe.md>), [nvl8](<https://devfeed.tech/tags/nvl8.md>), [nvshmem](<https://devfeed.tech/tags/nvshmem.md>), [pipeline-parallelism](<https://devfeed.tech/tags/pipeline-parallelism.md>), [router](<https://devfeed.tech/tags/router.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

Olmo-core's redesigned MoE training stack targets costs that grow with stored expert parameters even when per-token compute stays nearly constant. It replaces earlier FSDP with DDP plus expert and pipeline parallelism, a distributed optimizer, and activation recompute. NVSHMEM-based rowwise expert parallelism writes tokens directly to expert buffers; GPU-resident routing metadata and device-scheduled grouped GEMMs remove synchronization. Integrated MXFP8 reduces compute, activation-memory, and transport costs, while topology-agnostic checkpoints allow parallel layouts to change between runs. On NVL8 B300 nodes, configurations span 12.9 billion to 1.2 trillion parameters across up to 512 GPUs; the 1.2-trillion configuration reaches 858 useful-model TFLOP/s/GPU with MXFP8 and per-layer recompute. An experimental test with optional DeepEP v2 reaches 2.38 trillion parameters. The authors also report router exploitation of the load-balancing loss ("Token Gerrymandering") and warn that communication overlap can slow kernels more than it hides.

### Source excerpt

arXiv:2611.18956v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models add parameters without proportionally increasing per-token compute. However, training cost also depends on bytes moved, memory held, and kernels launched, and many of those costs follow total parameter size rather than active compute. On infrastructure built for dense models, this mismatch is costly: in early baselines, throughput fell as stored expert parameter size grew, even though the useful work per token stayed nearly fixed. To close this gap, we present a redesigned training stack in Olmo-core, built for efficient, scalable MoE training. We replace the Fully Sharded Data Parallel (FSDP) of earlier Olmo-core versions with an architecture built on Distributed Data Parallel (DDP), and add Expert Parallelism (EP), Pipeline Parallelism (PP), a distributed optimizer, and activation recompute. We propose NVSHMEM-based rowwise EP, which writes each token directly into its expert's buffer, avoiding dispatch and combine permutation overhead. We make EP fully synchronization-free by keeping routing metadata on the GPU and using device-scheduled grouped GEMMs. Closely integrated MXFP8 cuts compute, activation-memory, and transport costs, and topology-agnostic checkpoints let the parallel layout change between runs. On NVL8 B300 nodes, the stack supports training configurations from 12.9 billion to 1.2 trillion total parameters on up to 512 GPUs, reaching 858 useful-model TFLOP/s/GPU at 1.2 trillion with MXFP8 and per-layer recompute; an experimental capacity test with the optional DeepEP v2 backend reaches 2.38 trillion. We also explain each design choice and share findings for other training systems; for example, we identify a failure pattern we call Token Gerrymandering, in which the MoE router can learn to hack the load-balancing loss, and find that overlapping communication with computation can slow the overlapped kernels by more than it hides.

## One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink

DevFeed: [One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink](<https://devfeed.tech/articles/one-sided-expert-parallel-all-to-all-for-mixture-of-experts-inference-over-nvlink-127800.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13188>)

Author: Bo Li, Chulian Zhang, Or Biri, Anthony Chang, Anton Korzh, Noam Duvdevani, XingFei Xi, Daniel Stokes, Zhenhuan Chen, Ludwig Schneider, Artem Y. Polyakov, Julien Demouth, Chandler Zhou

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [backend](<https://devfeed.tech/tags/backend.md>), [batch](<https://devfeed.tech/tags/batch.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [communication](<https://devfeed.tech/tags/communication.md>), [communications](<https://devfeed.tech/tags/communications.md>), [compute](<https://devfeed.tech/tags/compute.md>), [copy](<https://devfeed.tech/tags/copy.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [deepep](<https://devfeed.tech/tags/deepep.md>), [deepgemm](<https://devfeed.tech/tags/deepgemm.md>), [dispatch](<https://devfeed.tech/tags/dispatch.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [frameworks](<https://devfeed.tech/tags/frameworks.md>), [inference](<https://devfeed.tech/tags/inference.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [nvlink](<https://devfeed.tech/tags/nvlink.md>)

### AI overview

NVLinkOneSided reduces fixed communication overhead in small-batch MoE inference by having each source GPU push tokens directly into rank-major buffers on target GPUs, without a prior metadata exchange; grouped GEMMs read the data in place rather than from per-expert copies. Its Fence protocol targets high throughput, while Sentinel targets low latency. On Blackwell GPUs, dispatch and combine are 1.2-3.0x faster than DeepEP, with the largest gains at small batch sizes; the resulting MoE layer is up to 2.4x faster than DeepEP with DeepGEMM. It also outperforms fused DeepGEMM MegaMoE at up to 128 tokens per GPU. The method is limited to GPUs within one NVLink domain, and its modular interface is described as independent of activation quantization format and MoE compute backend.

### Source excerpt

arXiv:2611.13188v1 Announce Type: new Abstract: Expert parallelism (EP) shards the experts of a Mixture-of-Experts (MoE) layer across GPUs, so every MoE layer requires two all-to-all communications, namely dispatch and combine. Compared with training, inference processes relatively few tokens per GPU at a time, and at small batch sizes fixed costs that do not grow with the number of tokens, rather than data movement, dominate the latency. We present NVLinkOneSided, a one-sided EP all-to-all for GPUs within a single NVLink domain. Each source rank pushes its tokens, without first exchanging metadata, into rank-major buffers on the target GPUs, and the grouped GEMMs read them in place instead of from a per-expert copy. We provide two protocols for the synchronization that the communication requires, namely Fence and Sentinel, for high-throughput and low-latency scenarios respectively. On Blackwell GPUs, NVLinkOneSided completes dispatch and combine 1.2-3.0$\times$ faster than DeepEP, with the largest speedups at small batch sizes. The resulting MoE layer is up to 2.4$\times$ faster than DeepEP with DeepGEMM. It even outperforms the fused DeepGEMM MegaMoE at up to 128 tokens per GPU, making it still competitive for decoding. We expose NVLinkOneSided through a modular interface, so that it is agnostic to the activation quantization format, decoupled from the MoE compute backend and easy to integrate into inference frameworks.

## OverlapAdvisor: Modeling and Adapting Computation-Communication Overlap for Distributed MoE Serving

DevFeed: [OverlapAdvisor: Modeling and Adapting Computation-Communication Overlap for Distributed MoE Serving](<https://devfeed.tech/articles/overlapadvisor-modeling-and-adapting-computation-communication-overlap-for-distributed-moe-serving-126733.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11857>)

Author: Chutong Ding, Kaixuan Zhang, Luping Wang, Shiyou Qian, Jian Cao, Cheng Huang, Guodong Yang, Liping Zhang

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [sglang](<https://devfeed.tech/topics/sglang.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [GPU](<https://devfeed.tech/topics/gpu.md>)

Tags: [adapting](<https://devfeed.tech/tags/adapting.md>), [communication](<https://devfeed.tech/tags/communication.md>), [compute](<https://devfeed.tech/tags/compute.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [deepep](<https://devfeed.tech/tags/deepep.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [engines](<https://devfeed.tech/tags/engines.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [inference](<https://devfeed.tech/tags/inference.md>), [input](<https://devfeed.tech/tags/input.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [moe](<https://devfeed.tech/tags/moe.md>), [overlap](<https://devfeed.tech/tags/overlap.md>), [serving](<https://devfeed.tech/tags/serving.md>)

### AI overview

OverlapAdvisor selects computation-communication overlap strategies at launch for distributed MoE serving, because no single strategy performed best across more than 80.6% of 103 production configurations spanning input lengths and four hardware platforms. Its unified performance model represents overlap dependencies at two granularities and accounts for workload- and hardware-specific deviations from peak compute, memory, and communication rates. Evaluated on four MoE models (30B-235B) against three baselines in SGLang and vLLM, it achieved 1.16x average and 1.59x maximum kernel speedups, with 3.3%-15.9% prediction MAPE, and up to 31.7% higher end-to-end throughput. Granularity-guided optimizations also yielded median speedups of 1.10x for DeepEP-v1 and 1.22x for FlashMoE.

### Source excerpt

arXiv:2611.11857v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) inference with expert parallelism (EP) relies on All-to-All communication, making computation-communication overlap strategies critical for achieving high performance. However, no strategy is universally optimal. Across 103 production configurations, varying input lengths, and four hardware platforms, we observe that the fastest strategy shifts based on model shape, EP degree, input length, and hardware. Since no strategy dominates more than 80.6% of configurations, configuration-aware selection is necessary. In this work, we propose OverlapAdvisor, which models overlap stages using two granularities to capture computation-communication dependencies. OverlapAdvisor builds a unified performance model, accounting for deviations from peak compute, memory, and communication performance across workloads and hardware. This model enables launch-time, configuration-aware strategy selection within two serving engines. We conduct experiments on four MoE models (30B-235B) against three baselines, utilizing SGLang and vLLM. At the kernel level, OverlapAdvisor achieves 1.16x average and 1.59x maximum speedups, with a prediction MAPE of 3.3%-15.9%. In SGLang and vLLM, it improves end-to-end throughput up to 31.7%. Its granularity-guided optimizations further achieve median speedups of 1.10x for DeepEP-v1 and 1.22x for FlashMoE.

## MoE Routing Strategies Explained: Top-K, Expert-Choice, Sigmoid Gating and Dynamic Routing (2026 Guide)

DevFeed: [MoE Routing Strategies Explained: Top-K, Expert-Choice, Sigmoid Gating and Dynamic Routing (2026 Guide)](<https://devfeed.tech/articles/moe-routing-strategies-explained-top-k-expert-choice-sigmoid-gating-and-dynamic-routing-2026-guide-122489.md>)

Original publisher: [Read original article](<https://www.marktechpost.com/2026/11/07/moe-routing-strategies-explained/>)

Author: Asif Razzaq

Published: 2026-11-08T06:18:33Z

Content type: article

Language: en

Sources: [MarkTechPost](<https://devfeed.tech/sources/marktechpost.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [applications](<https://devfeed.tech/tags/applications.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [balance](<https://devfeed.tech/tags/balance.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [bias](<https://devfeed.tech/tags/bias.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [causality](<https://devfeed.tech/tags/causality.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [deepseek-v3](<https://devfeed.tech/tags/deepseek-v3.md>), [editors-pick](<https://devfeed.tech/tags/editors-pick.md>), [enterprise-ai](<https://devfeed.tech/tags/enterprise-ai.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [faster](<https://devfeed.tech/tags/faster.md>), [for-devs](<https://devfeed.tech/tags/for-devs.md>), [guide](<https://devfeed.tech/tags/guide.md>), [hash](<https://devfeed.tech/tags/hash.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [mixtral](<https://devfeed.tech/tags/mixtral.md>), [models](<https://devfeed.tech/tags/models.md>), [moe](<https://devfeed.tech/tags/moe.md>), [moe-routing-strategies](<https://devfeed.tech/tags/moe-routing-strategies.md>), [physical-ai](<https://devfeed.tech/tags/physical-ai.md>), [routing](<https://devfeed.tech/tags/routing.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [staff](<https://devfeed.tech/tags/staff.md>), [tech-news](<https://devfeed.tech/tags/tech-news.md>), [technology](<https://devfeed.tech/tags/technology.md>), [token](<https://devfeed.tech/tags/token.md>), [top-k](<https://devfeed.tech/tags/top-k.md>)

### AI overview

MoE routers select a small set of experts for each token, affecting model computation and, in expert-parallel serving, all-to-all network traffic. The excerpt describes softmax top-k and top-1 token choice, shared plus routed experts, sigmoid gating with bias-based load balancing, node-limited routing, expert choice, hash routing, and dynamic top-k. It reports that expert choice gives experts fixed token buckets but can leak future-token information in causal models; one cited test found an abnormal loss drop of about 10%. The article calls sigmoid scoring with aux-loss-free bias balancing the prevailing recipe for frontier open MoE models and reports a MaxVio reduction from 0.72 to 0.04 at 1B with lower perplexity. For serving an existing checkpoint, it recommends checking expert count, experts per token, shared experts, and `topk_method` in `config.json`; changing top-k at inference changes the model and should be treated as a quality experiment. The supplied text is an excerpt and does not include the comparison table.

### Source excerpt

MoE routing strategies decide which experts process each token. This guide explains 7 methods, from Mixtral's top-2 softmax to DeepSeek-V4's hash routing, with the models that use each one. The post MoE Routing Strategies Explained: Top-K, Expert-Choice, Sigmoid Gating and Dynamic Routing (2026 Guide) appeared first on MarkTechPost.

## FastAFD: When Does Attention-FFN Disaggregation Pay on Rack-Scale NVLink?

DevFeed: [FastAFD: When Does Attention-FFN Disaggregation Pay on Rack-Scale NVLink?](<https://devfeed.tech/articles/fastafd-when-does-attention-ffn-disaggregation-pay-on-rack-scale-nvlink-120564.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07149>)

Author: Yichao Fu, Yuxuan Zhang, Minshen Zhang, Ruitian Wang, Junda Chen, Hao Zhang

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [GB200](<https://devfeed.tech/topics/gb200.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>), [groq](<https://devfeed.tech/topics/groq.md>)

Tags: [accelerators](<https://devfeed.tech/tags/accelerators.md>), [attention](<https://devfeed.tech/tags/attention.md>), [batch](<https://devfeed.tech/tags/batch.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decode](<https://devfeed.tech/tags/decode.md>), [disaggregation](<https://devfeed.tech/tags/disaggregation.md>), [engine](<https://devfeed.tech/tags/engine.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [gb200](<https://devfeed.tech/tags/gb200.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [groq](<https://devfeed.tech/tags/groq.md>), [lpx](<https://devfeed.tech/tags/lpx.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [moe](<https://devfeed.tech/tags/moe.md>), [nvl72](<https://devfeed.tech/tags/nvl72.md>), [nvlink](<https://devfeed.tech/tags/nvlink.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [rack-scale](<https://devfeed.tech/tags/rack-scale.md>)

### AI overview

FastAFD separates attention from MoE experts on GB200 NVL72, overlapping one microbatch's MoE work with another's attention, and compares against vLLM's best wide expert-parallel setting. It reports up to 1.17x throughput on Qwen3, 1.16x on Qwen3.5, and parity on MiniMax-M2.5. Its model predicts AFD throughput within 3% from colocated vLLM profiles. The model attributes gains to larger batches and overlap, weighed against the GPUs allocated to experts; AFD pays off when the gains exceed that cost. Speculative decoding particularly helps light-expert models, adding 8% to AFD's advantage over wide EP. The abstract projects an ideal 2.48x speedup with speculative decoding if experts use on-chip SRAM accelerators, which would remove most disaggregation cost.

### Source excerpt

arXiv:2611.07149v1 Announce Type: cross Abstract: Attention-FFN disaggregation (AFD) separates attention from the experts of a mixture-of-experts (MoE) model and has been reported to raise decode throughput over colocated serving. Whether this advantage holds on rack-scale NVLink systems such as GB200 NVL72 remains unclear: prior results were obtained on other hardware, against narrower colocated baselines, or with analytical models, whereas production engines serve large MoE models on these systems with wide expert parallelism (wide EP). We build and tune FastAFD, an AFD engine for GB200 NVL72 that overlaps the MoE of one microbatch with the attention of the other, and compare it against the best vLLM wide EP setting. Experiments show that AFD has a throughput advantage of up to 1.17x over wide EP on Qwen3 and 1.16x on Qwen3.5, and matches it on MiniMax-M2.5. We derive an analytical model that predicts AFD throughput within 3% using only profiles of colocated vLLM, telling whether switching to AFD would pay off before deploying it. The model yields a concise insight: AFD's speedup factors into a batch gain (larger batches, since attention GPUs hold no expert weights), an overlap gain (shorter steps, since the MoE runs on the expert GPUs behind attention), and a disaggregation cost (the GPUs given to experts). AFD has the advantage when the two gains outweigh the cost. This view explains which models benefit from AFD, and shows that speculative decoding helps light-expert models most, where the enlarged overlap gain outweighs the larger disaggregation cost and adds a further 8% to AFD's advantage over wide EP. Looking ahead, serving the experts on accelerators that keep their weights in on-chip SRAM, such as the Groq LPX accelerators paired with Vera Rubin NVL72, would remove most of the disaggregation cost, raising the ideal speedup to 2.48x with speculative decoding. FastAFD is open source at https://github.com/hao-ai-lab/FastAFD.

## KOSMOS: Self-scheduling Persistent Kernels for Communication-Compute Overlap

DevFeed: [KOSMOS: Self-scheduling Persistent Kernels for Communication-Compute Overlap](<https://devfeed.tech/articles/kosmos-self-scheduling-persistent-kernels-for-communication-compute-overlap-120197.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07477>)

Author: Alexander Magro, Aristotle Martin, Ye Wang, Wen Chen

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [kernels](<https://devfeed.tech/topics/kernels.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>)

Tags: [amd](<https://devfeed.tech/tags/amd.md>), [amd-instinct-mi355x](<https://devfeed.tech/tags/amd-instinct-mi355x.md>), [collective](<https://devfeed.tech/tags/collective.md>), [communication](<https://devfeed.tech/tags/communication.md>), [compute](<https://devfeed.tech/tags/compute.md>), [consumer](<https://devfeed.tech/tags/consumer.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [data](<https://devfeed.tech/tags/data.md>), [data-readiness](<https://devfeed.tech/tags/data-readiness.md>), [deepep](<https://devfeed.tech/tags/deepep.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [deepseek-v3](<https://devfeed.tech/tags/deepseek-v3.md>), [demand](<https://devfeed.tech/tags/demand.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [instinct](<https://devfeed.tech/tags/instinct.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [launch](<https://devfeed.tech/tags/launch.md>), [layer](<https://devfeed.tech/tags/layer.md>), [overlap](<https://devfeed.tech/tags/overlap.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>)

### AI overview

KOSMOS overlaps communication and computation in transformer parallelism by using persistent kernel workgroups that switch roles as data become ready, following queues ordered by consumer demand. This avoids a scheduler, per-call cost model, or tuned communication-computation split. It fuses each TP+SP collective with its GEMM and runs an entire expert layer, from router to fold, in one launch per direction. On AMD Instinct MI350X and MI355X, it reports 1.10-1.28x faster TP+SP end-to-end Megatron-LM training in BF16 and 1.10-1.25x in MXFP8. For DeepSeek-V3 and Qwen3-235B proxies, it reports 1.29-1.83x faster end-to-end training than DeepEP.

### Source excerpt

arXiv:2611.07477v1 Announce Type: cross Abstract: Tensor and sequence parallelism (TP+SP) and expert parallelism (EP) put collectives on the critical path of transformer layers. State-of-the-art fused kernels mitigate this with fixed roles or schedules, but this rigidity leaves large idle windows. Predicting a communication-computation split does not remove them either; a predetermined split cannot follow data readiness. We present KOSMOS, a library for self-scheduling persistent kernels whose workgroups use queues ordered by consumer demand to switch between communication and computation roles as data become ready, with no scheduler, per-call cost model, or tuned split. KOSMOS fuses every TP+SP collective of a transformer layer with its GEMM, and runs an entire expert layer, from router to fold, in one launch per direction. Evaluated on AMD Instinct MI350X and AMD Instinct MI355X GPUs, KOSMOS speeds up TP+SP end-to-end training in Megatron-LM by 1.10-1.28x in BF16 and 1.10-1.25x in MXFP8. KOSMOS accelerates end-to-end training of DeepSeek-V3 and Qwen3-235B proxies by 1.29-1.83x over DeepEP.

## Resource-Efficient Reinforcement Learning Post-Training through Elastic Expert Parallelism

DevFeed: [Resource-Efficient Reinforcement Learning Post-Training through Elastic Expert Parallelism](<https://devfeed.tech/articles/resource-efficient-reinforcement-learning-post-training-through-elastic-expert-parallelism-120145.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.06984>)

Author: Jia Wei, Ziyi Qiu, Yi Xu, Xuehai Qian

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [post-training](<https://devfeed.tech/topics/post-training.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [batching](<https://devfeed.tech/tags/batching.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [mixture-of-experts-models](<https://devfeed.tech/tags/mixture-of-experts-models.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>), [throughput](<https://devfeed.tech/tags/throughput.md>)

### AI overview

In mixture-of-experts reinforcement learning post-training, workers can remain occupied after finishing response generation because other workers still need their experts; the abstract estimates this accounts for about one-third of occupied worker time in the baseline. FlexFloor jointly adapts response scheduling and the active worker group, using historical response lengths and memory constraints to guide batching and prompt placement, then progressively releases workers while preserving needed expert computation and communication. Compared with the verl baseline, it reports 65.40% more response tokens per unit of occupied computing time and 52.74% higher response generation throughput, with little observed variation in rollout reward.

### Source excerpt

arXiv:2611.06984v1 Announce Type: new Abstract: Reinforcement learning post-training improves the capabilities of large language models, but generating training responses is a major computational bottleneck. In mixture-of-experts models, workers that finish their assigned responses may remain occupied because their experts are still needed by workers processing longer responses. Consequently, approximately one-third of occupied worker time in the baseline occurs after local response generation finishes. Here we present FlexFloor, a system that enables earlier resource release during synchronous response generation while preserving the expert computation required by ongoing responses. FlexFloor jointly adapts response scheduling and the active group of workers. It uses historical response lengths and memory constraints to guide batching and prompt placement, then progressively releases workers while maintaining expert availability and correct communication. Compared with the verl baseline, FlexFloor generates approximately 65.40% more response tokens per unit of occupied computing time and increases response generation throughput by 52.74%, with little observed variation in rollout reward. By making reinforcement learning post-training more efficient, FlexFloor supports faster development and iterative improvement of large language models and AI agents.

## GC-MoE: Overlapping Expert-Parallel Token Dispatch with FP8/FP4 Grouped GEMM via CUDA Green Contexts

DevFeed: [GC-MoE: Overlapping Expert-Parallel Token Dispatch with FP8/FP4 Grouped GEMM via CUDA Green Contexts](<https://devfeed.tech/articles/gc-moe-overlapping-expert-parallel-token-dispatch-with-fp8-fp4-grouped-gemm-via-cuda-green-contexts-116485.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.04222>)

Author: Yong Li, Kan Liu, Tao Lan

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [CUDA](<https://devfeed.tech/topics/cuda.md>), [CUDA Graphs](<https://devfeed.tech/topics/cuda-graphs.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [synchronization](<https://devfeed.tech/topics/synchronization.md>)

Tags: [blackwell](<https://devfeed.tech/tags/blackwell.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [decode](<https://devfeed.tech/tags/decode.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [deepseek-v4-1-flash](<https://devfeed.tech/tags/deepseek-v4-1-flash.md>), [dispatch](<https://devfeed.tech/tags/dispatch.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [glm](<https://devfeed.tech/tags/glm.md>), [glm-5-2](<https://devfeed.tech/tags/glm-5-2.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [graph](<https://devfeed.tech/tags/graph.md>), [handshake](<https://devfeed.tech/tags/handshake.md>), [hopper](<https://devfeed.tech/tags/hopper.md>), [inference](<https://devfeed.tech/tags/inference.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [open-source](<https://devfeed.tech/tags/open-source.md>)

### AI overview

GC-MoE overlaps expert-parallel token dispatch and grouped GEMM by using CUDA Green Contexts to assign communication and compute to separate GPU partitions, each with its own kernel resources and synchronization. A concurrent CUDA Graph launch, token-ring handshake, and double-buffered receive counters enable early computation and avoid a trailing cross-rank barrier between replays. For computation, it uses operand-swapped block-scaled MMA on Blackwell and register-A WGMMA with in-register FP4 decoding on Hopper; fused SwiGLU, requantization, and optional shared experts are supported. In the reported tests, it beat Mega MoE at all 54 matched-accuracy points: average decode speedups were 1.188x on 8 H20 GPUs for DeepSeek-V4.1-Flash, and 1.106x on 8 B300 GPUs for GLM-5.2 and DeepSeek-V4.1-Flash (1.123x with fused shared experts). Dispatch used 20 of 148 B300 SMs.

### Source excerpt

arXiv:2611.04222v1 Announce Type: new Abstract: Under expert parallelism, mixture-of-experts (MoE) layers pay all-to-all token dispatch, grouped GEMMs, and a combine step in every layer. In decode each rank holds only 1-256 tokens: dispatch latency and weight traffic dominate while tensor-core utilization is inherently low. In fused operators such as DeepGEMM's Mega MoE, dispatch and computation overlap only by sharing every SM: dispatch warps compete with TMA/MMA and epilogue warps for registers, shared memory, and issue slots; every phase shares one CTA resource budget, synchronization spans the whole SM array, and every replay ends with a workspace-clean pass and a cross-rank barrier. We present GC-MoE, an open-source FP8/FP4 MoE inference operator for NVIDIA Hopper and Blackwell GPUs that uses CUDA Green Contexts to split the device into a small communication partition and a large compute partition. Each role gets its own kernel, resource budget, and synchronization domain, so dispatch and GEMM tune independently. A three-node CUDA Graph (four on Hopper with fused shared experts) launches the dispatch and grouped-GEMM kernels as concurrent roots; a token-ring handshake starts computation on the first filled ring blocks, and double-buffered receive counters remove the trailing cross-rank barrier between replays. The compute partition runs operand-swapped tcgen05 block-scaled MMA on Blackwell and register-A WGMMA with in-register FP4 decoding on Hopper, with fused SwiGLU, requantization, and optional shared experts. On 8 H20 GPUs GC-MoE accelerates DeepSeek-V4.1-Flash decode by 1.188x on average on both the routed and shared-expert suites (peaks 1.296x and 1.353x); on 8 B300 GPUs it accelerates GLM-5.2 and DeepSeek-V4.1-Flash by 1.106x (1.123x with fused shared experts, peak 1.209x), beating Mega MoE at all 54 measured points at matched accuracy. GC-MoE reserves only 20 of 148 SMs on B300 for dispatch.

## RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts

DevFeed: [RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts](<https://devfeed.tech/articles/ramp-runtime-aware-megakernel-polymorphism-for-mixture-of-experts-113754.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2604.26039>)

Author: Vyom Sharma, Debajyoti Datta

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [backend](<https://devfeed.tech/tags/backend.md>), [batch](<https://devfeed.tech/tags/batch.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [moe](<https://devfeed.tech/tags/moe.md>), [offline](<https://devfeed.tech/tags/offline.md>), [polymorphism](<https://devfeed.tech/tags/polymorphism.md>), [ramp](<https://devfeed.tech/tags/ramp.md>), [runtime](<https://devfeed.tech/tags/runtime.md>)

### AI overview

RaMP addresses a limitation of offline GPU-kernel autotuning for Mixture-of-Experts: the best configuration depends on how the router distributes tokens across experts and GPUs, information unavailable until routing runs. It compiles top configurations into one polymorphic kernel, then uses a small on-GPU cost model to choose among them in a few microseconds based on the actual split. On serving traffic not used to fit the model, RaMP's choices were up to 1.45x faster than vLLM's offline autotuner for the same kernel on batches of 128 tokens or more (1.27x across all batches), and within 3.5% of exhaustive configuration search. The combined kernel and selection method made the MoE layer up to 1.84x faster than vLLM's Triton MoE at 128 tokens or more; in vLLM, time per output token improved by up to 1.11x over the fastest stock MoE backend under tensor and expert parallelism.

### Source excerpt

arXiv:2604.26039v2 Announce Type: replace-cross Abstract: LLM serving systems autotune their GPU kernels offline by timing candidate kernel configurations, like tile sizes, at a range of batch sizes. This works when the number of tokens determines a kernel's work, as in dense layers. However, we show that it does not hold for Mixture-of-Experts (MoE) layers where the router decides how the tokens are split across experts and GPUs. We find that the best kernel configuration depends on this split, and not only on the number of tokens--under expert parallelism, choosing the configuration for each GPU's actual token distribution is up to 1.45x faster than the existing choice. The split is not known until the router runs, so it cannot be looked up in advance. Hence, we introduce RaMP, which chooses the kernel configuration at run time from the routing split itself. RaMP compiles top configurations into one polymorphic kernel, and a small cost model on the GPU picks among them in a few microseconds. On serving traffic its cost model was never fit on, the configurations RaMP's cost model chooses from the kernel's full configuration space are up to 1.45x faster than those vLLM's offline autotuner picks for the same kernel on batches of 128 tokens or more (1.27x over all batches), and within 3.5% of an exhaustive search that times every configuration. RaMP's kernel and choice together make the MoE layer up to 1.84x faster than vLLM's Triton MoE on batches of 128 tokens or more. In vLLM, under both tensor and expert parallelism, RaMP lowers time per output token by up to 1.11x over the fastest stock MoE backend.

## ZeRO-Prefill: Zero Redundancy Overheads in MoE Prefill Serving

DevFeed: [ZeRO-Prefill: Zero Redundancy Overheads in MoE Prefill Serving](<https://devfeed.tech/articles/zero-prefill-zero-redundancy-overheads-in-moe-prefill-serving-103768.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.02960>)

Author: Zhaoyuan Su, Olatunji Ruwase, Karthik Ganesan, Aurick Qiao, Samyam Rajbhandari, Juncheng Yang, Yue Cheng, Yuxiong He

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [GPU](<https://devfeed.tech/topics/gpu.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [classification](<https://devfeed.tech/tags/classification.md>), [communication](<https://devfeed.tech/tags/communication.md>), [compute](<https://devfeed.tech/tags/compute.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [execution](<https://devfeed.tech/tags/execution.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [frontend](<https://devfeed.tech/tags/frontend.md>), [layer](<https://devfeed.tech/tags/layer.md>), [llm](<https://devfeed.tech/tags/llm.md>), [load](<https://devfeed.tech/tags/load.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [moe](<https://devfeed.tech/tags/moe.md>), [zero](<https://devfeed.tech/tags/zero.md>)

### AI overview

ZeRO-Prefill targets MoE workloads answered from a single prefill pass, where distributed execution can add communication and synchronization overhead. Its AsyncEP backend gathers expert weights asynchronously, overlapping weight AllGather with computation in place of per-layer activation AllToAll. A co-designed frontend uses prefix-aware routing and true-FLOPs load tracking to enforce a saturation threshold. On Qwen3-235B-A22B across four hardware and precision configurations, the abstract reports 1.34-1.37x the throughput of the strongest distributed baseline on real-world workloads and up to 1.59x on long-context synthetic workloads, with 29.8-36.2% per-GPU model FLOPs utilization.

### Source excerpt

arXiv:2605.02960v3 Announce Type: replace Abstract: Production LLM workloads increasingly serve discriminative tasks, such as classification, recommendation, and verification, whose answers are read from the logits of a single prefill pass with no autoregressive decoding. Serving these prefill-only workloads on mixture-of-experts (MoE) models is bottlenecked not by compute but by the distributed execution required to fit the model: existing parallel strategies (tensor, expert, and pipeline parallelism) trade memory pressure for redundant computation, communication, and synchronization, severely degrading MoE prefill serving efficiency. We observe that these overheads stem from coupling expert placement with synchronous activation routing -- a design inherited from the decoding era. The long, compute-bound forward passes of large-batch prefill open a per-layer window wide enough to stream expert weights in the background, replacing per-layer activation AllToAll with asynchronous weight AllGather fully overlapped with computation. We propose ZeRO-Prefill, a prefill-only serving system built on this idea. Its backend, AsyncEP (Asynchronous Expert Parallelism), gathers experts by weight rather than by activation; its co-designed frontend enforces a physically derived saturation threshold through prefix-aware routing and true-FLOPs load tracking. On Qwen3-235B-A22B across four hardware/precision configurations, ZeRO-Prefill delivers 1.34-1.37x throughput over the strongest distributed baseline on real-world workloads and up to 1.59x on long-context synthetic workloads, sustaining 29.8-36.2% per-GPU model FLOPs utilization.

## NanoCP: Taming Load Imbalance in Data-Expert Parallel Decoding under Variable-Length Workloads

DevFeed: [NanoCP: Taming Load Imbalance in Data-Expert Parallel Decoding under Variable-Length Workloads](<https://devfeed.tech/articles/nanocp-taming-load-imbalance-in-data-expert-parallel-decoding-under-variable-length-workloads-95145.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.21100>)

Author: Jiefei Chen, Binbin Lin, Jinming Ma, Jiangfei Duan, Haojie Duanmu, Hao Liu, Qinxiu Cheng, Xiuhong Li, Zhilin Pei, Hui Wang, Xingcheng Zhang, Dahua Lin

Published: 2026-10-22T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [Load Balancing](<https://devfeed.tech/topics/load-balancing.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [aot](<https://devfeed.tech/tags/aot.md>), [attention](<https://devfeed.tech/tags/attention.md>), [backend](<https://devfeed.tech/tags/backend.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [batch](<https://devfeed.tech/tags/batch.md>), [cache](<https://devfeed.tech/tags/cache.md>), [communication](<https://devfeed.tech/tags/communication.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [data](<https://devfeed.tech/tags/data.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [graph](<https://devfeed.tech/tags/graph.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [latency](<https://devfeed.tech/tags/latency.md>), [load](<https://devfeed.tech/tags/load.md>), [load-balancing](<https://devfeed.tech/tags/load-balancing.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [variable](<https://devfeed.tech/tags/variable.md>)

### AI overview

NanoCP addresses load imbalance in MoE decoding by separating MoE communication from KV-cache placement and dynamically assigning each request a context-parallel degree based on its KV footprint. Long requests spread attention across multiple instances, while short requests stay local, balancing cache occupancy and batch sizes within each node. An ahead-of-time graph engine and custom routing-based communication backend connect this dynamic scheme to static execution. On the reported DeepSeek-V3 Issue 1% and Issue 5% workloads, at 99% TPOT-SLO attainment, NanoCP achieved 3.27x and 2.11x the maximum sustainable request rate of the best-performing vLLM baseline, and reduced P99 TPOT by 1.94x and 2.01x, respectively.

### Source excerpt

arXiv:2605.21100v2 Announce Type: replace Abstract: Modern serving systems for Mixture-of-Experts (MoE) models adopt hybrid data-expert parallelism: expert parallelism (EP) shards experts across GPUs to scale capacity, while data parallelism (DP) replicates attention layers across instances to process independent requests. Existing systems bind each request's attention, MoE communication, and KV cache to a single instance. Because attention latency scales with KV cache size while MoE communication latency scales with batch size, this binding cannot balance both simultaneously, producing EP stragglers; it also fragments KV memory across instances, inflating tail latency under long contexts. While existing context parallelism (CP) mitigates these constraints, existing group-level and uniform CP designs incur prohibitive communication and attention-side overheads. We present NanoCP, which decouples MoE communication from KV cache placement and achieves dual balance through dynamic context parallelism (DCP). DCP assigns each request a context-parallel degree sized to its KV footprint: long requests distribute attention across multiple instances; short requests remain local. This dynamic parallelism effectively liquefies the KV cache across DP instances within each node, balancing both the per-instance KV cache occupancy and batch sizes without unnecessary load-balancing costs. To bridge DCP with static execution, NanoCP introduces an ahead-of-time (AOT) graph engine paired with a custom routing-based communication backend. Experimental results show that, under 99% TPOT-SLO attainment, NanoCP achieves $3.27\times$ and $2.11\times$ the maximum sustainable request rate of the best-performing vLLM baseline on the DeepSeek-V3 Issue 1% and Issue 5% workloads, respectively, while reducing P99 TPOT by $1.94\times$ and $2.01\times$ on the same two workloads, respectively.

## 3D-MoE: Complementarity-Guided Hardware-Software Co-Design for MoE Decode on 3D-Stacked NMP

DevFeed: [3D-MoE: Complementarity-Guided Hardware-Software Co-Design for MoE Decode on 3D-Stacked NMP](<https://devfeed.tech/articles/3d-moe-complementarity-guided-hardware-software-co-design-for-moe-decode-on-3d-stacked-nmp-93743.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.26991>)

Author: Yudong Pan, Lian Liu, Shixin Zhao, Yintao He, Kaiyan Chang, Tianhua Han, Chenyang Ai, Haonan Zhu, Mengdi Wang, Yinhe Han, Ying Wang

Published: 2026-10-21T04:00:00Z

Content type: news

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>), [Load Balancing](<https://devfeed.tech/topics/load-balancing.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [broadcast](<https://devfeed.tech/tags/broadcast.md>), [channel](<https://devfeed.tech/tags/channel.md>), [cmn](<https://devfeed.tech/tags/cmn.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [dram](<https://devfeed.tech/tags/dram.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [hardware-software-co-design](<https://devfeed.tech/tags/hardware-software-co-design.md>), [hotspot](<https://devfeed.tech/tags/hotspot.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [layer](<https://devfeed.tech/tags/layer.md>), [lightweight](<https://devfeed.tech/tags/lightweight.md>), [load-balancing](<https://devfeed.tech/tags/load-balancing.md>), [local](<https://devfeed.tech/tags/local.md>), [moe](<https://devfeed.tech/tags/moe.md>)

### AI overview

3D-MoE addresses load imbalance in MoE decoding on tiled, 3D-stacked DRAM near-memory processors while aiming to preserve locality. It groups experts offline using complementary load fluctuations across decode steps, maps them locally, and uses a Collaborative Memory-Network to stream hot experts' weight tiles from their host DRAM channel to helper cores. Cost-aware online remapping is intended to handle workload drift. The abstract reports an average 4.63x MoE-layer speedup over expert parallelism, 1.58x-1.86x higher end-to-end throughput than the strongest NMP baselines, and 2.11% additional logic-die area.

### Source excerpt

arXiv:2610.26991v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) have become a mainstream way to scale Large Language Models (LLMs), because each token activates only a small subset of experts, reducing inference cost. However, MoE decode remains highly memory-bound and exhibits strongly skewed, time-varying expert loads. 3D-stacked DRAM near-memory processing (NMP) provides the bandwidth that decode demands, yet its tiled organization makes load balancing expensive: expert parallelism (EP) preserves locality but suffers from hotspot-induced imbalance, while more distributed strategies weaken locality and increase NoC traffic. To address this locality-balance conflict, we observe that domain-specialized token routing creates offsetting expert-load fluctuations across decode steps, a property we call expert complementarity. Using this insight, we propose 3D-MoE, a hardware--software co-designed system for MoE decode on 3D-stacked NMP. 3D-MoE uses offline, one-time complementarity-guided grouping and locality-aware mapping to establish coarse-grained inter-group balance, handles residual intra-group hotspots through a lightweight Collaborative Memory-Network (CMN) that streams each hot expert's weight tiles once from its host DRAM channel to helper cores over a reconfigured local broadcast path, and uses cost-aware online remapping to remain robust to workload drift. Across MoE models, 3D-MoE achieves an average 4.63$\times$ MoE-layer speedup over EP and improves end-to-end throughput by 1.58$\times$--1.86$\times$ over the strongest NMP baselines, with 2.11\% additional logic-die area.

## Achieving near-zero-downtime scaling in Elastic Expert Parallelism

DevFeed: [Achieving near-zero-downtime scaling in Elastic Expert Parallelism](<https://devfeed.tech/articles/achieving-near-zero-downtime-scaling-in-elastic-expert-parallelism-90281.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-10-19-elastic-ep-near-zero-downtime>)

Author: Itay Alroy (NVIDIA)

Published: 2026-10-19T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [Deployment & Scaling](<https://devfeed.tech/topics/deployment-scaling.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [CUDA Graphs](<https://devfeed.tech/topics/cuda-graphs.md>), [Heroku AI](<https://devfeed.tech/topics/heroku-ai.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [autoscalers](<https://devfeed.tech/tags/autoscalers.md>), [batch](<https://devfeed.tech/tags/batch.md>), [compilation](<https://devfeed.tech/tags/compilation.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [cuda-graphs](<https://devfeed.tech/tags/cuda-graphs.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [deployments](<https://devfeed.tech/tags/deployments.md>), [downtime](<https://devfeed.tech/tags/downtime.md>), [elastic-ep](<https://devfeed.tech/tags/elastic-ep.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [large-scale-serving](<https://devfeed.tech/tags/large-scale-serving.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [moe](<https://devfeed.tech/tags/moe.md>), [nixl](<https://devfeed.tech/tags/nixl.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

The article describes how vLLM uses asynchronous preparation and NIXL EP to scale Elastic Expert Parallelism while keeping existing workers serving requests. In a DeepSeek V4 Pro benchmark, this approach reduced serving interruptions to 0.82 seconds when scaling from 16 to 20 GPUs and 1.96 seconds when scaling back. It explains how NIXL EP reuses communication resources and CUDA graphs, and how joining workers prepare independently. In a load test, the deployment completed all 11,391 requests while remaining below the stated p95 time-to-first-token objective.

### Source excerpt

How asynchronous preparation and NIXL EP enabled near-zero-downtime Elastic EP scaling in vLLM.