# GC-MoE: Overlapping Expert-Parallel Token Dispatch with FP8/FP4 Grouped GEMM via CUDA Green Contexts

DevFeed: [GC-MoE: Overlapping Expert-Parallel Token Dispatch with FP8/FP4 Grouped GEMM via CUDA Green Contexts](<https://devfeed.tech/articles/gc-moe-overlapping-expert-parallel-token-dispatch-with-fp8-fp4-grouped-gemm-via-cuda-green-contexts-116485.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.04222>)

Author: Yong Li, Kan Liu, Tao Lan

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [CUDA](<https://devfeed.tech/topics/cuda.md>), [CUDA Graphs](<https://devfeed.tech/topics/cuda-graphs.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [synchronization](<https://devfeed.tech/topics/synchronization.md>)

Tags: [blackwell](<https://devfeed.tech/tags/blackwell.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [decode](<https://devfeed.tech/tags/decode.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [deepseek-v4-1-flash](<https://devfeed.tech/tags/deepseek-v4-1-flash.md>), [dispatch](<https://devfeed.tech/tags/dispatch.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [glm](<https://devfeed.tech/tags/glm.md>), [glm-5-2](<https://devfeed.tech/tags/glm-5-2.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [graph](<https://devfeed.tech/tags/graph.md>), [handshake](<https://devfeed.tech/tags/handshake.md>), [hopper](<https://devfeed.tech/tags/hopper.md>), [inference](<https://devfeed.tech/tags/inference.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [open-source](<https://devfeed.tech/tags/open-source.md>)

## AI overview

GC-MoE overlaps expert-parallel token dispatch and grouped GEMM by using CUDA Green Contexts to assign communication and compute to separate GPU partitions, each with its own kernel resources and synchronization. A concurrent CUDA Graph launch, token-ring handshake, and double-buffered receive counters enable early computation and avoid a trailing cross-rank barrier between replays. For computation, it uses operand-swapped block-scaled MMA on Blackwell and register-A WGMMA with in-register FP4 decoding on Hopper; fused SwiGLU, requantization, and optional shared experts are supported. In the reported tests, it beat Mega MoE at all 54 matched-accuracy points: average decode speedups were 1.188x on 8 H20 GPUs for DeepSeek-V4.1-Flash, and 1.106x on 8 B300 GPUs for GLM-5.2 and DeepSeek-V4.1-Flash (1.123x with fused shared experts). Dispatch used 20 of 148 B300 SMs.

## Source excerpt

arXiv:2611.04222v1 Announce Type: new Abstract: Under expert parallelism, mixture-of-experts (MoE) layers pay all-to-all token dispatch, grouped GEMMs, and a combine step in every layer. In decode each rank holds only 1-256 tokens: dispatch latency and weight traffic dominate while tensor-core utilization is inherently low. In fused operators such as DeepGEMM's Mega MoE, dispatch and computation overlap only by sharing every SM: dispatch warps compete with TMA/MMA and epilogue warps for registers, shared memory, and issue slots; every phase shares one CTA resource budget, synchronization spans the whole SM array, and every replay ends with a workspace-clean pass and a cross-rank barrier. We present GC-MoE, an open-source FP8/FP4 MoE inference operator for NVIDIA Hopper and Blackwell GPUs that uses CUDA Green Contexts to split the device into a small communication partition and a large compute partition. Each role gets its own kernel, resource budget, and synchronization domain, so dispatch and GEMM tune independently. A three-node CUDA Graph (four on Hopper with fused shared experts) launches the dispatch and grouped-GEMM kernels as concurrent roots; a token-ring handshake starts computation on the first filled ring blocks, and double-buffered receive counters remove the trailing cross-rank barrier between replays. The compute partition runs operand-swapped tcgen05 block-scaled MMA on Blackwell and register-A WGMMA with in-register FP4 decoding on Hopper, with fused SwiGLU, requantization, and optional shared experts. On 8 H20 GPUs GC-MoE accelerates DeepSeek-V4.1-Flash decode by 1.188x on average on both the routed and shared-expert suites (peaks 1.296x and 1.353x); on 8 B300 GPUs it accelerates GLM-5.2 and DeepSeek-V4.1-Flash by 1.106x (1.123x with fused shared experts, peak 1.209x), beating Mega MoE at all 54 measured points at matched accuracy. GC-MoE reserves only 20 of 148 SMs on B300 for dispatch.