# 3D-MoE: Complementarity-Guided Hardware-Software Co-Design for MoE Decode on 3D-Stacked NMP

DevFeed: [3D-MoE: Complementarity-Guided Hardware-Software Co-Design for MoE Decode on 3D-Stacked NMP](<https://devfeed.tech/articles/3d-moe-complementarity-guided-hardware-software-co-design-for-moe-decode-on-3d-stacked-nmp-93743.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.26991>)

Author: Yudong Pan, Lian Liu, Shixin Zhao, Yintao He, Kaiyan Chang, Tianhua Han, Chenyang Ai, Haonan Zhu, Mengdi Wang, Yinhe Han, Ying Wang

Published: 2026-10-21T04:00:00Z

Content type: news

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>), [Load Balancing](<https://devfeed.tech/topics/load-balancing.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [broadcast](<https://devfeed.tech/tags/broadcast.md>), [channel](<https://devfeed.tech/tags/channel.md>), [cmn](<https://devfeed.tech/tags/cmn.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [dram](<https://devfeed.tech/tags/dram.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [hardware-software-co-design](<https://devfeed.tech/tags/hardware-software-co-design.md>), [hotspot](<https://devfeed.tech/tags/hotspot.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [layer](<https://devfeed.tech/tags/layer.md>), [lightweight](<https://devfeed.tech/tags/lightweight.md>), [load-balancing](<https://devfeed.tech/tags/load-balancing.md>), [local](<https://devfeed.tech/tags/local.md>), [moe](<https://devfeed.tech/tags/moe.md>)

## AI overview

3D-MoE addresses load imbalance in MoE decoding on tiled, 3D-stacked DRAM near-memory processors while aiming to preserve locality. It groups experts offline using complementary load fluctuations across decode steps, maps them locally, and uses a Collaborative Memory-Network to stream hot experts' weight tiles from their host DRAM channel to helper cores. Cost-aware online remapping is intended to handle workload drift. The abstract reports an average 4.63x MoE-layer speedup over expert parallelism, 1.58x-1.86x higher end-to-end throughput than the strongest NMP baselines, and 2.11% additional logic-die area.

## Source excerpt

arXiv:2610.26991v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) have become a mainstream way to scale Large Language Models (LLMs), because each token activates only a small subset of experts, reducing inference cost. However, MoE decode remains highly memory-bound and exhibits strongly skewed, time-varying expert loads. 3D-stacked DRAM near-memory processing (NMP) provides the bandwidth that decode demands, yet its tiled organization makes load balancing expensive: expert parallelism (EP) preserves locality but suffers from hotspot-induced imbalance, while more distributed strategies weaken locality and increase NoC traffic. To address this locality-balance conflict, we observe that domain-specialized token routing creates offsetting expert-load fluctuations across decode steps, a property we call expert complementarity. Using this insight, we propose 3D-MoE, a hardware--software co-designed system for MoE decode on 3D-stacked NMP. 3D-MoE uses offline, one-time complementarity-guided grouping and locality-aware mapping to establish coarse-grained inter-group balance, handles residual intra-group hotspots through a lightweight Collaborative Memory-Network (CMN) that streams each hot expert's weight tiles once from its host DRAM channel to helper cores over a reconfigured local broadcast path, and uses cost-aware online remapping to remain robust to workload drift. Across MoE models, 3D-MoE achieves an average 4.63$\times$ MoE-layer speedup over EP and improves end-to-end throughput by 1.58$\times$--1.86$\times$ over the strongest NMP baselines, with 2.11\% additional logic-die area.