# ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Serving

DevFeed: [ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Serving](<https://devfeed.tech/articles/elmoe-3d-leveraging-intrinsic-elasticity-of-moe-for-hybrid-bonding-enabled-self-speculative-decoding-in-on-premises-serving-116465.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2604.14626>)

Author: Yuseon Choi, Jingu Lee, Jungjun Oh, Sunjoo Whang, Byeongcheol Kim, Minsung Kim, Junhyuk Lee, Hoi-Jun Yoo, Sangjin Kim

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Model Optimizer](<https://devfeed.tech/topics/model-optimizer.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [batch](<https://devfeed.tech/tags/batch.md>), [cache](<https://devfeed.tech/tags/cache.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [draft](<https://devfeed.tech/tags/draft.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [elastic](<https://devfeed.tech/tags/elastic.md>), [energy](<https://devfeed.tech/tags/energy.md>), [energy-efficiency](<https://devfeed.tech/tags/energy-efficiency.md>), [experts](<https://devfeed.tech/tags/experts.md>), [exploits](<https://devfeed.tech/tags/exploits.md>), [hybrid-bonding](<https://devfeed.tech/tags/hybrid-bonding.md>), [moe](<https://devfeed.tech/tags/moe.md>), [serving](<https://devfeed.tech/tags/serving.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>)

## AI overview

ELMoE-3D combines cache-based acceleration with speculative decoding to address MoE serving's memory bottleneck across batch sizes. Its Elastic-SD scales expert and bit dimensions together, acting as both an expert cache and a self-draft model; hybrid-bonding bandwidth accelerates it. An LSB-augmented bit-sliced architecture supports bit-nested execution by exploiting redundancy in bit-slice representations. On the described 3D-stacked hardware, the authors report average gains of 6.6x in speed and 4.4x in energy efficiency over naive MoE serving on xPU for batch sizes 1-16, and 2.2x speed and 1.4x energy efficiency over prior accelerator baselines. These are reported abstract results; the excerpt does not provide evaluation details.

## Source excerpt

arXiv:2604.14626v3 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) models have become the dominant architecture for large-scale language models, yet on-premises serving remains fundamentally memory-bound as batching turns sparse per-token compute into dense memory activation. Memory-centric architectures (PIM, NMP) improve bandwidth but leave compute underutilized under MoE's low arithmetic intensity on high batch sizes. Speculative decoding (SD) trades idle compute for fewer target invocations, yet verification must load experts even for rejected tokens, severely limiting its benefit in MoE especially on low batch sizes. We propose ELMoE-3D, a hybrid-bonding (HB)-based HW-SW co-designed framework that unifies cache-based acceleration and speculative decoding to offer overall speedup across batch sizes. We identify two intrinsic elasticity axes of MoE---expert and bit---and jointly scale them to construct Elastic Self-Speculative Decoding (Elastic-SD), which serves as both an expert cache and a strongly aligned self-draft model accelerated by high HB bandwidth. Our LSB-augmented bit-sliced architecture exploits inherent redundancy in bit-slice representations to natively support bit-nested execution. On our 3D-stacked hardware, ELMoE-3D achieves an average $6.6\times$ speedup and $4.4\times$ energy efficiency gain over naive MoE serving on xPU across batch sizes 1--16, and delivers $2.2\times$ speedup and $1.4\times$ energy efficiency gain over prior accelerator baselines.