# Training an MoE LLM in MXFP4: How We Closed the Gap to BF16

DevFeed: [Training an MoE LLM in MXFP4: How We Closed the Gap to BF16](<https://devfeed.tech/articles/training-an-moe-llm-in-mxfp4-how-we-closed-the-gap-to-bf16-99974.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.32229>)

Author: Han Wang, Zhitao Wang, Natalia Frumkin, Xuanwu Yin, Alireza Khodamoradi, Dong Li, Emad Barsoum

Published: 2026-10-26T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [gpt-oss](<https://devfeed.tech/topics/gpt-oss.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llm](<https://devfeed.tech/tags/llm.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [mlperf](<https://devfeed.tech/tags/mlperf.md>), [moe](<https://devfeed.tech/tags/moe.md>), [mxfp4](<https://devfeed.tech/tags/mxfp4.md>), [results](<https://devfeed.tech/tags/results.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

ALTO's MXFP4 training recipe brings GPT-OSS-20B close to its BF16 validation result: on C4 at global batch size 16, MXFP4 with randomized Hadamard transforms, stochastic rounding, and weight de-oscillation reached loss 3.3350 at 16,128 steps, versus 3.3283 for BF16. De-oscillation closed about half the gap versus MXFP4 without it (loss 3.3418) and reduced FP4's extra steps to reach the MLPerf loss target of 3.34 from 20% to 5% over BF16. The recipe combines hybrid block quantization and other established techniques with a simplified de-oscillation method. Differential gradient estimation, outlier clipping, and macro-block scaling improved operator-level signal-to-noise ratio but hurt end-to-end results. Experiments used simulated MXFP4 kernels on AMD MI300; wall-clock performance was not optimized.

## Source excerpt

arXiv:2610.32229v1 Announce Type: new Abstract: Training large language models (LLMs) in sub-8-bit arithmetic promises substantial gains in memory bandwidth and compute throughput, yet FP4 formats introduce quantization error severe enough to destabilize optimization if left unaddressed. We present ALTO, an open-source training recipe for GPT-OSS-20B, a 20-billion-parameter mixture-of-experts (MoE) model, on the MLPerf Small MoE Training benchmark under the MXFP4 microscaling format. The recipe combines established techniques from Abecassis, Felix et al. [arXiv:2509.25149] (hybrid 1D/2D block quantization, randomized Hadamard transforms, and stochastic rounding) with a simplified weight de-oscillation scheme adapted from TetraJet-v2 [arXiv:2510.27527]. On C4 validation at global batch size 16, MXFP4 + RHT + SR + de-oscillation reaches a validation loss of 3.3350 at 16,128 steps, only 0.007 above the BF16 baseline (3.3283), closing roughly 50% of the gap left by MXFP4 without de-oscillation (3.3418). Measured by steps to reach the MLPerf quality target (validation loss 3.34), de-oscillation cuts the FP4 convergence overhead relative to BF16 from +20.0% to +5.0%. We also report negative end-to-end results from differential gradient estimation, outlier clipping, and macro-block scaling, despite each showing operator-level SNR gains. All experiments use simulated MXFP4 kernels on AMD MI300 hardware and wall-clock speed has not been optimized.