# DMuon: Scaling Matrix Optimizers in Data Parallelism with Topology-Aware Matrix Ownership and Communication

DevFeed: [DMuon: Scaling Matrix Optimizers in Data Parallelism with Topology-Aware Matrix Ownership and Communication](<https://devfeed.tech/articles/dmuon-scaling-matrix-optimizers-in-data-parallelism-with-topology-aware-matrix-ownership-and-communication-121260.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07084>)

Author: Starrick Liu, Ruizhi Chengze, Lufang Chen, Michael Zhao, Ran Yu, Xiaofan Li, dance kuzi, Yue Zhang, Zijuan Lin, SongAo Han, YongHua Chen, Ruyi Gan, Hao Wang

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [sharding](<https://devfeed.tech/topics/sharding.md>), [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>)

Tags: [cluster](<https://devfeed.tech/tags/cluster.md>), [collective](<https://devfeed.tech/tags/collective.md>), [communication](<https://devfeed.tech/tags/communication.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [gathering](<https://devfeed.tech/tags/gathering.md>), [global](<https://devfeed.tech/tags/global.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [h100](<https://devfeed.tech/tags/h100.md>), [iteration](<https://devfeed.tech/tags/iteration.md>), [layout](<https://devfeed.tech/tags/layout.md>), [llms](<https://devfeed.tech/tags/llms.md>), [load](<https://devfeed.tech/tags/load.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [muon](<https://devfeed.tech/tags/muon.md>), [network](<https://devfeed.tech/tags/network.md>), [ownership](<https://devfeed.tech/tags/ownership.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [topology](<https://devfeed.tech/tags/topology.md>)

## AI overview

DMuon makes matrix optimizers practical with sharded data-parallel training by assigning each weight matrix to one owner, which runs the Newton-Schulz iteration and update on the full matrix. Owners are balanced by computational cost. In HSDP, ownership spans replicas, keeping one cluster-wide copy of gradients and optimizer state; updated parameters are published across nodes as they become ready. DMuon schedules sparse, variable-length collectives as data becomes available and routes them over the cluster's two-level network. Across three dense LLMs and two VLA models on up to 256 H100 GPUs, it matched or exceeded PyTorch FSDP2 with AdamW, reaching 1.05x its median throughput at 256 GPUs. It achieved 2.1x the median throughput of a gather-then-replicate Muon baseline. These are reported results for the tested models and scale.

## Source excerpt

arXiv:2611.07084v1 Announce Type: new Abstract: Matrix optimizers such as Muon update whole weight matrices, which conflicts with the element- or row-wise sharding of data-parallel runtimes. Once parameters are sharded as well, matrix optimizers incur extra communication or repeated computation. We present DMuon, a matrix-granular optimizer runtime for PyTorch FSDP (ZeRO-2/3) and HSDP. DMuon assigns each matrix a unique owner, which performs the Newton--Schulz (NS) iteration and optimizer update on the full matrix, and balances the load across owners by computational cost. On HSDP, ownership is global across replicas, so the cluster keeps a single copy of gradients and optimizer state, and updated parameters are published across nodes as soon as they are ready, overlapping subsequent computation. This layout makes every collective sparse and variable in length, so DMuon schedules these collectives as their data becomes ready and routes them along the cluster's two-level network. On up to 256 H100 GPUs, across three dense LLMs and two VLA models, DMuon matches or exceeds the training throughput of PyTorch FSDP2 with AdamW, reaching 1.05$\times$ that of FSDP2 with AdamW at the median on 256 GPUs. The optimized parameter gathering and gradient reduction absorb the extra communication and computation a matrix optimizer requires. Against a gather-then-replicate Muon baseline, DMuon's throughput is 2.1 times as high at the median.