# DPA: Decentralized Primal Averaging with Quasi-Global Momentum for Highly Heterogeneous Data

DevFeed: [DPA: Decentralized Primal Averaging with Quasi-Global Momentum for Highly Heterogeneous Data](<https://devfeed.tech/articles/dpa-decentralized-primal-averaging-with-quasi-global-momentum-for-highly-heterogeneous-data-132285.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16871>)

Author: Diyako Ghaderyan, Stefan Werner

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Federated Learning](<https://devfeed.tech/topics/federated-learning.md>), [cifar-10](<https://devfeed.tech/topics/cifar-10.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [cifar-10](<https://devfeed.tech/tags/cifar-10.md>), [communication](<https://devfeed.tech/tags/communication.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decentralized](<https://devfeed.tech/tags/decentralized.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [image-classification](<https://devfeed.tech/tags/image-classification.md>), [linear](<https://devfeed.tech/tags/linear.md>), [momentum](<https://devfeed.tech/tags/momentum.md>), [on-device-learning](<https://devfeed.tech/tags/on-device-learning.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [primal](<https://devfeed.tech/tags/primal.md>), [rings](<https://devfeed.tech/tags/rings.md>)

## AI overview

DPA combines primal averaging with a local quasi-global momentum buffer to address heterogeneous data in decentralized training, including over sparse communication graphs. It maintains coupled sequences for optimization, gradient evaluation, and output averaging, and uses consecutive mixed iterates to form a network-aware momentum direction without sending an auxiliary tracking variable. For smooth nonconvex objectives, the authors establish a convergence bound and conditions for linear speedup with enough iterations. DPA-1G uses one communication round per iteration, matching the per-iteration communication cost of DSGD-style baselines. In the reported GUT image-classification experiments, DPA compares favorably with five named baselines across datasets, architectures, topologies, and heterogeneity levels; gains are largest under high heterogeneity on sparse rings. At equal total communication cost, both variants achieve higher mean test accuracy than the best baseline on CIFAR-10 and CIFAR-100 with ResNet-20.

## Source excerpt

arXiv:2611.16871v1 Announce Type: cross Abstract: Decentralized training of deep learning models keeps training data local and supports on-device learning over networks. In realistic scenarios, heterogeneity across agents' local data poses an optimization challenge and can severely degrade test accuracy, especially on sparse communication graphs. Building on recent advances in centralized primal averaging, we propose Decentralized Primal Averaging (DPA), a decentralized learning algorithm that combines primal averaging with a local quasi-global momentum buffer. DPA maintains three coupled sequences for optimization, gradient evaluation, and output averaging, while using consecutive mixed iterates to construct a network-aware momentum direction without transmitting an auxiliary tracking variable. For smooth nonconvex objectives, we establish a convergence bound for DPA and identify conditions for linear speedup in the number of agents for sufficiently many iterations. We also introduce DPA-1G, a variant that uses one communication round per iteration and matches the per-iteration communication cost of DSGD-style baselines. Experiments following the GUT decentralized image-classification benchmark show that DPA compares favorably with DSGD, gradient tracking, QG-DSGDm, momentum tracking, and QG-GUTm across datasets, architectures, topologies, and heterogeneity levels, with the largest gains under high heterogeneity on sparse rings. At the same total communication cost, both DPA variants achieve higher mean test accuracy than the best baseline on CIFAR-10 and CIFAR-100 with ResNet-20. Our code is publicly available.