# Server-Side Sparse Assignment for Federated Mixture-of-Experts

DevFeed: [Server-Side Sparse Assignment for Federated Mixture-of-Experts](<https://devfeed.tech/articles/server-side-sparse-assignment-for-federated-mixture-of-experts-120552.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07093>)

Author: Xue Zheng, Aylin Yener

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [cs-it](<https://devfeed.tech/tags/cs-it.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data-privacy](<https://devfeed.tech/tags/data-privacy.md>), [edge](<https://devfeed.tech/tags/edge.md>), [error](<https://devfeed.tech/tags/error.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [experts](<https://devfeed.tech/tags/experts.md>), [exploit](<https://devfeed.tech/tags/exploit.md>), [federated](<https://devfeed.tech/tags/federated.md>), [federated-learning](<https://devfeed.tech/tags/federated-learning.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [fine-tuning-llms](<https://devfeed.tech/tags/fine-tuning-llms.md>), [math-it](<https://devfeed.tech/tags/math-it.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [moe](<https://devfeed.tech/tags/moe.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [relevance](<https://devfeed.tech/tags/relevance.md>), [routing](<https://devfeed.tech/tags/routing.md>), [server](<https://devfeed.tech/tags/server.md>), [top-k](<https://devfeed.tech/tags/top-k.md>)

## AI overview

S³-MoE assigns each federated client a sparse set of relevant experts using server-side estimates from partial observations, refined through an exploit-explore mechanism. The authors state relevance-estimation error and exact Top-K recovery guarantees. In a CNN-based MoE experiment on CIFAR-100, the method recovered task-dependent client-expert preferences and reduced conflicts from correlated expert selection. Against Full-20 FedAvg, which exposes all 20 experts for local Top-5 routing, S³-MoE reported 1.31 percentage points higher mean final accuracy while loading five experts per client. These results are limited to the described experiment.

## Source excerpt

arXiv:2611.07093v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly recognized for their remarkable capabilities; federated learning (FL) offers a promising framework for collaboratively fine-tuning LLMs on edge devices while achieving data privacy. However, edge devices are inherently heterogeneous: clients may exhibit diverse data distributions and resource constraints, including limited memory and computation budgets. Recent federated Mixture-of-Experts (MoE) approaches address these challenges by using sparse expert activation or partial expert loading. However, existing partial-expert methods rely on heuristic full-pool observations or prioritize system-wide load balance, without guarantees for recovering client-specific relevant experts from partial observations. We propose $S^3$-MoE, a server-side sparse expert assignment framework that estimates client-specific expert relevance from partial observations and refines assignments through an exploit--explore mechanism. We also establish relevance-estimation error and exact Top-$K$ recovery guarantees. Experiments using a CNN-based MoE on CIFAR-100 show that $S^3$-MoE recovers task-dependent client--expert preferences and reduces conflicts caused by correlated expert selection across clients. Compared with Full-20 FedAvg, which gives each client access to all 20 experts for local Top-5 routing, $S^3$-MoE achieves 1.31 percentage points higher mean final accuracy while loading only $5$ experts per client.