# Do Mixture-of-Experts Routers Benefit from Ensembling?

DevFeed: [Do Mixture-of-Experts Routers Benefit from Ensembling?](<https://devfeed.tech/articles/do-mixture-of-experts-routers-benefit-from-ensembling-135069.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17472>)

Author: Viktor Vesel\\'y, Sebastian Jitaru

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>)

Tags: [better](<https://devfeed.tech/tags/better.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diversity](<https://devfeed.tech/tags/diversity.md>), [image-classification](<https://devfeed.tech/tags/image-classification.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [neural-networks](<https://devfeed.tech/tags/neural-networks.md>), [regression](<https://devfeed.tech/tags/regression.md>), [routers](<https://devfeed.tech/tags/routers.md>), [routing](<https://devfeed.tech/tags/routing.md>)

## AI overview

The study finds that ensembling Mixture-of-Experts routers mildly improves routing performance, but the gains come with the computational cost of training extra routers. In underparameterized networks evaluated on two image-classification and two tabular-regression tasks, adding routers lowered validation loss. Explicitly encouraging expert diversity improved mean performance on three of the four tasks, while explicitly maximizing router diversity had no effect. Combining router weights by arithmetic mean performed better than using geometric mean. These results are limited to the four evaluated tasks.

## Source excerpt

arXiv:2611.17472v1 Announce Type: cross Abstract: Ensembles of neural networks have repeatedly been shown to outperform single networks. Ensemble theory attributes this gain to the accuracy of the individual members and the diversity among them. Mixture-of-experts models already combine many experts, but a single router decides how the experts are weighted for each input. Motivated by ensemble theory, we apply ensembling to the routing step: an ensemble of routers, trained jointly with the experts, proposes the expert weights. We systematically evaluate this approach using underparameterized networks on two image-classification and two tabular-regression tasks. We analyse each experiment with a performance-diversity decomposition of the router ensemble, which, to our knowledge, has not previously been developed for routers. We report four findings. First, more routers lower the validation loss. Second, training routers to explicitly maximize expert diversity improves the mean performance on three of the four tasks. Third, explicitly maximizing router diversity has no effect on performance. Fourth, combining router weights with the arithmetic mean works better than combining them with the geometric mean. Routing benefits mildly from ensembling, but this benefit is offset by the computational cost of training additional routers.