# Mixture of Experts (MoEs) in Transformers

DevFeed: [Mixture of Experts (MoEs) in Transformers](<https://devfeed.tech/articles/mixture-of-experts-moes-in-transformers-7358.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/moe-transformers>)

Author: Aritra Roy Gosthipaty; Pedro Cuenca; merve; Ilyas Moutawwakil; Arthur Zucker; Sergio Paniego; Pablo Montalvo

Published: 2026-02-26T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [LLMs](<https://devfeed.tech/topics/llms.md>), [Inference Performance](<https://devfeed.tech/topics/inference-performance.md>), [quantization](<https://devfeed.tech/topics/quantization.md>)

Tags: [compute](<https://devfeed.tech/tags/compute.md>), [inference](<https://devfeed.tech/tags/inference.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llms](<https://devfeed.tech/tags/llms.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [router](<https://devfeed.tech/tags/router.md>), [speed](<https://devfeed.tech/tags/speed.md>), [systems](<https://devfeed.tech/tags/systems.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [transformers-internal](<https://devfeed.tech/tags/transformers-internal.md>)

## AI overview

This article explains how Mixture of Experts models extend Transformer-based language models by replacing selected dense feed-forward layers with multiple learnable expert subnetworks. A router activates only a subset of experts for each token, allowing high total model capacity while reducing the number of active parameters used during inference. It also discusses compute efficiency, inference speed, memory requirements, parallelization, and quantization.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.