# Mixture of Experts (MoE)

A machine learning technique that combines specialized expert models, with a gating network activating experts for an input.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## CGM: Chaos-Guided Graph Attention and Expert Routing for Autism Stereotypy Detection

DevFeed: [CGM: Chaos-Guided Graph Attention and Expert Routing for Autism Stereotypy Detection](<https://devfeed.tech/articles/cgm-chaos-guided-graph-attention-and-expert-routing-for-autism-stereotypy-detection-145094.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25730>)

Author: Hafsa Aslam, Yue Li, Saba Aslam, Abdur Rasool, Qiao Pan, Muhammad Rizwan, Ovidiu Bagdasar

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Video Analytics](<https://devfeed.tech/topics/video-analytics.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [amplitude](<https://devfeed.tech/tags/amplitude.md>), [angular](<https://devfeed.tech/tags/angular.md>), [attention](<https://devfeed.tech/tags/attention.md>), [autism](<https://devfeed.tech/tags/autism.md>), [chaos](<https://devfeed.tech/tags/chaos.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [detection](<https://devfeed.tech/tags/detection.md>), [frame](<https://devfeed.tech/tags/frame.md>), [gate](<https://devfeed.tech/tags/gate.md>), [graph](<https://devfeed.tech/tags/graph.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [model](<https://devfeed.tech/tags/model.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [pairs](<https://devfeed.tech/tags/pairs.md>), [public](<https://devfeed.tech/tags/public.md>), [recordings](<https://devfeed.tech/tags/recordings.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [routing](<https://devfeed.tech/tags/routing.md>), [sample](<https://devfeed.tech/tags/sample.md>), [sessions](<https://devfeed.tech/tags/sessions.md>)

### AI overview

CGM uses per-joint, per-frame angular-velocity irregularity to guide skeleton graph attention and route each frame to two of four experts. In the reported cohort, windowed Sample Entropy rose in stereotyped clips while amplitude-based descriptors did not. On 116 clips with subject-grouped five-fold cross-validation, CGM achieved 0.930 balanced accuracy, 0.187 above the graph-attention baseline. On held-out Panoptic Studio sessions, it maintained 0.86 specificity without retraining, versus at most 0.35 for chaos-free baselines. Ablations found no reliable gain from routing without irregularity; chaos modulation alone contributed +0.132. The excerpt also reports a severity index with 0.87 split-half reliability and consistency within children across therapy recordings. These are reported study results, not evidence here of broader clinical validation.

### Source excerpt

arXiv:2611.25730v1 Announce Type: cross Abstract: Skeleton graph attention for autism video analysis weights joints by content alone, giving an erratic wrist the same standing as a still one. Stereotypy, however, is defined by how unpredictably movement unfolds rather than how much of it occurs: in our cohort, every amplitude-based kinematic descriptor falls in stereotyped clips, while windowed Sample Entropy of angular velocity alone rises (Hedges' g = +1.05). We compute this irregularity as a per-joint, per-frame tensor that structurally drives computation. The proposed chaos-guided graph attention and mixture-of-experts model (CGM) couples two mechanisms. First, a learned pairwise term added to the graph-attention logit concentrates attention on irregular joints and re-ranks the remaining edge weight toward jointly irregular pairs; its unconstrained coefficient neither vanishes nor inverts, and zeroing the tensor realigns attention in 94% of clips. Second, a gate conditioned on the same signal routes each frame to two of four experts, whose weights track frame-level irregularity. Under subject-grouped five-fold cross-validation on 116 clips, CGM reaches 0.930 balanced accuracy, exceeding the graph-attention baseline by +0.187; the component ordering replicates on the public MMASD dataset, and on held-out Panoptic Studio sessions, CGM keeps 0.86 specificity without retraining, against at most 0.35 for chaos-free baselines. In ablation, routing without irregularity yields no reliable gain, whereas chaos modulation alone contributes +0.132. A severity index from the same tensor reaches 0.87 split-half reliability, separates the cohorts with Hedges' g = +1.08, and stays consistent within children across therapy recordings, supporting video-only monitoring of stereotypy during intervention.

## Beam Model Technical Report

DevFeed: [Beam Model Technical Report](<https://devfeed.tech/articles/beam-model-technical-report-145663.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26358>)

Author: Reflection Team, Adam Abdulhamid, Osama Abuelsorour, Yasmin Alameddine, Carlo Alfano, Duarte Alves, Brandon Amos, Selvaraj Anandaraj, Arseni Anisimovich, George Antoniadis, Ioannis Antonoglou, Sheila

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Model Optimizer](<https://devfeed.tech/topics/model-optimizer.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>)

Tags: [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [beam](<https://devfeed.tech/tags/beam.md>), [co-design](<https://devfeed.tech/tags/co-design.md>), [coding](<https://devfeed.tech/tags/coding.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [gb300](<https://devfeed.tech/tags/gb300.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [open](<https://devfeed.tech/tags/open.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [throughput](<https://devfeed.tech/tags/throughput.md>)

### AI overview

Beam is a 501.3B-parameter sparse MoE model with 23.4B active parameters, designed for coding, reasoning, and agentic workloads. The abstract reports a 1M-token context window enabled by two-stage midtraining on verifiable, correctness-anchored data, following pretraining on 23.8T tokens. Post-training used asynchronous RL on 10.5K GB300 GPUs, producing over 100M rollouts in four weeks; measured capabilities continued to improve as RL compute increased, with no plateau reported. Hardware-model co-design, FP8/NVFP4 quantization, and a 1.93B-parameter DFlash 2 speculative decoding drafter target serving throughput. The report claims 3-4x less inference compute than many comparable models. On its safety evaluation, Beam refused or pivoted on 88.9% of harmful requests while complying with 81.5% of harmless requests; the text says its harmful-request refusal/pivot rate was highest among open-weight models on the Pareto frontier.

### Source excerpt

arXiv:2611.26358v1 Announce Type: cross Abstract: Beam is a sparse Mixture-of-Experts (MoE) model with 501.3B total parameters and 23.4B active parameters, specifically built and optimized for coding, reasoning, and agentic workloads. Beam delivers frontier level reasoning, while requiring 3-4x less inference compute than many models of its class. Pretrained on 23.8T diverse, curated, high-quality tokens, Beam features a 1M-token context window achieved through a novel two-stage midtraining phase, built on verifiable, correctness-anchored data. Post-training leveraged a large-scale, fully asynchronous reinforcement learning (RL) campaign across 10.5K GB300 GPUs, generating over 100M rollouts over 4 weeks of training. We believe this is one of the largest scale RL runs conducted by any open lab to date. Across our evaluation suite, capabilities continued to improve as we increased RL compute, with no sign of a plateau. Supported by deep hardware-model co-design, FP8/NVFP4 quantization, and a 1.93B-parameter speculative decoding drafter (DFlash 2), Beam maximizes serving throughput with efficient, flexible deployment. Averaged across conversation turns and harm categories, Beam refuses or pivots on 88.9% of harmful requests while complying with 81.5% of harmless requests, achieving the highest refusal/pivot rate on harmful requests among open-weight models on the Pareto frontier.

## Relax Within, Balance Across: Load Balancing under Composition Shifts in Vision-Language Mixture-of-Experts

DevFeed: [Relax Within, Balance Across: Load Balancing under Composition Shifts in Vision-Language Mixture-of-Experts](<https://devfeed.tech/articles/relax-within-balance-across-load-balancing-under-composition-shifts-in-vision-language-mixture-of-experts-145359.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.00574>)

Author: Ziang Wu, Peng Jin, Qishen Yin, Munan Ning, Hao Li, Peizhen Zhang, Huihui Zhou, Li Yuan

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>)

Tags: [balance](<https://devfeed.tech/tags/balance.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [cancellation](<https://devfeed.tech/tags/cancellation.md>), [changes](<https://devfeed.tech/tags/changes.md>), [code](<https://devfeed.tech/tags/code.md>), [composition](<https://devfeed.tech/tags/composition.md>), [counts](<https://devfeed.tech/tags/counts.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [errors](<https://devfeed.tech/tags/errors.md>), [experts](<https://devfeed.tech/tags/experts.md>), [fast](<https://devfeed.tech/tags/fast.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [flat](<https://devfeed.tech/tags/flat.md>), [gap](<https://devfeed.tech/tags/gap.md>), [image](<https://devfeed.tech/tags/image.md>), [language](<https://devfeed.tech/tags/language.md>), [load](<https://devfeed.tech/tags/load.md>), [load-balancing](<https://devfeed.tech/tags/load-balancing.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

On a fixed vision-language MoE router, balancing image and text tokens in one pooled auxiliary loss can conceal opposing modality load errors: they cancel near the training image-to-text ratio, but load skews as resolution, tiling, or image count changes that mix. The paper derives a quadratic response law under fixed modality profiles, with the image-text load gap governing sensitivity. It also reports that patches from an image route similarly as patch counts vary, allowing large images to dominate flat pooling. ReBA separates modality losses by token fraction and weights images equally, allowing per-image specialization while balancing visual load overall. Across four split backbones, the authors report stable balance across resolutions, tilings, and image counts without hurting accuracy; large modality gaps in released native MoEs suggest potential for native post-training.

### Source excerpt

arXiv:2608.00574v2 Announce Type: replace-cross Abstract: Vision-language mixture-of-experts (MoE) models such as Qwen3-VL need balanced loads for fast parallel serving. Standard training pools image and text tokens into one auxiliary loss, assuming training balance holds at inference. Yet we find a surprising mismatch: on a fixed router, balance shifts sharply whenever resolution, dynamic tiling, or image count changes the image-to-text token ratio. Splitting load by modality reveals cross-modal load cancellation: opposing image and text errors cancel near the training ratio, hiding strong skew until the mix changes. To quantify this shift, we derive an exact quadratic law: under fixed modality profiles, one reference run predicts the full response curve, and the image-text gap governs sensitivity. Meanwhile, patches within an image route similarly while patch counts vary, letting large images dominate flat pooling. We propose ReBA (Relax Within, Balance Across) to separate modality losses by token fraction and weight images equally, letting each image specialize while keeping overall visual load even. Across four split backbones, ReBA turns fragile single-ratio balance into stable balance across resolutions, tilings, and image counts without hurting accuracy. Large modality gaps in released native MoEs also show its promise for native post-training. Code is available at https://github.com/ZiangWu-77/ReBA.

## Game-Theoretic Data Allocation for Expert Specialization and Routing

DevFeed: [Game-Theoretic Data Allocation for Expert Specialization and Routing](<https://devfeed.tech/articles/game-theoretic-data-allocation-for-expert-specialization-and-routing-145017.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25167>)

Author: Mark Lovett, Denis Lim, Soroush Vosoughi

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>)

Tags: [behavior](<https://devfeed.tech/tags/behavior.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ma](<https://devfeed.tech/tags/cs-ma.md>), [data](<https://devfeed.tech/tags/data.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [experts](<https://devfeed.tech/tags/experts.md>), [game](<https://devfeed.tech/tags/game.md>), [gate](<https://devfeed.tech/tags/gate.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [lora](<https://devfeed.tech/tags/lora.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompts](<https://devfeed.tech/tags/prompts.md>), [response](<https://devfeed.tech/tags/response.md>), [router](<https://devfeed.tech/tags/router.md>), [routing](<https://devfeed.tech/tags/routing.md>), [safety](<https://devfeed.tech/tags/safety.md>), [specialization](<https://devfeed.tech/tags/specialization.md>), [theoretical](<https://devfeed.tech/tags/theoretical.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

InflEns uses one location game in a fixed prompt-trait space to determine both training weights for LoRA experts and inference routing, without using expert losses or training an explicit gate. On a corpus from seven safety-related benchmarks, the highest-share expert had the lowest held-out response NLL on about 90% of prompts across four kernels with a sharpened simplex representation; permuting allocations across prompts reduced this to about 15%. The method reportedly performed comparably to published learned gates. Its fixed router also let the authors isolate and retrain one harmful expert, lowering HarmBench attack success from 0.51 to 0.02 while maintaining instruction following. Specialization can be controlled through kernel choice and parameters, trait-space geometry, and expert count. These findings are reported for the described evaluation; broader applicability is presented as a framework claim.

### Source excerpt

arXiv:2611.25167v1 Announce Type: new Abstract: Can the allocation used to train experts also predict which expert will perform best on a prompt? Such a connection would make the intended division of labor inspectable before training and allow the allocation itself to guide inference. We introduce InflEns, a mixture of LoRA experts whose training weights and inference routes come from the same location game in a fixed prompt-trait space. Players choose positions to maximize their expected shares of the prompt distribution, without using expert losses. We utilize the game's kernel-general theoretical criterion for when co-located positions become locally unstable to separation. We evaluate InflEns on a corpus assembled from seven safety-related benchmarks, which demonstrates that the highest-share expert has the lowest held-out response negative log-likelihood (NLL) on approximately 90% of prompts across four kernels under a sharpened simplex representation, compared with approximately 15% after permuting allocations across prompts. We demonstrate that the method performs comparably to published learned gates without having to train an explicit gate on the experts. Because the router is fixed, we can trace harmful behavior to a single expert and retrain only that expert, which lowers HarmBench attack success from 0.51 to 0.02 while keeping instruction following. The game also gives us several controls over how the experts specialize, including the kernel choice and its parameters, the trait-space geometry, and the number of experts. Importantly, InflEns is a general framework that can be extended to any encoder and data distribution.

## RouteVerify: Routing Interventions for Step-Level Verification in Mixture-of-Experts Models

DevFeed: [RouteVerify: Routing Interventions for Step-Level Verification in Mixture-of-Experts Models](<https://devfeed.tech/articles/routeverify-routing-interventions-for-step-level-verification-in-mixture-of-experts-models-145103.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25780>)

Author: Yucheng Du, Ryan A. Rossi, Xiyang Hu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [routing](<https://devfeed.tech/tags/routing.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

RouteVerify finds that routing signals can reveal step correctness without reliably carrying that correctness into a model's verification judgment. Across nine MoE checkpoints, router-logit probes distinguished correct from erroneous steps in eight (AUROC 0.80-0.88), and replacing router logits across all routed layers changed the INCORRECT-versus-CORRECT verification margin in all nine, more than norm-matched random replacements. Yet the effect rarely depended on whether the replacement came from a correct or erroneous step: at one layer, margins were indistinguishable in all checkpoints; with all-layer routing, a positive difference appeared in only one. Local residual replacements showed such a difference in eight of nine checkpoints, suggesting the measure could detect source-correctness dependence when present. The study also compared MoE-output interventions, but the supplied excerpt gives no specific result for that interface.

### Source excerpt

arXiv:2611.25780v1 Announce Type: new Abstract: Routing states in mixture-of-experts (MoE) models can distinguish correct from erroneous reasoning steps. If changing routing also changes a model's step-level verification judgment, it is tempting to conclude that the judgment follows what routing encodes. We test this inference with RouteVerify, a controlled intervention study that replaces selected internal activations from one solution trace with activations from another, while keeping the evaluated input unchanged. Across nine MoE checkpoints, we compare three intervention interfaces: router logits, MoE outputs, and post-MoE residual states. We vary the layers and token positions at which activations are replaced. We measure changes in a verification margin that quantifies the model's preference for INCORRECT over CORRECT. We also evaluate the same input with activations from correct versus erroneous steps to test whether the resulting judgment depends on the source step's correctness. We report four linked results. First, probes on router logits separate correct from erroneous steps in eight checkpoints (AUROC 0.80-0.88). Second, replacing router logits across all routed layers changes the judgment in all nine checkpoints, and by more than a norm-matched random replacement. Third, this change rarely depends on whether the source step is correct. At one layer, routing from erroneous steps and from correct steps produces indistinguishable margins in all nine checkpoints. Under all-layer routing, their difference is resolved and positive in only one. Fourth, the same difference is positive for local residual replacements in eight of nine checkpoints, so the measure does detect such dependence where it exists. Correctness is therefore readable from routing, and routing replacements move the judgment, but the movement rarely follows correctness.

## Validation-driven Deployment of LLM Inference Platforms for Agentic Workloads

DevFeed: [Validation-driven Deployment of LLM Inference Platforms for Agentic Workloads](<https://devfeed.tech/articles/validation-driven-deployment-of-llm-inference-platforms-for-agentic-workloads-145608.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25905>)

Author: Shai Gretz, Alon Halfon, Yoav Katz, Lena Dankin, Michal Jacovi, Rachel Tzoref-Brill, Kfir Toledo, Elron Bandel, Natan Ergas, Michal Malka, Pravein Govindan Kannan, Maroon Ayoub, Michal Shmueli-Scheuer

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [case-study](<https://devfeed.tech/tags/case-study.md>), [code-coverage](<https://devfeed.tech/tags/code-coverage.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-d](<https://devfeed.tech/tags/llm-d.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [platforms](<https://devfeed.tech/tags/platforms.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [trace](<https://devfeed.tech/tags/trace.md>), [traces](<https://devfeed.tech/tags/traces.md>), [validation](<https://devfeed.tech/tags/validation.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

The paper presents an end-to-end validation pipeline for agentic LLM serving, from small-scale testing through production. In its experiments, replaying captured traces reproduced per-session input-token load more consistently than live reruns: at the 95th percentile, replay stayed within 1.18x of recorded tokens, while live reruns reached 4.98x. In the measured vLLM configuration, live, replayed, and synthetic workloads each exercised Python serving paths the others did not. A rollout case study of four large MoE models on llm-d found about half the optimal concurrency and 60% of the throughput on captured workloads compared with public traces. The authors say they open-source tools and release a trace dataset.

### Source excerpt

arXiv:2611.25905v1 Announce Type: new Abstract: Enterprises are moving agentic workloads from closed APIs to their own open-weight model deployments. For this reason, lifecycle-wide validation, from small-scale testing through sizing and pre-production to production, becomes all the more important. Typically, rollouts are evaluated with ad hoc tools whose accuracy is rarely checked. We present an end-to-end evaluation pipeline for agentic LLM serving and empirically ground the design choices that connect its stages. Our experiments show that replaying captured traces reproduces the per-session load of the original agent sessions: at the 95th percentile replay stays within 1.18x of the recorded input tokens while live re-runs reach 4.98x, so a small set of recorded sessions yields a repeatable per-session token load. A code-coverage analysis of vLLM further shows that the evaluated live, replayed, and synthetic workloads are complementary in this configuration: each exercises Python serving paths not reached in the measured runs of the others. In a case study rolling out four large mixture-of-experts (MoE) models on llm-d, experiments show that the target models reached about half the optimal concurrency, and correspondingly about 60% of the throughput, on captured workloads compared with public traces. We open-source the relevant tools and release a trace dataset.

## Understanding MoE Decisions Beyond Gradients

DevFeed: [Understanding MoE Decisions Beyond Gradients](<https://devfeed.tech/articles/understanding-moe-decisions-beyond-gradients-145136.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26075>)

Author: Zhen Xu, Zitao Zhang, Kevin Zhao, Xuefeng Liu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [error](<https://devfeed.tech/tags/error.md>), [experts](<https://devfeed.tech/tags/experts.md>), [gemma4](<https://devfeed.tech/tags/gemma4.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [influence](<https://devfeed.tech/tags/influence.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [layers](<https://devfeed.tech/tags/layers.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [model](<https://devfeed.tech/tags/model.md>), [moe](<https://devfeed.tech/tags/moe.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [questions](<https://devfeed.tech/tags/questions.md>), [qwen3-6](<https://devfeed.tech/tags/qwen3-6.md>), [score](<https://devfeed.tech/tags/score.md>), [selection](<https://devfeed.tech/tags/selection.md>), [switches](<https://devfeed.tech/tags/switches.md>)

### AI overview

In mixture-of-experts language models, replacing an expert can change which experts later layers select, producing answer-score effects that gradients and second derivatives may miss. The proposed predictor uses the model's expert outputs and selection rules to estimate these additional effects and adds them to a second-order prediction. Across questions from three benchmarks, it reduced answer-score-change prediction error by 60% on Qwen3.6 and 47% on Gemma4 compared with a multistep baseline using gradients and second derivatives.

### Source excerpt

arXiv:2611.26075v1 Announce Type: cross Abstract: Understanding a language model's decision requires knowing how its internal computations affect its preference between answers. Gradient-based methods estimate this influence by predicting the effect of changing those computations. In mixture-of-experts (MoE) models, replacing one expert can change which experts later layers select. Those switches can alter the answer scores in ways that derivatives do not capture. We introduce a predictor that uses the model's own expert outputs and selection rules to estimate these extra effects, then adds them to a second-order prediction. Across questions from three benchmarks, our method reduces the error in predicting answer-score changes by 60\% on Qwen3.6 and 47\% on Gemma4, relative to a multistep baseline using gradients and second derivatives.

## Sparse Layers are Critical to Scaling Looped Language Models

DevFeed: [Sparse Layers are Critical to Scaling Looped Language Models](<https://devfeed.tech/articles/sparse-layers-are-critical-to-scaling-looped-language-models-143256.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.09165>)

Author: Ryan Lee, Jacob Biloki, Edward J. Hu, Jonathan May

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [layers](<https://devfeed.tech/tags/layers.md>), [loops](<https://devfeed.tech/tags/loops.md>), [memory](<https://devfeed.tech/tags/memory.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [quality](<https://devfeed.tech/tags/quality.md>), [routing](<https://devfeed.tech/tags/routing.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [standard](<https://devfeed.tech/tags/standard.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

The abstract reports that Looped-MoE models scale better than the standard transformer baseline, while dense looped models do not. It attributes the difference to routing divergence: shared layers activate different experts on successive passes, adding expressivity without additional parameters. It also finds that loop boundaries are better early-exit points than those in standard models because outputs converge earlier there. The authors report improved compute-quality trade-offs and potential memory and inference savings with minimal quality degradation; the supplied excerpt gives no quantitative results or experimental details.

### Source excerpt

arXiv:2605.09165v3 Announce Type: replace Abstract: Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries. However, looped models do not scale as favorably as standard transformers with unique layers. We compare standard and Mixture-of-Experts (MoE) transformers, with and without looping, and find two main results. First, we find Looped-MoE models scale better than the standard baseline while dense looped models do not. We trace this to routing divergence between loops: in Looped-MoE models, different experts are activated on each pass through the same shared layers, recovering expressivity without additional parameters. Our second finding is that looped models have better compute-quality trade-offs with early exits than standard models. Because each loop ends with the same layers that produce the final output, loop boundaries are superior exit points, as confirmed by earlier output convergence at these points. In sum, we provide a clear direction for scaling looped models: a Looped-MoE model with early exits can not only beat standard transformers at scale, but also enable significant memory and inference savings with minimal degradation in quality.

## AllotNet: an Interpretable Mixture-of-Experts Choice Model for Moral Judgement in Humanitarian Aid Allocation, Validated against a Planted Synthetic Population

DevFeed: [AllotNet: an Interpretable Mixture-of-Experts Choice Model for Moral Judgement in Humanitarian Aid Allocation, Validated against a Planted Synthetic Population](<https://devfeed.tech/articles/allotnet-an-interpretable-mixture-of-experts-choice-model-for-moral-judgement-in-humanitarian-aid-allocation-validated-against-a-planted-synthetic-population-142660.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23474>)

Author: Mark Nowak (Conflux Laboratory)

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>)

Tags: [bayesian](<https://devfeed.tech/tags/bayesian.md>), [budget](<https://devfeed.tech/tags/budget.md>), [choice](<https://devfeed.tech/tags/choice.md>), [code](<https://devfeed.tech/tags/code.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [encoder](<https://devfeed.tech/tags/encoder.md>), [failure](<https://devfeed.tech/tags/failure.md>), [gate](<https://devfeed.tech/tags/gate.md>), [generation](<https://devfeed.tech/tags/generation.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [java](<https://devfeed.tech/tags/java.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [planner](<https://devfeed.tech/tags/planner.md>), [python](<https://devfeed.tech/tags/python.md>), [queue](<https://devfeed.tech/tags/queue.md>), [rank](<https://devfeed.tech/tags/rank.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>)

### AI overview

AllotNet combines eight interpretable taste experts over seven ethical criteria with participant-context weighting to model allocation judgements elicited by the pre-registered, six-language Allot instrument. Its budget-constrained planner creates a review queue and never rejects anyone. On a planted synthetic population of 5.6 million sessions, the authors report that all 16 pre-specified checks passed, including criterion-ranking correlation of 1.000 and planner performance at 99.4% of an oracle. They also flag that two checks are weaker than their titles suggest and that a generation failure cost 260,712 seeds. The 23,550-parameter model has a 125 KB bundle and C11, Java, Rust, and Python runtimes. No real data has been used for training; a hierarchical Bayesian mixed logit is the registered comparator at the study's real scale.

### Source excerpt

arXiv:2611.23474v1 Announce Type: cross Abstract: Official development assistance fell by 23.1 per cent in 2025, and the decision of who is helped first when there is not enough for everyone is moving into software. A leading published machine-learning targeting system excluded 53 per cent of eligible households as deployed. If the rule of distribution ends up in code, it should carry the judgement of many cultures, and the machine should execute while a person judges. AllotNet is an interpretable mixture-of-experts conditional logit for the allocation judgements elicited by Allot, a pre-registered six-language instrument: eight linear taste experts over seven ethical criteria, weighted by a gate over participant context, with a damped residual, an encoder for household intake, an outcome head and a budget-constrained planner that builds a review queue and never rejects anyone. A model of human judgement has no answer key on real data, so we wrote one first: a planted synthetic population of 5.6 million sessions and 104 million responses (98 million usable choices), generated by the production code path from a ground truth hidden from the loss. All sixteen pre-specified checks passed: criterion ranking $\rho = 1.000$, hidden segments recovered (ARI 0.671), outcome head at 99 per cent of the noise ceiling, planner at 99.4 per cent of oracle. The caveats are reported with them: two checks are weaker than their titles, and a live-generation failure cost 260,712 seeds. The model has 23,550 parameters, is trained quantisation-aware for NVFP4 with its interpretable core in fp32 (welfare-rank parity $\rho = 0.9937$ at W4A16 on the shipped bundle; one parity threshold was relaxed post hoc), and ships as a 125 KB bundle with C11, Java, Rust and Python runtimes. Nothing has been trained on real data yet; at the study's real scale a hierarchical Bayesian mixed logit is the registered comparator.

## MECI: Multi-Element Collaborative Interaction for Multimodal Entity Linking

DevFeed: [MECI: Multi-Element Collaborative Interaction for Multimodal Entity Linking](<https://devfeed.tech/articles/meci-multi-element-collaborative-interaction-for-multimodal-entity-linking-142806.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24522>)

Author: Jie Peng, Yongxue Shan, Yongfu Zha, Xiaodong Wang

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Entity resolution](<https://devfeed.tech/topics/entity-resolution.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>)

Tags: [collaboration](<https://devfeed.tech/tags/collaboration.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [divide-and-conquer](<https://devfeed.tech/tags/divide-and-conquer.md>), [element](<https://devfeed.tech/tags/element.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [experts](<https://devfeed.tech/tags/experts.md>), [feature](<https://devfeed.tech/tags/feature.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generative](<https://devfeed.tech/tags/generative.md>), [global](<https://devfeed.tech/tags/global.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [knowledge-base](<https://devfeed.tech/tags/knowledge-base.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large](<https://devfeed.tech/tags/large.md>), [linking](<https://devfeed.tech/tags/linking.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>)

### AI overview

MECI addresses element dominance in multimodal entity linking with decoupled feature learning and hierarchical cross-modal interaction. Its Multi-view Experts Network uses KL-guided routing to coordinate specialized experts, while a modality-aware weighting network combines evidence across semantic levels. A final generative refinement stage uses large language models for zero-shot re-ranking. On WikiDiverse, RichpediaMEL, and WikiMEL, the abstract reports Hits@1 improvements of 1.95%, 7.30%, and 2.31%, respectively, over state-of-the-art baselines.

### Source excerpt

arXiv:2611.24522v1 Announce Type: cross Abstract: Multimodal Entity Linking (MEL) aims to disambiguate mentions in multimodal contexts by grounding them to specific entities in a knowledge base. A pivotal challenge in MEL is capturing multi-level correspondences: the semantic consistency between mention-entity pairs and the complementary correlations across modalities. However, existing methods often suffer from element dominance due to their reliance on coupled interactions or coarse global aggregations. In response, we propose the Multi-Element Collaborative Interaction (MECI) framework. First, to capture multi-element mention-entity correspondences, we develop a Multi-view Experts Network that leverages a "divide-and-conquer" strategy for decoupled feature learning to mitigate element dominance, supported by a KL-guided routing mechanism that governs expert specialization and collaboration. Furthermore, to model cross-modal complementary correlations, we propose a Hierarchical Multimodal Interaction Module, where a dynamic modality-aware weighting network refines interactions across hierarchical semantic levels, thereby integrating multi-granular evidence to counteract element dominance. Finally, we incorporate a generative semantic refinement stage that utilizes large language models for zero-shot re-ranking. Extensive experiments on WikiDiverse, RichpediaMEL, and WikiMEL show that MECI consistently outperforms state-of-the-art baselines, improving Hits@1 by 1.95%, 7.30%, and 2.31%, respectively.

## Soft Mixture of Quantum Experts for Hybrid Quantum-Classical Learning

DevFeed: [Soft Mixture of Quantum Experts for Hybrid Quantum-Classical Learning](<https://devfeed.tech/articles/soft-mixture-of-quantum-experts-for-hybrid-quantum-classical-learning-142834.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24798>)

Author: Valeria Repetto, Romina D'Aurizio, Piero Farinelli

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [circuits](<https://devfeed.tech/tags/circuits.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [experts](<https://devfeed.tech/tags/experts.md>), [feature](<https://devfeed.tech/tags/feature.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [learning](<https://devfeed.tech/tags/learning.md>), [mnist](<https://devfeed.tech/tags/mnist.md>), [qml](<https://devfeed.tech/tags/qml.md>), [quant-ph](<https://devfeed.tech/tags/quant-ph.md>), [quantum](<https://devfeed.tech/tags/quantum.md>), [quantum-circuits](<https://devfeed.tech/tags/quantum-circuits.md>), [quantum-classical](<https://devfeed.tech/tags/quantum-classical.md>), [quantum-machine-learning](<https://devfeed.tech/tags/quantum-machine-learning.md>), [results](<https://devfeed.tech/tags/results.md>), [routing](<https://devfeed.tech/tags/routing.md>), [selection](<https://devfeed.tech/tags/selection.md>), [training](<https://devfeed.tech/tags/training.md>), [weight](<https://devfeed.tech/tags/weight.md>)

### AI overview

Soft-MoQE avoids training quantum circuit parameters by combining fixed quantum feature maps through a classical gating network that selects weights for each input. On PCA-reduced MNIST parity, MNIST 10-class, and FashionMNIST tasks, it outperformed the best single expert, uniform expert averaging, and a linear baseline. Its performance was comparable, depending on the task, to a trainable quantum neural network, with quantum training cost reported as orders of magnitude lower. The learned routing combined a few experts per input and varied by class. These findings are limited to the three evaluated tasks.

### Source excerpt

arXiv:2611.24798v1 Announce Type: cross Abstract: Quantum Machine Learning (QML) has attracted increasing interest as a possible route to construct feature representations that are difficult to reproduce with classical models. However, near-term QML architectures based on trainable parametrized quantum circuits remain affected by limited hardware resources, noise, and severe optimization difficulties such as barren plateaus. In this work we propose Soft-MoQE, a Soft Mixture of Quantum Experts architecture that replaces variational quantum training with the input-dependent selection and combination of fixed quantum feature maps. The model is composed of a heterogeneous pool of non-trainable quantum circuits, each defining a distinct embedding, and a classical gating network that learns how to weight these embeddings for each data point. We benchmark Soft-MoQE on three PCA-reduced tasks: MNIST parity, MNIST 10-class, and FashionMNIST. Soft-MoQE outperforms the single best expert, the uniform average of all experts and a linear baseline, achieving task-dependent performance comparable to that of a trainable quantum neural network, without optimizing any quantum parameter and with a quantum training cost that is lower by orders of magnitude. The analysis of the class- conditioned expert usage shows that the learned routing combines a few experts for each input, with a selection that depends on the class. These results support the learned routing among fixed quantum circuits as a trainability-aware design principle for hybrid QML.

## FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment

DevFeed: [FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment](<https://devfeed.tech/articles/flexmoe-learning-one-for-all-multi-budget-expert-pruning-for-flexible-moe-deployment-143284.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.27866>)

Author: Fan Mo, Yuxuan Han, Geng Zhang, Wangbo Zhao, Yang You

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [acceleration](<https://devfeed.tech/tags/acceleration.md>), [actions](<https://devfeed.tech/tags/actions.md>), [activation](<https://devfeed.tech/tags/activation.md>), [budget](<https://devfeed.tech/tags/budget.md>), [co-design](<https://devfeed.tech/tags/co-design.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [experts](<https://devfeed.tech/tags/experts.md>), [ffn](<https://devfeed.tech/tags/ffn.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [flexible](<https://devfeed.tech/tags/flexible.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [learning](<https://devfeed.tech/tags/learning.md>), [llms](<https://devfeed.tech/tags/llms.md>), [masks](<https://devfeed.tech/tags/masks.md>), [memory](<https://devfeed.tech/tags/memory.md>), [moe](<https://devfeed.tech/tags/moe.md>), [pruning](<https://devfeed.tech/tags/pruning.md>)

### AI overview

FlexMoE aims to produce deployable MoE subnetworks at multiple parameter budgets from one pretrained model, avoiding separate compression runs for each budget. It ranks expert FFN channels to create a shared sliceable weight space, then learns expert-wise discrete retention actions under increasing cost pressure. A single fine-tune at a mid-range budget can recover performance and transfer to unseen budgets, reducing repeated recovery and separate checkpoint storage. The abstract reports gains over tested compression baselines; on Qwen2-57B-A14B, pruning 50% of routed-expert parameters retained about 99.8% of base performance without recovery fine-tuning. It also reports memory reduction and throughput gains, with kernel co-design enabling runtime budget switching without reloading checkpoints.

### Source excerpt

arXiv:2606.27866v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) language models scale model capacity through sparse expert activation, yet deployment still requires storing and serving all experts. Moreover, deployment budgets can vary substantially across devices, users, and workloads. Existing MoE compression methods typically optimize a separate compressed model for each target budget. We introduce FlexMoE, a post-training one-for-all framework that converts a large pretrained MoE model into a multi-budget family of smaller deployable subnetworks. FlexMoE first ranks expert FFN channels by importance to construct a sliceable weight space shared across budgets, and then learns expert-wise discrete retention actions under gradually increasing cost pressure, allowing a single action-training run to produce reliable pruning masks across budgets. Moreover, only a single mid-budget fine-tune can further recover degraded performance and yields a shared recovered model that transfers to other unseen budgets, avoiding repeated per-budget recovery and separated checkpoint storage. Across tested MoE LLMs with diverse architectures and scales, FlexMoE consistently outperforms strong MoE compression baselines. In particular, on Qwen2-57B-A14B, it retains ~99.8% of base performance while pruning 50% of routed-expert parameters even without recovery fine-tuning. For deployment, pruned subnetworks deliver real memory reduction and end-to-end throughput gains, while our kernel co-design enables runtime budget switching without checkpoint reloading and delivers substantial inference acceleration.

## MOLT: A Hugging Face-Native, Minimal, and Scalable Framework for Agentic Reinforcement Learning

DevFeed: [MOLT: A Hugging Face-Native, Minimal, and Scalable Framework for Agentic Reinforcement Learning](<https://devfeed.tech/articles/molt-a-hugging-face-native-minimal-and-scalable-framework-for-agentic-reinforcement-learning-142962.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.21653>)

Author: Jian Hu, Huiying Li, Hao Zhang, Binfeng Xu, Yifan Zhang, Shaokun Zhang, Hemil Desai, Michael Demoret, Pavlo Molchanov, Jan Kautz, Yi Dong

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [ai-coding-agent](<https://devfeed.tech/tags/ai-coding-agent.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [anthropic-sdk](<https://devfeed.tech/tags/anthropic-sdk.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [framework](<https://devfeed.tech/tags/framework.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [learning](<https://devfeed.tech/tags/learning.md>), [minimal](<https://devfeed.tech/tags/minimal.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [native](<https://devfeed.tech/tags/native.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [openai](<https://devfeed.tech/tags/openai.md>), [osworld](<https://devfeed.tech/tags/osworld.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>), [python](<https://devfeed.tech/tags/python.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [research](<https://devfeed.tech/tags/research.md>), [rl](<https://devfeed.tech/tags/rl.md>)

### AI overview

MOLT aims to make agentic RL code both modifiable and scalable: its approximately 9.2K-line Python RL path uses one training backend and one rollout engine, and loads Hugging Face checkpoints directly into a PyTorch FSDP2 trainer with tensor, expert, and context parallelism, avoiding model-family bridge code. The same training code is reported to scale from 4B to 753B models. Existing agents using OpenAI or Anthropic SDKs can be trained by redirecting their clients; MOLT handles agents that rewrite their context, runs training asynchronously, and stores trajectories in a distributed object store. Reported results include stable training of a 30B mixture-of-experts tool agent for 1,464 asynchronous updates, improvement by a visual computer-use agent on OSWorld, asynchronous training 2.40 times faster than synchronous training, and pipeline throughput at 92% of a Megatron-based stack. The abstract says its NeMo AutoModel trainer supports models from one to 2.8 trillion parameters.

### Source excerpt

arXiv:2607.21653v4 Announce Type: replace-cross Abstract: Agentic reinforcement learning (RL) research iterates quickly on agents, rewards, and learning algorithms, while policies have grown to hundreds of billions of parameters and beyond. Existing open frameworks force a choice: those that are easy to modify do not scale to the largest models, and those that scale rely on Megatron, which needs dedicated bridge code for every model family and several times more framework code. We present MOLT, an open-source framework built for research that is Hugging Face-native, minimal, and scalable. Its code is easy to modify: the entire RL path is approximately 9.2K lines of Python with one training backend and one rollout engine, so a researcher or an AI coding agent can read it in full and edit the code that actually runs. A Hugging Face checkpoint is loaded directly into a PyTorch FSDP2 trainer with tensor, expert, and context parallelism, so a new model family needs no bridge, and the same code scales from a 4B model to a 753B one. Existing OpenAI- and Anthropic-SDK agents are trained by redirecting their client, with automatic handling of agents that rewrite their own context; training is fully asynchronous; and trajectory data stay in a distributed object store. With MOLT, a 30B mixture-of-experts tool agent trains stably for 1,464 asynchronous updates, a visual computer-use agent improves on OSWorld, asynchronous training is 2.40 times faster than synchronous training, and pipeline throughput reaches 92% of a Megatron-based stack. Its trainer, NeMo AutoModel, trains models of one to 2.8 trillion parameters.

## Plug-and-Play Multi-Licensee Robust Watermarking for Mixture-of-Experts LLMs

DevFeed: [Plug-and-Play Multi-Licensee Robust Watermarking for Mixture-of-Experts LLMs](<https://devfeed.tech/articles/plug-and-play-multi-licensee-robust-watermarking-for-mixture-of-experts-llms-143137.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23838>)

Author: Jianshuo Liu, Kang G. Shin, Haining Wang

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [attribution](<https://devfeed.tech/tags/attribution.md>), [black-box](<https://devfeed.tech/tags/black-box.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [framework](<https://devfeed.tech/tags/framework.md>), [incremental](<https://devfeed.tech/tags/incremental.md>), [intellectual-property](<https://devfeed.tech/tags/intellectual-property.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [llms](<https://devfeed.tech/tags/llms.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [model-weights](<https://devfeed.tech/tags/model-weights.md>), [modules](<https://devfeed.tech/tags/modules.md>), [moe](<https://devfeed.tech/tags/moe.md>), [robust](<https://devfeed.tech/tags/robust.md>), [robustness](<https://devfeed.tech/tags/robustness.md>), [router](<https://devfeed.tech/tags/router.md>), [routing](<https://devfeed.tech/tags/routing.md>), [watermarking](<https://devfeed.tech/tags/watermarking.md>)

### AI overview

PPMoE embeds licensee-specific watermarks in MoE LLMs by selectively merging combinations of pre-trained incremental weight modules that steer router decisions, avoiding separate per-licensee models and additional training at distribution time. For attribution, white-box analysis inspects triggered routing decisions or merged modules; black-box analysis uses output-token likelihood ratios. The abstract reports experiments across five MoE models and seven datasets, with high attribution accuracy, preserved base-model utility, and robustness to adaptive watermark-removal attacks. It does not provide quantitative results or details of the attacks.

### Source excerpt

arXiv:2611.23838v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) architectures enable high-capacity large language models (LLMs) with high sparse computation, making them attractive for deployment on private hardware. However, distributing model weights exposes valuable intellectual property (IP) to leakage and unauthorized redistribution, creating an urgent need to both verify model ownership and trace leaked copies to individual licensees. Existing model-watermarking techniques are ill-suited for meeting this need because they are fragile under MoE's dynamic routing or require costly retraining to embed a distinct watermark for each licensee, thus making scalable distribution difficult. To address these challenges, we present PPMoE, the first plug-and-play multi-licensee watermarking framework for MoE LLMs. Rather than training a separate model per user, PPMoE trains a compact set of mergeable incremental weight modules that steer the MoE router toward distinct, mutually compatible routing patterns. During model distribution, the model owner selectively merges unique combinations of these pre-trained, mutually compatible modules into the base model on the fly, embedding a licensee-specific watermark without additional training. PPMoE supports reliable multi-licensee attribution under both white- and black-box threat models. In the white-box setting, attribution is done by inspecting triggered routing decisions or the merged weight modules responsible for these decisions, whereas in the black-box setting, attribution relies on analyzing output token likelihood ratios. Extensive experiments across five state-of-the-art MoE models, seven benchmark datasets, and varying number of licensees show that PPMoE achieves very high attribution accuracy while preserving base model utility and maintaining strong robustness to various adaptive watermark-removal attacks.

## MoMoVL: Modality-Aware Adaptive Compression for On-Device Sparsely-Activated Vision-Language Models

DevFeed: [MoMoVL: Modality-Aware Adaptive Compression for On-Device Sparsely-Activated Vision-Language Models](<https://devfeed.tech/articles/momovl-modality-aware-adaptive-compression-for-on-device-sparsely-activated-vision-language-models-143435.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23695>)

Author: Dongkeun Lee, Wonjun Lee

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [device](<https://devfeed.tech/tags/device.md>), [edge](<https://devfeed.tech/tags/edge.md>), [edge-devices](<https://devfeed.tech/tags/edge-devices.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [memory](<https://devfeed.tech/tags/memory.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [moe](<https://devfeed.tech/tags/moe.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [sparsity](<https://devfeed.tech/tags/sparsity.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [trade](<https://devfeed.tech/tags/trade.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>)

### AI overview

MoMoVL assigns mixed-precision bit-widths in MoE vision-language models using both per-expert activation profiles and each modality's quantization sensitivity. The approach accounts for differences between visual and text tokens, which can affect expert quantization. The abstract reports a better accuracy-compression trade-off than state-of-the-art mixed-precision baselines and up to 15.44% lower memory use than a uniform-precision baseline, with a modest accuracy drop. The abstract does not specify evaluation details or the size of that drop.

### Source excerpt

arXiv:2611.23695v1 Announce Type: cross Abstract: The rise of the agentic web has been accompanied by growing demand for on-device Vision-Language Models (VLMs) that perform diverse cross-modal web tasks with enhanced privacy and reduced latency. Mixture-of-Experts (MoE) models are widely adopted in modern VLMs for compute-efficient scalability, yet their multi-expert architecture poses deployment challenges due to substantial memory demands. Mixed-precision quantization is an effective compression technique that reduces bit-widths while adapting to the heterogeneous hardware budgets of edge devices, particularly well-suited for MoE due to its inherent structural sparsity, which causes imbalanced importance among experts. While prior work has applied quantization for efficient MoE inference at the edge, we point out that in MoE-based VLMs, the modality-specific characteristics of visual and text tokens further impact expert quantization. To address this issue, we present MoMoVL, a modality-aware adaptive compression framework for MoE-based VLMs that brings vision-language inference within reach of on-device resource budgets. Motivated by our empirical findings on modality-induced disparities across experts, MoMoVL jointly exploits the per-expert activation profile and per-modality quantization sensitivity to guide bit-width assignment. Experimental results show that MoMoVL improves the accuracy-compression trade-off relative to state-of-the-art mixed-precision baselines and reduces memory footprint by up to 15.44% with a modest drop in accuracy over its uniform-precision baseline. Our code and data are publicly available at https://github.com/sco-edge/MoMoVL.

## Systolic Linear Attention: Co-Designing Tile-Grouped Gated Delta Attention for TPUs

DevFeed: [Systolic Linear Attention: Co-Designing Tile-Grouped Gated Delta Attention for TPUs](<https://devfeed.tech/articles/systolic-linear-attention-co-designing-tile-grouped-gated-delta-attention-for-tpus-143344.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.06828>)

Author: Gagik Amirkhanyan, Nuojin Cheng

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [infini-attention](<https://devfeed.tech/topics/infini-attention.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Tensor Cores](<https://devfeed.tech/topics/tensor-cores.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [delta](<https://devfeed.tech/tags/delta.md>), [deltanet](<https://devfeed.tech/tags/deltanet.md>), [gated-deltanet](<https://devfeed.tech/tags/gated-deltanet.md>), [google](<https://devfeed.tech/tags/google.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [hbm](<https://devfeed.tech/tags/hbm.md>), [high-bandwidth-memory](<https://devfeed.tech/tags/high-bandwidth-memory.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [linear](<https://devfeed.tech/tags/linear.md>), [linear-attention](<https://devfeed.tech/tags/linear-attention.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [moe](<https://devfeed.tech/tags/moe.md>), [perplexity](<https://devfeed.tech/tags/perplexity.md>), [tensor-core](<https://devfeed.tech/tags/tensor-core.md>), [tpus](<https://devfeed.tech/tags/tpus.md>)

### AI overview

Systolic Linear Attention (SLA) reshapes gated-delta linear attention for Google TPUs' wide 256x256 matrix units. It widens keys to 256, shares keys and scalar write gates across grouped value heads to reduce repeated chunk solves, groups decay gates into four 64-channel tiles, and fuses exact FP32 chunk inversion with backward reuse in one on-chip kernel. On TPU v7x, two-stage SLA-R4 was 2.1-2.6x faster than open-source MaxText KDA and 1.4-2.4x faster than Tokamax KDA, using 49-61% less temporary HBM. In a 2,048-chip pre-training study of sparse MoE models from 6.5B to 63.1B nominal parameters, it trained 11-24% faster than MaxText KDA and 4-21% faster than Tokamax KDA. Perplexity matched KDA across three paired seeds at 12.5B and was within 3% at other tested scales.

### Source excerpt

arXiv:2611.06828v2 Announce Type: replace Abstract: Gated-delta linear attention layers such as Gated DeltaNet (GDN), Kimi Delta Attention (KDA), and GDN-2 are tuned for GPUs, where small tensor-core tiles and fast vector registers make unpacked heads, per-channel decay rescaling, and per-value-head chunk solves cheap. On Google TPUs, which concentrate compute in wide $256 \times 256$ systolic matrix units backed by a slower vector unit, direct ports leave half of each matrix tile padded (for $d_k = 128$ heads) and repeat decay rescaling and chunk solves across every value head. We present \textbf{Systolic Linear Attention (SLA)}, which reshapes gated delta attention around the matrix unit by widening keys to $256$ and sharing both keys and scalar write gates $\beta_t$ across grouped value heads ($R\times$ fewer chunk solves), grouping decay gates into four $64$-channel tiles, and fusing exact FP32 block-doubling chunk inversion and its backward reuse inside one on-chip kernel. With exact chunk inversion on TPU v7x, $2$-stage \texttt{SLA-R4} (four value heads per key head) runs $2.1\times$--$2.6\times$ faster than open-source MaxText KDA and $1.4\times$--$2.4\times$ faster than open-source Tokamax KDA, while using $49\%$--$61\%$ less temporary high-bandwidth memory (HBM). Across a $2{,}048$-chip TPU v7x pre-training study of sparse Mixture-of-Experts (MoE) models from $6.5\text{B}$ to $63.1\text{B}$ nominal parameters, SLA trains $11\%$--$24\%$ faster than MaxText KDA ($4\%$--$21\%$ faster than Tokamax KDA), matches KDA perplexity on three paired seeds at $12.5\text{B}$, and stays within $3\%$ at other scales.

## Inherited Adapter Experts for Replay-Free Domain-Incremental Learning

DevFeed: [Inherited Adapter Experts for Replay-Free Domain-Incremental Learning](<https://devfeed.tech/articles/inherited-adapter-experts-for-replay-free-domain-incremental-learning-142758.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24145>)

Author: Gyutae Oh, Hyunmin Kong, Jitae Shin

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [image recognition](<https://devfeed.tech/topics/image-recognition.md>)

Tags: [adapter](<https://devfeed.tech/tags/adapter.md>), [adapters](<https://devfeed.tech/tags/adapters.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [computer-vision](<https://devfeed.tech/tags/computer-vision.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [deepfake](<https://devfeed.tech/tags/deepfake.md>), [domain](<https://devfeed.tech/tags/domain.md>), [domains](<https://devfeed.tech/tags/domains.md>), [experts](<https://devfeed.tech/tags/experts.md>), [free](<https://devfeed.tech/tags/free.md>), [incremental](<https://devfeed.tech/tags/incremental.md>), [learning](<https://devfeed.tech/tags/learning.md>), [points](<https://devfeed.tech/tags/points.md>), [router](<https://devfeed.tech/tags/router.md>), [vit](<https://devfeed.tech/tags/vit.md>)

### AI overview

Lamarck addresses replay-free domain-incremental learning by strengthening per-domain experts rather than training a router: the abstract reports that a training-free Mahalanobis router costs the analyzed learners under 2.7 points versus always choosing the own-domain expert. Lamarck adds adapters after each deep ViT block, improving final accuracy by 4.4 points on CDDB-Hard, and initializes each new expert from its predecessor at no extra cost. This is intended to reduce the cost when an image is routed to a later expert, since domains share classes. Against twelve frozen-backbone baselines under one protocol, Lamarck ranks first on CDDB-Hard by 5.2 points, second on DomainNet and OfficeHome, and highest in mean accuracy.

### Source excerpt

arXiv:2611.24145v1 Announce Type: cross Abstract: Replay-free domain-incremental learning trains a model on the same classes in successive domains without keeping earlier images. Per-domain learners freeze a pre-trained ViT and train an additional expert for each new domain: the small set of parameters and the classification head learned on that domain. Test images carry no domain label, so a router selects one expert for each image. Recent work trains this router. However, a training-free Mahalanobis router costs the analyzed learners under 2.7 points of final accuracy relative to always selecting each image's own-domain expert. The experts, not this router, limit accuracy. Our learner, Lamarck, therefore uses this router and strengthens the experts. Its adapters, small trainable layers that correct every image token after each deep ViT block, add 4.4 points of final accuracy on the deepfake benchmark CDDB-Hard. In addition, each expert is initialized from its predecessor at no extra cost: the domains share their classes, so an image misrouted to a later expert loses less. Against twelve frozen-backbone baselines under one protocol, Lamarck is first on CDDB-Hard by 5.2 points, second on DomainNet and OfficeHome, and highest in mean accuracy.

## Training-Free Dynamic Upcycling of Expert Language Models

DevFeed: [Training-Free Dynamic Upcycling of Expert Language Models](<https://devfeed.tech/articles/training-free-dynamic-upcycling-of-expert-language-models-143249.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2603.29765>)

Author: Eros Fan\\\`i, O\\u{g}uzhan Ersoy

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [domains](<https://devfeed.tech/tags/domains.md>), [experts](<https://devfeed.tech/tags/experts.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [performance](<https://devfeed.tech/tags/performance.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [training](<https://devfeed.tech/tags/training.md>), [upcycling](<https://devfeed.tech/tags/upcycling.md>)

### AI overview

DUME combines dense language models trained on different domains into a single Mixture of Experts model without additional training. It uses a closed-form ridge regression solution, avoiding further optimization and allowing experts to be added or removed dynamically. Across causal language modeling and reasoning experiments with models from 115M to 9B parameters, the authors report that DUME outperforms baseline approaches. In reasoning, it averages 104% of the domain expert's performance. The resulting model can also be fine-tuned to improve performance on complex reasoning tasks.

### Source excerpt

arXiv:2603.29765v2 Announce Type: replace Abstract: Large Language Models (LLMs) have achieved remarkable performance on a wide range of specialized tasks, exhibiting strong problem-solving capabilities. However, training these models is prohibitively expensive, and they often lack domain-specific expertise because they rely on general knowledge datasets. Expertise finetuning can address this issue; however, it often leads to overspecialization, and developing a single multi-domain expert remains difficult due to diverging objectives. Furthermore, multitask training is challenging due to interference and catastrophic forgetting. Existing work proposes combining the expertise of dense models within a Mixture of Experts (MoE) architecture, although this approach still requires multitask finetuning. To address these issues, we introduce Dynamic Upcycling MoE (DUME), a novel approach that reuses dense experts trained on different domains to construct a unified MoE model. Our method builds a single multitask model that preserves the capabilities of the original dense experts without requiring additional training. DUME is both cost-efficient and scalable: by leveraging the closed-form solution of ridge regression, it eliminates the need for further optimization and enables experts to be added or removed dynamically. Our extensive experiments on different tasks (in causal language modeling and reasoning settings) with several model sizes (between 115M and 9B parameters) and architectures demonstrate that DUME consistently outperforms baseline approaches. In particular, in a reasoning setting, DUME can also surpass the dense expert model's performance in its domain, achieving 104% of the dense expert's performance on average. Finally, we also show that the DUME model can be fine-tuned to further improve performance in complex reasoning scenarios. Our code is available at: github.com/gensyn-ai/dume.

## Harvesting Expert-Parallel Bubbles for Fine-Tuning

DevFeed: [Harvesting Expert-Parallel Bubbles for Fine-Tuning](<https://devfeed.tech/articles/harvesting-expert-parallel-bubbles-for-fine-tuning-142803.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24496>)

Author: Oana Balmau, Saurav Banka, Jiaxuan Chen, Anne-Marie Kermarrec, Ramya Narasimha Prabhu, Rafael Pires, Mathis Randl, Martijn de Vos

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [a100](<https://devfeed.tech/tags/a100.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [bubble](<https://devfeed.tech/tags/bubble.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [colocation](<https://devfeed.tech/tags/colocation.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-os](<https://devfeed.tech/tags/cs-os.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [engine](<https://devfeed.tech/tags/engine.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [expert-parallel](<https://devfeed.tech/tags/expert-parallel.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>)

### AI overview

BubbleTea uses per-GPU idle periods created by uneven expert-parallel token routing to run fine-tuning alongside LLM serving. It predicts each GPU's available bubble at every layer and schedules short fine-tuning chunks that reuse serving weights; compiler passes derive the chunks from the served model without porting its architecture to a separate training engine. On three MoE models served across four A100 GPUs, BubbleTea delivered 1.9-6.2x the fine-tuning throughput of LLMStation and kept p90 inference latency within 10% of serving alone in 50 of 63 workloads, versus 44 for LLMStation. On bursty production traces, it trained 3.3x faster than LLMStation and 14x faster than a stock trainer under CUDA MPS, with p90 latency within 7% of serving alone.

### Source excerpt

arXiv:2611.24496v1 Announce Type: cross Abstract: GPUs are expensive, yet latency-critical large language model (LLM) inference leaves part of them idle. Colocation reclaims this capacity by running best-effort work, such as fine-tuning, alongside serving. Many leading LLMs now use a Mixture-of-Experts (MoE) architecture, which adds a new source of idle time. Under expert parallelism, MoE experts are spread across GPUs, and because the router sends uneven numbers of tokens to each GPU, lightly loaded GPUs wait for the most loaded one. Even after applying load balancing solutions, such bubbles still take 8--13% of prefill time. We present BubbleTea, the first colocation system that leverages expert-parallel bubbles for fine-tuning. Unlike existing colocation systems, which make one scheduling decision for all GPUs per forward pass, BubbleTea predicts each GPU's bubble at every layer from the router's token assignment, and fills it with short chunks of fine-tuning work that reuse the weights already loaded for serving. We also introduce compiler passes that automatically derive these chunks from the served LLM as-is, without porting each model architecture to a separate training engine. Our evaluation on three MoE models, served on four A100 GPUs, shows that fine-tuning throughput with BubbleTea is 1.9--6.2$\times$ faster than with LLMStation, a state-of-the-art colocation system. BubbleTea keeps p90 inference latency within 10% of serving alone in 50 of 63 workloads, compared to 44 for LLMStation. On bursty production traces, BubbleTea trains 3.3$\times$ faster than LLMStation and 14$\times$ faster than a stock trainer colocated under CUDA MPS, with p90 inference latency staying within 7% of serving alone.

## ElectrolyteFM: Unifying Electrolyte Property Prediction through Cross-Property Knowledge Learning

DevFeed: [ElectrolyteFM: Unifying Electrolyte Property Prediction through Cross-Property Knowledge Learning](<https://devfeed.tech/articles/electrolytefm-unifying-electrolyte-property-prediction-through-cross-property-knowledge-learning-141612.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.39340>)

Author: Jiaxin Yu, Shuo Wang, Peng Wang, Yongcai Wang, Deying Li

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Multitask learning](<https://devfeed.tech/topics/multitask-learning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>)

Tags: [cross](<https://devfeed.tech/tags/cross.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [electrolyte](<https://devfeed.tech/tags/electrolyte.md>), [error](<https://devfeed.tech/tags/error.md>), [features](<https://devfeed.tech/tags/features.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [learning](<https://devfeed.tech/tags/learning.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [multiple](<https://devfeed.tech/tags/multiple.md>), [pool](<https://devfeed.tech/tags/pool.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [properties](<https://devfeed.tech/tags/properties.md>), [property](<https://devfeed.tech/tags/property.md>), [router](<https://devfeed.tech/tags/router.md>), [separate](<https://devfeed.tech/tags/separate.md>), [sharing](<https://devfeed.tech/tags/sharing.md>), [sodium](<https://devfeed.tech/tags/sodium.md>), [target](<https://devfeed.tech/tags/target.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

ElectrolyteFM addresses uneven transfer between electrolyte-property prediction tasks: jointly learning tasks can help or hurt, and effects can be asymmetric. It keeps property-specific representations independent while a separately trained expert pool captures cross-property knowledge. A router selects shared information for each formulation and target property, and property-specific residual adapters turn it into prediction corrections. On Electrolyte12, it lowers mean normalized absolute error across 12 properties by 14.8% versus the strongest electrolyte-specific baseline. On an independent sodium-electrolyte dataset unseen during training, it lowers conductivity mean absolute error by 6.7% versus the best-performing baseline.

### Source excerpt

arXiv:2609.39340v2 Announce Type: replace Abstract: Electrolyte formulation design requires balancing multiple physicochemical properties, yet existing models often focus on a limited subset. Learning each property in isolation can overlook transferable chemical information, whereas indiscriminate sharing can introduce cross-property interference. Our directed transfer analysis shows that jointly learning two property prediction tasks can improve or degrade prediction relative to separate training, with asymmetric transfer effects between the tasks. We propose ElectrolyteFM, a unified multi-property prediction model which can more accurately predict multiple properties of each electrolyte by effectively identifying and utilizing property-specific features and knowledge shared across properties. More specifically, ElectrolyteFM learns property-specific representations independently and captures cross-property knowledge through a separately trained expert pool. A router selects relevant shared information for each formulation and target property, and property-specific residual adapters convert this information into corrections to the corresponding representation for prediction. Experiments on Electrolyte12 show that ElectrolyteFM reduces normalized mean absolute error averaged across 12 electrolyte properties by 14.8% relative to the strongest electrolyte-specific baseline. On an independent sodium-electrolyte dataset unseen during training, it reduces conductivity mean absolute error by 6.7% relative to the best-performing baseline.

## What RL on Expert-Built Tasks Teaches Nemotron 3.5 Super

DevFeed: [What RL on Expert-Built Tasks Teaches Nemotron 3.5 Super](<https://devfeed.tech/articles/what-rl-on-expert-built-tasks-teaches-nemotron-3-5-super-139598.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22161>)

Author: Sushant Mehta, Edwin Chen

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Nemotron](<https://devfeed.tech/topics/nemotron.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [nemotron](<https://devfeed.tech/tags/nemotron.md>), [nemotron-3-5](<https://devfeed.tech/tags/nemotron-3-5.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [rl](<https://devfeed.tech/tags/rl.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [tool-use](<https://devfeed.tech/tags/tool-use.md>)

### AI overview

Reinforcement learning on five expert-built task datasets improved NVIDIA's Nemotron 3.5 Super across the reported benchmarks, with gains from 4.5 points on IFBench to 13.8 on GDP_pdf (10.4% to 24.2%). The study trained separate rank-32 LoRA adapters with CISPO on Tinker, without a supervised stage, and compared each adapter with the base model under the same task settings. Reported gains include SWE-bench Pro V2 (86.9% to 92.1%), Terminal-Bench 2.1 (63.9% to 70.3%), MultiChallenge (58.3% to 66.6%), AutomationBench (14.8% to 27.5%) and Chartography (39.4% to 50.6%). The document adapter also gained 8 points on AutomationBench despite training without API tools. On newly solved tasks, adapters more often addressed observed base-model failures: locating data and rules before tool use, creating test or scratch files in coding tasks, retaining earlier constraints, measuring chart marks in code, and reading spreadsheets.

### Source excerpt

arXiv:2611.22161v1 Announce Type: cross Abstract: We post-train NVIDIA's Nemotron 3.5 Super (120B total, 12B active parameters) with reinforcement learning (RL) on five expert-built task datasets, one at a time: real-world software engineering, complex constraints, long-horizon tool use, professional chart understanding and enterprise document reasoning. Each run trains a fresh rank-32 LoRA adapter with CISPO on Tinker, with no supervised stage, and each adapter is compared with the base model on the same tasks, harness and settings. On twelve headline benchmarks, eight built by other groups and four by Surge AI, the gains range from +4.5 points on IFBench to +13.8 on GDP_pdf, which rises from 10.4% to 24.2%; SWE-bench Pro V2 rises from 86.9% to 92.1%, Terminal-Bench 2.1 from 63.9% to 70.3%, MultiChallenge from 58.3% to 66.6%, AutomationBench from 14.8% to 27.5% and Chartography from 39.4% to 50.6%. The document adapter, trained without API tools, also gains +8.0 points on AutomationBench. In attempts read side by side with the base model's on tasks each adapter newly solves, the base model's failures recur in a few modes, and each adapter does more often what the base model missed: the tool-use adapter finds the data and the rules before acting, the coding adapter writes its own test or scratch files on SWE-bench Pro V2, the constraints adapter carries earlier turns into its answers, the chart adapter measures the marks in code, and the document adapter reads AutomationBench's spreadsheets.

## Localized LoRA-MoE: Mixture of Low-Rank Experts With Adaptive Routing

DevFeed: [Localized LoRA-MoE: Mixture of Low-Rank Experts With Adaptive Routing](<https://devfeed.tech/articles/localized-lora-moe-mixture-of-low-rank-experts-with-adaptive-routing-139787.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.05114>)

Author: Babak Barazandeh, Subhabrata Majumdar, Vinay Prithyani, George Michailidis

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [lora](<https://devfeed.tech/topics/lora.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [adapter](<https://devfeed.tech/tags/adapter.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [lora](<https://devfeed.tech/tags/lora.md>), [moe](<https://devfeed.tech/tags/moe.md>), [peft](<https://devfeed.tech/tags/peft.md>), [routing](<https://devfeed.tech/tags/routing.md>), [spatial](<https://devfeed.tech/tags/spatial.md>)

### AI overview

Localized LoRA-MoE combines spatially localized low-rank adapters with context-conditioned routing to reduce interference when training streams mix operational contexts. Its Block-Wise variant uses one gate for the full grid; Cell-Wise assigns a separate gate to each grid cell. In the reported structured benchmarks, both beat static baselines at comparable adapter budgets, while Cell-Wise performed better when contexts affected different spatial regions. On six interleaved GLUE tasks, task-conditioned routing beat LoRA and static localized adapters at a fixed budget and matched per-task adapters using six times the budget. On Qwen2.5-1.5B across eight commonsense benchmarks, both variants beat static baselines; Cell-Wise scored 3.9 points above LoRA at matched active capacity.

### Source excerpt

arXiv:2607.05114v2 Announce Type: replace-cross Abstract: Parameter-efficient fine-tuning (PEFT) methods such as LoRA adapt large models through a single low-rank update. Such updates are susceptible to gradient interference: when a training stream interleaves multiple operational contexts, conflicting updates to shared adapter weights collapse the adapter into an unspecialized average. While recent methods introduce block-wise isolation to remedy this issue, their topologies are static and cannot adapt to context switching or localized sensor degradation. In this work, we introduce Localized LoRA-MoE, a framework that combines localized spatial blocking with context-conditioned expert routing. We propose and evaluate two architectural variants: Block-Wise LoRA-MoE (centralized routing), which modulates the entire structural grid through a single context-conditioned gate, and Cell-Wise LoRA-MoE (decentralized routing), which assigns an independent gate to every coordinate cell of the matrix grid. Across three structured transformation benchmarks--a high-dimensional SVD matrix simulation, context-switched transformations over real tabular data, and spatial reconstruction under simulated sensor occlusion--both variants outperform static baselines under comparable adapter budgets. Both variants perform similarly when context shifts the grid uniformly, while cell-wise routing is strictly better when contexts are spatially heterogeneous. When fine-tuning a language model on six interleaved GLUE tasks, task-conditioned routing at a fixed adapter budget outperforms LoRA and static localized adapters, matching per-task adapters with 6x budgets. At decoder scale on Qwen2.5-1.5B, both routed variants outperform static baselines across eight commonsense benchmarks, with cell-wise routing achieving +3.9 points over LoRA at matched active capacity.

## The Hidden Cost of MoE Diffusion Decoding: Profiling and Fixing the Sampler

DevFeed: [The Hidden Cost of MoE Diffusion Decoding: Profiling and Fixing the Sampler](<https://devfeed.tech/articles/the-hidden-cost-of-moe-diffusion-decoding-profiling-and-fixing-the-sampler-139920.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21852>)

Author: Harshit Samrat, Ramakrishna Paramahamsa Kompella

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Model Optimizer](<https://devfeed.tech/topics/model-optimizer.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [moe](<https://devfeed.tech/tags/moe.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [profiling](<https://devfeed.tech/tags/profiling.md>), [sampling](<https://devfeed.tech/tags/sampling.md>)

### AI overview

After weight quantization, the per-token sampling and stopping pipeline over the full logits canvas becomes the largest decode-time cost in the profiled 26B MoE diffusion model on an NVIDIA GB10. Diffusion decoding handles S times as many logit entries per output token as autoregressive decoding, where S is the number of denoising steps. The authors remove redundant or fusable work in two tiers: a bit-exact tier matching the reference output byte for byte, and an accuracy-parity tier with a fused entropy kernel and Gumbel-max sampling. Combined with a W4A16 expert kernel, the stack achieves 2.4-2.7x end-to-end speedup over bf16 at unchanged GSM8K, HumanEval, and MATH accuracy. A second diffusion MoE shows a 28.8% sampler share and 3.3x fused W4A16 speedup at accuracy parity; the sampler cost falls to 3.6% on a dense diffusion model. The excerpt attributes this cost to MoE diffusion after quantization makes the backbone cheaper.

### Source excerpt

arXiv:2611.21852v1 Announce Type: cross Abstract: Masked diffusion language models (dLLMs) decode by iteratively denoising a full canvas of tokens, so every forward pass runs a per-token sampling and stopping pipeline over the entire canvas-by-vocabulary logits tensor. We profile single-user decoding of a 26B-parameter diffusion Mixture-of-Experts model on a bandwidth-bound edge accelerator (NVIDIA GB10) and report a result that is easy to miss: after weight quantisation, this logits pipeline, not the expert GEMMs, is the single largest consumer of decode time. Unlike autoregressive decoding, where sampling touches one position's logits per committed token, diffusion decoding processes S times as many logit entries per output token, where S is the number of denoising steps. We show this pipeline is largely redundant or fusable and remove it in two lossless tiers: a bit-exact tier whose output is byte-identical to the reference decoder, and an accuracy-parity tier using a fused single-pass entropy kernel and Gumbel-max sampling. Combined with a W4A16 expert kernel, the full stack reaches 2.4-2.7 times end-to-end speedup over bf16 at unchanged benchmark accuracy on GSM8K, HumanEval, and MATH, on a single 273 GB/s device. We confirm the finding on a second, independently built diffusion MoE (LLaDA2.0-mini: pipeline 28.8% of decode, fused W4A16 3.3 times at accuracy parity) and show it disappears on a dense diffusion model (3.6%). The cost is specific to MoE diffusion, where quantisation makes the backbone cheap. Code and quantised weights are released.

## Communicate the Change: Efficient Expert Parallelism via Activation Compression

DevFeed: [Communicate the Change: Efficient Expert Parallelism via Activation Compression](<https://devfeed.tech/articles/communicate-the-change-efficient-expert-parallelism-via-activation-compression-139983.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22215>)

Author: Fengyuan Liu, Jay Gala, Siva Reddy, Dzmitry Bahdanau

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [quantization](<https://devfeed.tech/topics/quantization.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [collective](<https://devfeed.tech/tags/collective.md>), [communication](<https://devfeed.tech/tags/communication.md>), [compression](<https://devfeed.tech/tags/compression.md>), [compute](<https://devfeed.tech/tags/compute.md>), [critical](<https://devfeed.tech/tags/critical.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-inference](<https://devfeed.tech/tags/distributed-inference.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [ethernet](<https://devfeed.tech/tags/ethernet.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [fewer](<https://devfeed.tech/tags/fewer.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [hidden-states](<https://devfeed.tech/tags/hidden-states.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>)

### AI overview

EchoMoE reduces communication overhead in expert-parallel MoE inference by sending a compressed difference between the current hidden state and a periodically refreshed earlier-layer reference, rather than sending each hidden state in full. The abstract reports 58% lower inter-node communication volume and 22.6% higher throughput, with average performance within 3.8 percentage points across 18 benchmarks. On bandwidth-constrained Ethernet, throughput improves by up to 2.38x. The modification is designed to retrofit pretrained MoE models, requiring about 8,500x fewer training tokens than full pretraining; the communicated representations can also be quantized without measurable quality loss. The authors report an implementation for pretrained models and an inference integration for vLLM.

### Source excerpt

arXiv:2611.22215v1 Announce Type: cross Abstract: Scaling has driven the success of large language models (LLM), with sparse mixture-of-experts (MoE) architectures enabling greater parameter capacity without a proportional increase in computation. At large scales, MoE models rely on expert parallelism to distribute experts across devices. Each MoE layer requires two synchronous collective communication steps during inference: dispatching hidden states to the activated experts and combining the resulting expert outputs. As compute throughput has outpaced interconnect bandwidth, these synchronous collectives have become a major bottleneck to MoE scaling and efficiency. We address this challenge with EchoMoE, a lightweight architecture modification that retrofits existing MoE models, requiring approximately 8,500x fewer training tokens than full pretraining. Motivated by the observation that transformer blocks update the residual stream incrementally, we use a hidden state from an earlier layer as a reference and communicate only the compressed difference between it and the current state, rather than communicating each hidden state in full. EchoMoE periodically refreshes these reference states and transmits only the compressed differences synchronously, reducing the traffic on latency-critical paths without significant computational overhead. EchoMoE reduces inter-node communication volume by 58% and improves throughput by 22.6%, while preserving average performance within 3.8 percentage points across 18 benchmarks. EchoMoE further improves inference throughput by up to 2.38x over bandwidth-constrained Ethernet, paving the way for wide-area distributed inference. EchoMoE also tolerates quantization of the communicated representations without measurable quality loss. We release an efficient implementation for applying EchoMoE to pretrained MoE models, together with an inference integration for vLLM.

[Next page](<https://devfeed.tech/topics/mixture-of-experts-moe.md?cursor=WyIyMDI2LTExLTE4VDA1OjAwOjAwKzAwOjAwIiwgIjIwYWM3N2JjLWE3MTQtNDNhNC05NTc5LTU2YzEyNjEyYTFhMCJd>)