# Model Merging Does Not Require Uniform LoRA Ranks

DevFeed: [Model Merging Does Not Require Uniform LoRA Ranks](<https://devfeed.tech/articles/model-merging-does-not-require-uniform-lora-ranks-118473.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.06486>)

Author: Zirong Xie, Zainab Afolabi, Nikita Kozodoi, Jack Butler

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [lora](<https://devfeed.tech/topics/lora.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>)

Tags: [adapter](<https://devfeed.tech/tags/adapter.md>), [components](<https://devfeed.tech/tags/components.md>), [conflict-resolution](<https://devfeed.tech/tags/conflict-resolution.md>), [coupling](<https://devfeed.tech/tags/coupling.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [domain](<https://devfeed.tech/tags/domain.md>), [experts](<https://devfeed.tech/tags/experts.md>), [lora](<https://devfeed.tech/tags/lora.md>), [model](<https://devfeed.tech/tags/model.md>), [rank](<https://devfeed.tech/tags/rank.md>)

## AI overview

The study finds that LoRA experts can use different ranks without hurting averaging or utility-based selection, with multi-domain accuracy gains of up to 1.8 percentage points. Conflict-resolution methods are less consistent and can lose up to 4 points. The reported mechanism is component selection: magnitude-based methods keep a fixed share of pooled spectral components, often dropping the instruction adapter with its smaller singular values; IFEval then falls by 16-25 points. Selecting components by their effect on loss retains all three experts and keeps accuracy nearly flat across an 8x rank range (0.7-point variation, versus 6.7 points for TIES at 4B scale). The results support per-domain rank choices when the merge operator does not select components by magnitude.

## Source excerpt

arXiv:2611.06486v1 Announce Type: cross Abstract: Model merging combines separately fine-tuned experts into a single model without retraining. Standard practice trains every expert at the same LoRA rank, which couples the rank choice across experts even though each domain has its own best rank. We show this coupling is a property of the merge method, not of merging. We train 36 experts on three domains (mathematics, instruction following, multilingual) at four ranks and three model scales, then merge with six methods at uniform ranks and at each domain's own best rank. Giving experts different ranks costs averaging and utility-based selection nothing, and gains them up to 1.8 pp of multi-domain accuracy, while the conflict-resolution methods are inconsistent and can lose up to 4 pp. The split has a simple cause. Merging keeps only a fixed share of the pooled spectral components, and methods that choose them by size discard many that would have reduced loss. They also tend to discard the same expert: the instruction adapter has the smallest singular values, so it is dropped almost entirely and IFEval falls by 16 to 25 pp. Choosing components by their effect on loss instead keeps all three experts, and leaves accuracy nearly flat across an 8$\times$ rank range (0.7 pp, against 6.7 pp for TIES at 4B scale). Rank can therefore be chosen per expert whenever the merge operator does not select by magnitude.