# Scaling Laws for Multimodal Data Mixtures

DevFeed: [Scaling Laws for Multimodal Data Mixtures](<https://devfeed.tech/articles/scaling-laws-for-multimodal-data-mixtures-116775.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.14423>)

Author: Aditi Khandelwal, Ayush K. Tarun, Yixuan Xu, Imanol Schlag, Golnoosh Farnadi, Siva Reddy, Luke Zettlemoyer, Antoine Bosselut

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [audio](<https://devfeed.tech/tags/audio.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [experts](<https://devfeed.tech/tags/experts.md>), [laws](<https://devfeed.tech/tags/laws.md>), [mixing](<https://devfeed.tech/tags/mixing.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [moe](<https://devfeed.tech/tags/moe.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>), [text](<https://devfeed.tech/tags/text.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [training](<https://devfeed.tech/tags/training.md>), [vision](<https://devfeed.tech/tags/vision.md>)

## AI overview

The study derives scaling laws for mixing text, vision, and audio in multimodal pretraining, using MoE models across 268 experiments, 457M-8.3B parameters, and up to 150 billion tokens. It reports modality-specific compute optima: audio favors scaling model parameters over tokens, while text and vision benefit from balancing model size and data. Repeating vision or audio data beyond 2x brings negligible gains. Cross-modal interference is asymmetric: vision strongly interferes with audio, while audio has negligible interference with text and vision. These findings are based on the described MoE experiments.

## Source excerpt

arXiv:2610.14423v2 Announce Type: replace-cross Abstract: Frontier AI systems are increasingly natively multimodal, jointly pretrained on multiple modalities, such as text, vision, and audio. However, the scientific understanding of the multimodal data mixtures used to pretrain these systems --- how much of each modality to mix, and at what potential interference cost to the others --- is largely absent or gatekept. In this work, we derive data mixing scaling laws for three modalities: text, vision, and audio. Multimodal Large Language Models (MLLMs) increasingly adopt Mixture-of-Experts (MoE) architectures, as MoEs scale efficiently and partially limit cross-modal interference. We therefore use MoEs to conduct 268 experiments ranging from 457M to 8.3B parameters, on up to 150 billion tokens. Our work leads to multiple actionable insights into multimodal data mixing for training MLLMs. First, compute optima are modality-specific: audio modality is model-heavy, favoring scaling parameters over tokens, while text and vision require balanced scaling of model size and data. Second, repeating vision or audio data beyond 2x yields negligible benefits. Finally, cross-modal interactions are highly asymmetric: vision strongly interferes with audio, whereas audio's interference with learning text and vision is negligible. Overall, we provide a principled foundation for understanding the multimodal data mixtures needed to train frontier MLLMs.