# scaling laws

Scaling laws describe how neural language model performance and training behavior vary with model size, dataset size, and compute.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Optimizer-dependent training dynamics converge to the same one-third optimal data scaling

DevFeed: [Optimizer-dependent training dynamics converge to the same one-third optimal data scaling](<https://devfeed.tech/articles/optimizer-dependent-training-dynamics-converge-to-the-same-one-third-optimal-data-scaling-144944.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.37745>)

Author: Hyunseok Lee, Mihir Basil, Yizhou Liu, Jeff Gore

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [learning](<https://devfeed.tech/tags/learning.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

In an online teacher-student model, the training-step loss decay depends on the optimizer, while the optimally tuned loss scales as dataset size D to the −1/3 across the seven optimizers studied. The authors split loss into radial norm growth and tangential alignment. For SGD, both decay exponents are near 1/3; for Adam, they differ substantially (about 0.48 and 0.08). This difference makes the optimal learning rate independent of D for SGD but decreasing with D for Adam. A stochastic-dynamics analysis relates the channels through 2αr + αt = 1, yielding the shared optimal data exponent. The abstract reports that optimizers change the prefactor and per-step learning dynamics, but not the tuned loss rate per sample.

### Source excerpt

arXiv:2609.37745v2 Announce Type: replace Abstract: Neural scaling, in which loss falls as a power law with training, is central to large language models, and one recent proposal is that a $1/3$ exponent emerges from learning peaked distributions. That account describes SGD, but models in practice are trained with adaptive optimizers. Here we separate two exponents the $1/3$ account does not distinguish: how fast the loss falls with training steps along a single run, and how fast the optimally tuned loss falls with dataset size $D$. We show that the first, a dynamic exponent, is optimizer-specific while the second, an optimal data exponent, converges to $1/3$ across optimizers. In an online teacher-student model we decompose the loss into norm growth (radial) and alignment toward the teacher direction (tangential), each decaying as a power law with dynamic exponents $\alpha_{r}$ and $\alpha_{t}$. Under SGD, both are close to $1/3$, so the data exponent is also $1/3$ across different learning rates. Under Adam the two separate: $\alpha_{r} \simeq 0.48$ but $\alpha_{t} \simeq 0.08$. Since the total loss is minimized when these two parts are balanced, the optimal learning rate is optimizer-dependent: $D$-independent for SGD but falls with $D$ for Adam. Yet tuned to that optimum, the loss returns to $D^{-1/3}$ for both. A stochastic-dynamics analysis explains why: the optimizers can trade decay speed between the two channels, but they all fall on a single dynamic exponent relation, $2\alpha_{r}+ \alpha_{t} = 1$, which fixes the optimal data exponent at $1/3$. Across seven optimizers, including Muon, the measured exponents are consistent with this relation, and the optimal-loss envelopes agree with $D^{-1/3}$ across them. The optimizer sets how fast a model learns per step; tuned optimally, it changes the prefactor but not the rate at which loss falls per sample.

## Learning from Transients: Linear RNN Scaling Laws in Stationary Prediction

DevFeed: [Learning from Transients: Linear RNN Scaling Laws in Stationary Prediction](<https://devfeed.tech/articles/learning-from-transients-linear-rnn-scaling-laws-in-stationary-prediction-144933.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.27964>)

Author: Ziyan Chen, Linhe Zheng, Peilin Liu, Zhongzhu Zhou, Ding-Xuan Zhou

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [approximation](<https://devfeed.tech/tags/approximation.md>), [batch](<https://devfeed.tech/tags/batch.md>), [bias](<https://devfeed.tech/tags/bias.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [gradient-descent](<https://devfeed.tech/tags/gradient-descent.md>), [laws](<https://devfeed.tech/tags/laws.md>), [learning](<https://devfeed.tech/tags/learning.md>), [linear](<https://devfeed.tech/tags/linear.md>), [model](<https://devfeed.tech/tags/model.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [recurrent](<https://devfeed.tech/tags/recurrent.md>), [risk](<https://devfeed.tech/tags/risk.md>), [rnn](<https://devfeed.tech/tags/rnn.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>)

### AI overview

In a stylized Gaussian-sketched linear recurrent teacher-student model, independently initialized trajectories create transients that can excite weak spectral directions. For full-batch gradient descent with stable deterministic learning rates, the authors derive matching scaling laws for stationary approximation, bias, variance, and total risk from dependent trajectory data. If initialization decays spectrally at least as fast as innovations, the leading variance rate depends on the product of trajectory count and length. With heavier-tailed initialization, transients can improve length-dependent finite-time bias and, at fixed transition count, variance; the stationary oracle shift remains controlled by representation approximation. Under token and compute caps, constant-length trajectories achieve the optimal risk scaling order within the theorem-controlled range. Synthetic experiments offer finite-scale evidence in this stylized setting.

### Source excerpt

arXiv:2609.27964v3 Announce Type: replace Abstract: Does the organization of a fixed number of transitions into independent trajectories affect a fixed stationary prediction task? We study a Gaussian-sketched linear recurrent teacher--student model, where independently initialized training trajectories create transients that can excite weak spectral directions. For full-batch gradient descent with stable deterministic learning rates, we establish matching stationary approximation, bias, variance, and total-risk scaling laws from dependent trajectory data. When initialization decays at least as fast spectrally as the innovations, trajectory count and length enter the leading variance rate through their product. With a heavier initialization tail, transient excitation can improve the trajectory-length-dependent finite-time bias and, at fixed transition count, variance, while the stationary oracle shift remains controlled by representation approximation. Under token and compute caps, constant-length trajectories attain the optimal risk scaling order within the theorem-controlled range. Synthetic experiments provide finite-scale evidence in this stylized model.

## GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives

DevFeed: [GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives](<https://devfeed.tech/articles/groundingpi-a-grounding-foundation-model-towards-physical-intelligence-with-visual-primitives-144951.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.39601>)

Author: Qize Yu, Lianrui Fan, Boyu Chen, Jiaqi Liang, Xini Ding, Yue Chen, Zetian Song, Yuran Wang, Yi Zou, Kaixuan Wang, Tianxing Chen, Wenxuan Song, Bohan Zhou, Mingleyang Li, Siqiao Huang, Yuqi Ye, Caigao

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [Physical AI](<https://devfeed.tech/topics/physical-ai.md>), [Robot Manipulation](<https://devfeed.tech/topics/robot-manipulation.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [autonomous-driving](<https://devfeed.tech/tags/autonomous-driving.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [foundation-model](<https://devfeed.tech/tags/foundation-model.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [ocr](<https://devfeed.tech/tags/ocr.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>), [robotic-manipulation](<https://devfeed.tech/tags/robotic-manipulation.md>)

### AI overview

GroundingPI is a 4B model that represents points and boxes as quantized coordinates in a shared vocabulary, trained with multimodal and spatial pretraining, supervised fine-tuning, and GRPO reinforcement learning. The abstract reports an average of 73.68% across 34 grounding benchmarks, ahead of the larger GPT-6 Astra at 71.54%. As a visual backbone, it outperformed evaluated alternatives in all four out-of-distribution settings across RoboTwin 2.0 and RoboCasa-GR1, with gains up to 24.8%; on RoboCasa-GR1, using 50% of demonstrations beat baselines using 75%. Its mean open-loop L2 error on nuScenes was 0.296 m. The authors report that greater pretraining scale improved grounding and manipulation, dense grounding benefited both, and OCR may aid perceptual learning.

### Source excerpt

arXiv:2609.39601v3 Announce Type: replace-cross Abstract: Precise grounding matters. It specifies which object is the target and where that object is, even in clutter and for tiny objects, and it has to be fast enough for closed-loop control. Yet vision-language-action (VLA) and world-action models (WAMs) take perception from general-purpose vision-language and video-generation backbones, which still fail in these settings. We introduce GroundingPI, a 4B grounding foundation model that generates points and boxes as quantized coordinates in a shared vocabulary. Training combines multimodal and spatial pretraining, supervised fine-tuning, and reinforcement learning with GRPO, using supervision from public datasets and dedicated data engines. Against 44 baselines across 34 grounding benchmarks spanning 11 perceptual capabilities, GroundingPI establishes a new state of the art, averaging 73.68%, above the larger GPT-6 Astra (71.54%). As a downstream visual backbone, GroundingPI improves performance on robotic manipulation and autonomous driving. Across RoboTwin 2.0 and RoboCasa-GR1, it outperforms all evaluated backbones in all four out-of-distribution settings, with relative gains of up to 24.8% over the strongest baseline. On RoboCasa-GR1, GroundingPI trained with 50% of the demonstrations outperforms those baselines trained with 75%. Its mean open-loop L2 error on nuScenes is 0.296 m. We systematically analyze GroundingPI's pretraining in scale and data composition. Grounding quality and downstream manipulation improve with pretraining scale, with generalization tracking perceptual gains most closely. Analyzing the data recipe across these 11 perceptual capabilities shows dense grounding's substantial benefits for both, and OCR's potential as a catalyst for perceptual learning. These results support grounding as a perceptual foundation, and dedicated perceptual pretraining as a promising direction for foundation models of physical intelligence.

## Sparse Layers are Critical to Scaling Looped Language Models

DevFeed: [Sparse Layers are Critical to Scaling Looped Language Models](<https://devfeed.tech/articles/sparse-layers-are-critical-to-scaling-looped-language-models-143256.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.09165>)

Author: Ryan Lee, Jacob Biloki, Edward J. Hu, Jonathan May

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [layers](<https://devfeed.tech/tags/layers.md>), [loops](<https://devfeed.tech/tags/loops.md>), [memory](<https://devfeed.tech/tags/memory.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [quality](<https://devfeed.tech/tags/quality.md>), [routing](<https://devfeed.tech/tags/routing.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [standard](<https://devfeed.tech/tags/standard.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

The abstract reports that Looped-MoE models scale better than the standard transformer baseline, while dense looped models do not. It attributes the difference to routing divergence: shared layers activate different experts on successive passes, adding expressivity without additional parameters. It also finds that loop boundaries are better early-exit points than those in standard models because outputs converge earlier there. The authors report improved compute-quality trade-offs and potential memory and inference savings with minimal quality degradation; the supplied excerpt gives no quantitative results or experimental details.

### Source excerpt

arXiv:2605.09165v3 Announce Type: replace Abstract: Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries. However, looped models do not scale as favorably as standard transformers with unique layers. We compare standard and Mixture-of-Experts (MoE) transformers, with and without looping, and find two main results. First, we find Looped-MoE models scale better than the standard baseline while dense looped models do not. We trace this to routing divergence between loops: in Looped-MoE models, different experts are activated on each pass through the same shared layers, recovering expressivity without additional parameters. Our second finding is that looped models have better compute-quality trade-offs with early exits than standard models. Because each loop ends with the same layers that produce the final output, loop boundaries are superior exit points, as confirmed by earlier output convergence at these points. In sum, we provide a clear direction for scaling looped models: a Looped-MoE model with early exits can not only beat standard transformers at scale, but also enable significant memory and inference savings with minimal degradation in quality.

## OpenGNS: An Open Dataset of the Gradient Noise Scale across Vision, Language, and Diffusion Models

DevFeed: [OpenGNS: An Open Dataset of the Gradient Noise Scale across Vision, Language, and Diffusion Models](<https://devfeed.tech/articles/opengns-an-open-dataset-of-the-gradient-noise-scale-across-vision-language-and-diffusion-models-142979.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24331>)

Author: Marcel Aach, Jiangtao Wang, Stefan Kesselheim, Andreas Lintermann

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [batch](<https://devfeed.tech/tags/batch.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [diffusion-models](<https://devfeed.tech/tags/diffusion-models.md>), [diffusion-transformers](<https://devfeed.tech/tags/diffusion-transformers.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [language](<https://devfeed.tech/tags/language.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [measurements](<https://devfeed.tech/tags/measurements.md>), [metric](<https://devfeed.tech/tags/metric.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [models](<https://devfeed.tech/tags/models.md>), [mup](<https://devfeed.tech/tags/mup.md>), [noise](<https://devfeed.tech/tags/noise.md>), [open](<https://devfeed.tech/tags/open.md>), [scale](<https://devfeed.tech/tags/scale.md>), [size](<https://devfeed.tech/tags/size.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

OpenGNS releases over 400 Gradient Noise Scale measurement trajectories for ResNets, GPTs, and Diffusion Transformers, covering 11M-1B parameter models trained with muP and dense batch-size and learning-rate sweeps. The authors report that at equivalent loss, GNS remains stable across model sizes under muP, behaving like a consistent "temperature of training." Its relationship to Critical Batch Size appears task-dependent: GNS tends to overestimate CBS for classification and underestimate it for generative tasks, with the direction holding across common CBS definitions.

### Source excerpt

arXiv:2611.24331v1 Announce Type: cross Abstract: The Gradient Noise Scale (GNS) metric measures the signal-to-noise ratio of gradients in stochastic gradient descent and, as shown by McCandlish et al. (2018), can be used to approximate the Critical Batch Size (CBS) during neural network training. Despite its appearance in technical reports of frontier models, no open dataset of GNS measurements across modern architectures exists, limiting systematic study of the metric. We close this gap by releasing OpenGNS, a dataset of more than 400 GNS measurement trajectories spanning three domains: image classification with ResNets, autoregressive language modeling with Generative Pre-trained Transformers (GPTs), and image synthesis with Diffusion Transformers (DiTs), covering models from 11M to 1B parameters trained under Maximal Update Parameterization (muP) with dense batch-size and learning-rate sweeps. Building on this dataset, we study how the GNS depends on hyperparameters, scale, and training progress, and how it relates to the CBS. We find that the GNS behaves as a consistent "temperature of training", remaining stable across model sizes under muP at equivalent loss, and our results suggest task-dependent biases in its relation to the CBS: a tendency to overestimate the CBS for classification and to underestimate it for generative tasks, consistent in direction across common CBS definitions. The dataset is available at https://huggingface.co/datasets/olympique-marcel/OpenGNS

## General progress substitutes for specific training data in language models

DevFeed: [General progress substitutes for specific training data in language models](<https://devfeed.tech/articles/general-progress-substitutes-for-specific-training-data-in-language-models-142838.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24844>)

Author: Mark Marosi, Peter Antal

Published: 2026-11-19T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [capabilities](<https://devfeed.tech/tags/capabilities.md>), [chess](<https://devfeed.tech/tags/chess.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [data](<https://devfeed.tech/tags/data.md>), [delay](<https://devfeed.tech/tags/delay.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [models](<https://devfeed.tech/tags/models.md>), [progress](<https://devfeed.tech/tags/progress.md>), [training](<https://devfeed.tech/tags/training.md>), [training-data](<https://devfeed.tech/tags/training-data.md>)

### AI overview

Across experiments on chess, Othello, molecular strings, bracket languages and web text, general training progress partly substituted for withheld skill-specific data. Larger models' delays on withheld skills were predicted from smaller models. A chess model trained without check-response examples ranked legal responses nearly as well as one trained on them; when withheld data was later added, models acquired the skill using a fraction of the data needed from initialization, with that fraction shrinking as training progressed. An equally difficult control rule was learned more slowly, and a programming language withheld from a 7-billion-parameter model's training text was regained similarly. The findings suggest filtering delays capabilities, but the data needed to acquire them later may decline as models improve.

### Source excerpt

arXiv:2611.24844v1 Announce Type: cross Abstract: Language models acquire their capabilities from training data, and developers withhold data during training to keep models from learning capabilities that could be misused. How long this delays a capability, and how much data is needed to acquire it later, is unknown. Here we show that the general progress a model makes on the rest of its training data substitutes for the data that was withheld. We withheld the training signal for individual skills in models of chess, Othello, molecular strings, bracket languages and web text. The delay of a withheld skill in larger models was predicted from smaller ones. A chess model never explicitly trained to respond to check ranked the legal responses almost as well as a model that was. When models were then trained on the withheld data, they acquired the skill with a small fraction of the data needed from initialization, and this fraction decreased as the models progressed. A control rule of equal difficulty was learned much more slowly. A programming language withheld from the training text of a 7-billion-parameter model was regained in the same way. Data filtering therefore delays a capability, and the data required to regain it decreases as models improve.

## Test-Time Symmetrization: Scaling Laws for Model-Agnostic Learning under Structure

DevFeed: [Test-Time Symmetrization: Scaling Laws for Model-Agnostic Learning under Structure](<https://devfeed.tech/articles/test-time-symmetrization-scaling-laws-for-model-agnostic-learning-under-structure-142743.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24032>)

Author: Behrooz Tahmasebi, Melanie Weber

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [approximation](<https://devfeed.tech/tags/approximation.md>), [chain-of-thought](<https://devfeed.tech/tags/chain-of-thought.md>), [choice](<https://devfeed.tech/tags/choice.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [graph](<https://devfeed.tech/tags/graph.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [log](<https://devfeed.tech/tags/log.md>), [logical](<https://devfeed.tech/tags/logical.md>), [model-agnostic](<https://devfeed.tech/tags/model-agnostic.md>), [random](<https://devfeed.tech/tags/random.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>), [study](<https://devfeed.tech/tags/study.md>)

### AI overview

Test-time symmetrization improves inference by using forward passes on equivalent rewritings of an input and aggregating the answers. The paper proves that repeated sampling from one fixed representation cannot substitute for rewriting, while O(log |G|) random rewritings can recover the information from all |G| rewritings; this logarithmic rate is optimal in general. The same rewritings support simultaneous approximation across models, decoding rules and sampled reasoning traces. Experiments with four Qwen, Phi and Llama models on graph reasoning, logical deduction and multiple-choice questions found rewritings substantially more accurate than repeated sampling at the same forward-pass count when answers depended on input form. The abstract presents this as a way to enforce symmetries at inference time when they are not built into the model.

### Source excerpt

arXiv:2611.24032v1 Announce Type: new Abstract: Test-time compute has emerged as a powerful way to improve the accuracy of language models at inference time, with scaling laws quantifying how accuracy improves with additional forward passes. Existing scaling laws, however, focus on repeated sampling from a fixed prompt, even though many tasks admit multiple equivalent representations of the same underlying problem. A graph does not depend on how its nodes are numbered, and a program does not depend on the names of its variables, yet language models can change their answers under such rewritings. In this paper, we study test-time symmetrization, which uses forward passes on rewritings of the input and aggregates the resulting answers over the group $G$ of rewritings. We prove that no amount of sampling and repeated decoding from a single representation can replace rewriting, while only $O(\log |G|)$ random rewritings suffice to recover the information provided by all $|G|$ rewritings. The same set of $O(\log |G|)$ rewritings provides simultaneous approximation across models, decoding rules, and sampled reasoning traces, and the logarithmic scaling is optimal in general. We validate the theory with experiments on Qwen2.5-7B, Phi-4-mini, Llama-3.1-8B, and Qwen2.5-14B, with chain-of-thought reasoning, on graph reasoning, logical deduction, and multiple-choice question answering. At the same number of forward passes, spending them on rewritings of the input is substantially more accurate than spending them on repeated sampling from the original prompt whenever the model's answer depends on how the input is written, and the number of rewritings needed grows only logarithmically with the number of possible rewritings, as the theory predicts. Our results provide a principled way to enforce symmetries at inference time when they cannot be built into the model, which we call model-agnostic symmetrization.

## Towards Reliable Scaling Laws for Tabular Foundation Models

DevFeed: [Towards Reliable Scaling Laws for Tabular Foundation Models](<https://devfeed.tech/articles/towards-reliable-scaling-laws-for-tabular-foundation-models-142768.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24214>)

Author: Johannes Hog, Neeratyoy Mallik, Steven Adriaensen, Jan Hendrik Metzen, Tom Viering, Danny Stoll, Frank Hutter

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>)

Tags: [compute-allocation](<https://devfeed.tech/tags/compute-allocation.md>), [compute-optimal](<https://devfeed.tech/tags/compute-optimal.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [distributions](<https://devfeed.tech/tags/distributions.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [features](<https://devfeed.tech/tags/features.md>), [foundation-models](<https://devfeed.tech/tags/foundation-models.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [model-performance](<https://devfeed.tech/tags/model-performance.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>), [synthetic-data](<https://devfeed.tech/tags/synthetic-data.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

For TabPFNv2-style tabular foundation models, a well-fitting scaling law may still give unreliable guidance on unseen real-world datasets: choosing configurations from too few evaluation datasets produces systematically over-optimistic estimates. The abstract reports that synthetic and real-world scaling behavior differs, and that compute-optimal scaling requires increasing synthetic pretraining dataset sizes as model size and training compute grow. Raising the maximum sampled rows and features improves performance and makes scaling trends more predictable. These findings are presented as guidance for more efficient training using nanoTabPFN's open-source pipeline.

### Source excerpt

arXiv:2611.24214v1 Announce Type: cross Abstract: Scaling laws are widely used to predict foundation model performance and guide compute allocation. However, tabular foundation models (TFMs) differ from conventional foundation models in that they are pretrained on virtually unlimited synthetic data from configurable distributions, while ultimately being evaluated on real-world datasets. We investigate whether scaling laws for TabPFNv2-style models reliably extrapolate to larger compute budgets and generalize across evaluation settings. Our analysis reveals differences between synthetic and real-world scaling behavior and shows that accurately fitting a scaling law does not necessarily yield reliable prescriptions on unseen datasets. In particular, selecting configurations using too few evaluation datasets leads to systematically over-optimistic performance estimates. We further demonstrate that compute-optimal scaling requires expanding the range of synthetic pretraining dataset sizes alongside model size and training compute. Increasing the maximum numbers of rows and features sampled during pretraining improves performance and produces more predictable scaling trends. These insights guide the training of more efficient TabPFNv2-like models, using the open-source training pipeline of nanoTabPFN.

## When Does Reallocating a Sampling Budget Pay? Instance-wise Allocation for Neural Routing Solvers

DevFeed: [When Does Reallocating a Sampling Budget Pay? Instance-wise Allocation for Neural Routing Solvers](<https://devfeed.tech/articles/when-does-reallocating-a-sampling-budget-pay-instance-wise-allocation-for-neural-routing-solvers-142565.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.27917>)

Author: Jinhyung Bae

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [budget](<https://devfeed.tech/tags/budget.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [math-oc](<https://devfeed.tech/tags/math-oc.md>), [neural](<https://devfeed.tech/tags/neural.md>), [routing](<https://devfeed.tech/tags/routing.md>), [sampling](<https://devfeed.tech/tags/sampling.md>)

### AI overview

For neural routing solvers, reallocating a fixed sampling budget helps most when instance difficulty varies widely within a workload. On the training distribution, it cuts total optimality gap by at most 2.5%, leaving uniform allocation close to optimal. In workloads mixing out-of-distribution instances, it cuts the gap by 11.8-13.3% for the two solvers that degrade under the shift, and has almost no effect for the solver that does not. The study uses exact best-of-k curves, since random-ordering estimates overstate gains and a split-sample evaluation understates them. A power-law allocation fitted using 20 charged probe samples per instance recovers 88-93% of the uncharged optimum; successive halving performs worse. Applying 2-opt removes most of the remaining gap at negligible cost, leaving reallocation gains of only a few hundredths of a percent of tour length.

### Source excerpt

arXiv:2609.27917v2 Announce Type: replace-cross Abstract: Neural routing solvers sample many solutions per instance and keep the best, with the same number of samples for every instance. We measure how much a non-uniform split of the same total budget is worth and which property of a workload decides it. Best-of-k curves estimated by averaging random orderings of stored samples overstate the gain, and deciding on one half of the samples while evaluating on the other understates it; we use exact best-of-k curves, a U-statistic over all orderings. For three pretrained solvers (POMO, the Attention Model and Sym-NCO) on the traveling salesman problem, reallocation reduces the total optimality gap by at most 2.5% on the training distribution, so uniform allocation is close to optimal there. On workloads that mix in out-of-distribution instances it reduces the gap by 11.8-13.3% (1.1-1.8% of tour length) for the two solvers that degrade under the shift, and almost nothing for the one that does not. Across two heterogeneity mechanisms, three solvers, two confirmation collections and budgets of 25 to 200 samples per instance, the gain is predicted by the spread of difficulty within a workload, not its average. Allocating on a power-law curve fitted to twenty charged probe samples per instance recovers 88-93% of the uncharged optimum; successive halving does worse. A 2-opt local search, however, removes most of the gap at negligible cost, after which reallocation adds a few hundredths of a percent of tour length.

## A Differential Theory of Quality-Resource Trade-offs for AI Model Selection: The Multiresource Frontier and Its Minimax Price

DevFeed: [A Differential Theory of Quality-Resource Trade-offs for AI Model Selection: The Multiresource Frontier and Its Minimax Price](<https://devfeed.tech/articles/a-differential-theory-of-quality-resource-trade-offs-for-ai-model-selection-the-multiresource-frontier-and-its-minimax-price-143377.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23324>)

Author: Dmitry Ignatov

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Model Routing](<https://devfeed.tech/topics/model-routing.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [cs-it](<https://devfeed.tech/tags/cs-it.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [math-it](<https://devfeed.tech/tags/math-it.md>), [math-oc](<https://devfeed.tech/tags/math-oc.md>), [minimax](<https://devfeed.tech/tags/minimax.md>), [quality](<https://devfeed.tech/tags/quality.md>), [resource](<https://devfeed.tech/tags/resource.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [theory](<https://devfeed.tech/tags/theory.md>), [trade](<https://devfeed.tech/tags/trade.md>)

### AI overview

The paper proposes a differential framework for comparing AI model quality-resource frontiers. It represents each frontier through the relation between effective active resource channels (squared slope over curvature) and remaining log-error depth, separating intrinsic shape from the resource axis's scale and origin. Complete monotonicity and divisibility yield inequalities and frontier bounds. For N resources with one active channel each, it derives an operating rule of N_eff = 1 in delivered units and minimax prices λ*_i = α_i/(N+1), with worst-case regret ln(N+1) nats. The authors say this enables resource-cost comparisons without hand-chosen exchange rates. They report evaluating the theory on 117 published frontiers and implementing nn-rank to derive prices from measured quality and cost, rank candidates, and report supported choices and minimax regret.

### Source excerpt

arXiv:2611.23324v1 Announce Type: cross Abstract: Comparing AI models across resource budgets yields quality-resource frontiers relating error to latency, parameter count, training compute, and energy. Offset power laws summarize such frontiers, but their fitted parameters reveal neither the price-independent geometry nor the manner in which trade-offs extend across several resource channels. This paper addresses both gaps by developing a differential theory in which this structure is made explicit. Each frontier admits a price-independent plane-curve representation relating the effective number of active resource channels, $N_{\mathrm{eff}} = s^2/\kappa$ (squared slope over curvature), to the remaining log-error depth, $\Delta S = \ln(\varepsilon/\varepsilon_{\infty})$. This representation separates intrinsic shape from the scale and origin of the resource axis. Complete monotonicity and divisibility yield differential inequalities and frontier bounds. When each of $N$ resources contributes one active channel, the theory gives the parameter-free operating rule $N_{\mathrm{eff}} = 1$ in delivered units and derives minimax resource prices $\lambda^{\star}_i = \alpha_i/(N+1)$ with an exact worst-case regret of $\ln(N+1)$ nats. The results make it possible to interpret measured frontiers and compare resource costs without hand-chosen exchange rates. The theory is evaluated on 117 published frontiers and implemented in the open-source package nn-rank, which derives prices from measured quality and cost, ranks candidates, and reports supported choices and minimax regret: https://github.com/ABrain-One/nn-rank

## Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models

DevFeed: [Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models](<https://devfeed.tech/articles/bigger-isn-t-always-memorizing-early-stopping-overparameterized-diffusion-models-143219.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2505.16959>)

Author: Alessandro Favero, Antonio Sclocchi, Matthieu Wyart

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [diffusers](<https://devfeed.tech/topics/diffusers.md>), [Occam's razor machine learning](<https://devfeed.tech/topics/occam-s-razor-machine-learning.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [competition](<https://devfeed.tech/tags/competition.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [diagram](<https://devfeed.tech/tags/diagram.md>), [diffusion-models](<https://devfeed.tech/tags/diffusion-models.md>), [grammar](<https://devfeed.tech/tags/grammar.md>), [image](<https://devfeed.tech/tags/image.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [language](<https://devfeed.tech/tags/language.md>), [noise](<https://devfeed.tech/tags/noise.md>), [optimize](<https://devfeed.tech/tags/optimize.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [probabilistic](<https://devfeed.tech/tags/probabilistic.md>), [random](<https://devfeed.tech/tags/random.md>), [regression](<https://devfeed.tech/tags/regression.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

In highly overparameterized diffusion models, generalization on natural data emerges during training before memorization begins; fitting training loss completely can lead to reproducing training examples. Across image and language models, the authors report that memorization time scales with dataset size, consistent with a kernel-regression bound on fitting the empirical score at low noise. They interpret generalization and memorization as competing time scales. A random-rule probabilistic context-free grammar model shows a similar pattern: deeper rules are acquired over time, and the authors characterize the generalization cost of stopping early. Their results support dataset-size-scaled early stopping as a way to improve generalization while avoiding memorization, with possible relevance to hyperparameter transfer and privacy-sensitive applications. The evidence provided is an abstract, so it does not specify a concrete stopping rule or experimental details.

### Source excerpt

arXiv:2505.16959v3 Announce Type: replace Abstract: Diffusion probabilistic models have become a cornerstone of modern generative AI, yet the mechanisms underlying their generalization remain poorly understood. In fact, if these models were perfectly minimizing their training loss, they would just generate data belonging to their training set, i.e., memorize, as empirically found in the overparameterized regime. We revisit this view by showing that, in highly overparameterized diffusion models, generalization in natural data domains is progressively achieved during training before the onset of memorization. Our results, ranging from image to language diffusion models, systematically support the empirical law that memorization time is proportional to the dataset size, consistent with a kernel-regression bound on the time required to fit the empirical score at low noise. Generalization vs. memorization is then best understood as a competition between time scales. We show that this phenomenology is recovered in diffusion models learning a simple probabilistic context-free grammar with random rules, where generalization corresponds to the hierarchical acquisition of deeper grammar rules as training time grows, and the generalization cost of early stopping can be characterized. We summarize these results in a phase diagram. Overall, our results support that a principled early-stopping criterion - scaling with dataset size - can effectively optimize generalization while avoiding memorization, with direct implications for hyperparameter transfer and privacy-sensitive applications.

## What's in a Tree ? Hierarchical $n$-gram models as controlled approximators of language

DevFeed: [What's in a Tree ? Hierarchical $n$-gram models as controlled approximators of language](<https://devfeed.tech/articles/what-s-in-a-tree-hierarchical-n-gram-models-as-controlled-approximators-of-language-140119.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23093>)

Author: Aritra Das, Francesco Cagnetta, Darshil Doshi, Victor V. Albert, Maissam Barkeshli, Andrey Gromov

Published: 2026-11-18T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [composition](<https://devfeed.tech/tags/composition.md>), [cond-mat-dis-nn](<https://devfeed.tech/tags/cond-mat-dis-nn.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [conditional-probability](<https://devfeed.tech/tags/conditional-probability.md>), [context](<https://devfeed.tech/tags/context.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [generative-models](<https://devfeed.tech/tags/generative-models.md>), [gram](<https://devfeed.tech/tags/gram.md>), [language](<https://devfeed.tech/tags/language.md>), [models](<https://devfeed.tech/tags/models.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pruning](<https://devfeed.tech/tags/pruning.md>), [routing](<https://devfeed.tech/tags/routing.md>), [sparsity](<https://devfeed.tech/tags/sparsity.md>), [tensors](<https://devfeed.tech/tags/tensors.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

Tree-grams are autoregressive n-gram models that compose order-2 and order-3 tensors to form conditional probabilities. Their context window is controlled by n and their capacity by rank d, providing interpretable controls over each. The authors report that tree-grams outperform existing n-gram baselines and remain competitive with deep transformers. As generative models, they make narrower-model hyperparameter transfer under μP more robust when tuning on simpler tree-gram-generated data with moderate n and d. The abstract also reports that entropy and correlation range shape post-fine-tuning performance gaps, and that routing across candidate topologies becomes sparse enough to prune up to 95% without significant performance change.

### Source excerpt

arXiv:2611.23093v1 Announce Type: cross Abstract: Hierarchical models have been proposed as synthetic settings for studying language. However, their relevance to realistic language corpora remains underexplored. In this work, we introduce tree-grams, autoregressive generative $n$-gram models that build the conditional probability tensor $p(x_n \mid x_{1:n-1})$ through a hierarchical composition of order-2 and order-3 tensors. $n$ controls the maximal context window, whereas a rank hyperparameter $d$ controls model capacity, yielding an interpretable and systematically scalable family of approximators of language. We show how tree-grams outperform existing $n$-gram baselines, while remaining competitive with deep transformer-based language models. We demonstrate the utility of such controlled approximations by using tree-grams as generative models. In particular, we show that hyperparameter transfer under $\mu$P can be made more robust when narrower models are tuned on simpler data, generated by tree-grams with moderate $n$ and $d$. In addition, we show that entropy and effective range of correlations control performance gap after fine-tuning. This is because tree-grams allow us to explicitly control horizon and capacity. When allowed to route across candidate topologies, we observe emergent sparsity in tree-grams. Such concentrated routing permits pruning upto 95\% of topologies without significant change in performance.

## Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks

DevFeed: [Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks](<https://devfeed.tech/articles/sharp-feature-learning-transitions-and-bayes-optimal-neural-scaling-laws-in-extensive-width-networks-141557.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.10395>)

Author: Minh-Toan Nguyen, Jean Barbier

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Deep learning](<https://devfeed.tech/topics/deep-learning.md>)

Tags: [bayes](<https://devfeed.tech/tags/bayes.md>), [cond-mat-dis-nn](<https://devfeed.tech/tags/cond-mat-dis-nn.md>), [cond-mat-stat-mech](<https://devfeed.tech/tags/cond-mat-stat-mech.md>), [cs-it](<https://devfeed.tech/tags/cs-it.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [feature](<https://devfeed.tech/tags/feature.md>), [laws](<https://devfeed.tech/tags/laws.md>), [learning](<https://devfeed.tech/tags/learning.md>), [math-it](<https://devfeed.tech/tags/math-it.md>), [networks](<https://devfeed.tech/tags/networks.md>), [neural](<https://devfeed.tech/tags/neural.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>), [sharp](<https://devfeed.tech/tags/sharp.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

In a high-dimensional teacher-student setting, the study derives Bayes-optimal learning behavior for a one-hidden-layer teacher whose width grows linearly with input dimension. Its fixed-point equations predict sequential, discontinuous jumps in recoverability of hierarchical features as data increases. The resulting effective width--the number of learnable features at a given data budget--connects two error regimes: n^(1/(2β)-1) during feature learning and n^-1 during refinement, both summarized by ε^BO = Θ(k_c d/n), for hierarchy exponent β > 1/2. Empirically, a student trained with Adam near the effective width attains these scaling laws with a small algorithmic gap. The analysis relies on a heuristic leave-one-out argument validated numerically; the abstract does not provide further methodological details.

### Source excerpt

arXiv:2605.10395v2 Announce Type: replace-cross Abstract: We study the information-theoretic limits of learning a one-hidden-layer teacher network with hierarchical features from noisy queries, in the context of knowledge transfer to a smaller student model. We work in the high-dimensional regime where the teacher width $k$ scales linearly with the input dimension $d$ -- a setting that captures large-but-finite-width networks and has only recently become analytically tractable. Using a heuristic leave-one-out decoupling argument, validated numerically throughout, we derive asymptotically sharp characterizations of the Bayes-optimal generalization error and individual feature overlaps via a system of closed fixed-point equations. These equations reveal that feature learnability is governed by a sequence of sharp phase transitions: as data grows, teacher features become recoverable sequentially, each through a discontinuous jump in overlap. This sequential acquisition underlies a precise notion of \textit{effective width} $k_c$ -- the number of learnable features at a given data budget $n$ -- which unifies two distinct scaling regimes: a feature-learning regime in which the Bayes-optimal generalization error $\varepsilon^{\rm BO}$ scales as $ n^{1/(2\beta)-1}$, and a refinement regime in which it scales as $n^{-1}$, where $\beta>1/2$ is the exponent of the power-law feature hierarchy. Both laws collapse to the single relation $\varepsilon^{\rm BO}=\Theta(k_c d/n)$. We further show empirically that a student trained with \textsc{Adam} near the effective width $k_c$ achieves these optimal scaling laws (up to a small algorithmic gap), and provide an information-theoretic account of the associated scaling in model size.

## An Open-Source Training Dataset for Foundation Models for Hyperparameter Optimization

DevFeed: [An Open-Source Training Dataset for Foundation Models for Hyperparameter Optimization](<https://devfeed.tech/articles/an-open-source-training-dataset-for-foundation-models-for-hyperparameter-optimization-141568.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.23417>)

Author: Aaron Klein, Herilalaina Rakotoarison, Luca Thale-Bombien, David Salinas

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [foundation-models](<https://devfeed.tech/tags/foundation-models.md>), [hyperparameter-optimization](<https://devfeed.tech/tags/hyperparameter-optimization.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [training](<https://devfeed.tech/tags/training.md>), [training-dataset](<https://devfeed.tech/tags/training-dataset.md>)

### AI overview

BBO-Pile is an open-source dataset of over 500,000 optimization trajectories across 3,095 hyperparameter optimization tasks and global optimization problems, addressing the shortage of large, public pre-training data. The authors train foundation models ranging from 2M to 80M parameters and 200M to 2B training tokens, and study scaling with compute. They report that large-scale pre-training can effectively imitate hyperparameter optimization methods. The supplied text does not detail specific performance comparisons or limitations.

### Source excerpt

arXiv:2605.23417v2 Announce Type: replace Abstract: Most hyperparameter optimization methods require extensive tuning of their own hyperparameters, often limiting their ability to generalize across different benchmark families. Foundation models for hyperparameter optimization that learn optimization principles from a large collection of optimization trajectories offer a promising alternative, with the potential to outperform manually designed methods across diverse problem classes. However, prior work as either relied on non-public datasets or on purely synthetic data, limiting reproducibility and generalization to real-world problems. As a result, progress in this area has been constrained by the lack of large-scale, real-world, publicly available pre-training data. We introduce BBO-Pile, the first open-source dataset comprising over 500K optimization trajectories evaluated across 3,095 different hyperparameter optimization tasks and global optimization problems for different optimizers, which represents by far the largest public dataset for this task. Using this dataset, we train a family of foundation models at multiple scales, ranging from 2M to 80M parameters and from 200M to 2B training tokens, and study their scaling behavior with respect to compute. Our results demonstrate that large-scale pre-training is a viable and effective approach to imitate hyperparameter optimization methods, paving the way for future research in this direction.

## Scaling Laws for Vision-Language Fusion

DevFeed: [Scaling Laws for Vision-Language Fusion](<https://devfeed.tech/articles/scaling-laws-for-vision-language-fusion-141769.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22351>)

Author: Seng Pei Liew, Teppei Suzuki, Seiya Tokui

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [fusion](<https://devfeed.tech/tags/fusion.md>), [laws](<https://devfeed.tech/tags/laws.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

The paper proposes a scaling law for vision-language pre-training that separates modality-specific learning trajectories from shifts caused by competition between image and text data. It reports that the law predicts validation losses across data mixture ratios, training steps, and model sizes. Under a fixed data budget, early fusion preserves text performance while late fusion favors vision performance; the reported trade-off tilts increasingly toward early fusion as model size grows. The abstract says experiments support the law's ability to capture these dynamics and inform fusion choices.

### Source excerpt

arXiv:2611.22351v1 Announce Type: new Abstract: Vision-language models (VLMs) have emerged as central foundation models combining multimodal perception with text comprehension. While VLMs are traditionally built via late fusion by continued pre-training an off-the-shelf language model on multimodal (image-text) data, mixing modalities from the start of pre-training (early fusion) has recently gained traction. However, the fundamental trade-offs between these paradigms remain poorly understood, resulting in expensive and heuristic-driven training recipes. In this paper, we systematically analyze the learning dynamics of these multimodal pre-training processes, showing that validation losses during vision-language fusion can be modeled by separating the base learning trajectory, where different data types contribute at distinct rates, from a distribution shift driven by the competition between modalities. We derive a scaling law that unifiedly describes these dynamics, enabling accurate loss predictions across mixture ratios, training steps, and model sizes. Crucially, our law characterizes the trade-off between early and late fusion: under a fixed data budget, early fusion preserves text performance while late fusion favors vision performance, yielding a trade-off that increasingly tilts toward early fusion as model size scales. Extensive experiments confirm that our scaling law reliably captures multimodal learning dynamics and governs fusion decisions.

## Amortizing Scaling Law Construction Costs

DevFeed: [Amortizing Scaling Law Construction Costs](<https://devfeed.tech/articles/amortizing-scaling-law-construction-costs-139814.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.05016>)

Author: Abhash Kumar Jha, Diana Alexandra Onu\\c{t}u, Neeratyoy Mallik, Swagatam Haldar, Sam Laing, Niccol\\\`o Ajroldi, Shiwei Liu, Joaquin Vanschoren, Aaron Klein

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>)

Tags: [acquisition](<https://devfeed.tech/tags/acquisition.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [budget](<https://devfeed.tech/tags/budget.md>), [collection](<https://devfeed.tech/tags/collection.md>), [compute](<https://devfeed.tech/tags/compute.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [construction](<https://devfeed.tech/tags/construction.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [design](<https://devfeed.tech/tags/design.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [exhaustive](<https://devfeed.tech/tags/exhaustive.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [foundation-models](<https://devfeed.tech/tags/foundation-models.md>), [framework](<https://devfeed.tech/tags/framework.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [full](<https://devfeed.tech/tags/full.md>), [law](<https://devfeed.tech/tags/law.md>), [scaling](<https://devfeed.tech/tags/scaling.md>)

### AI overview

The framework aims to reduce the cost of fitting scaling laws by collecting the best-loss frontier across compute scales instead of training an exhaustive grid. It formulates configuration selection as Bayesian optimization, progressively expands the acquisition compute budget, and uses surrogate-fantasized evaluations to reconstruct the broader grid. The authors report that these steps can closely match fits from a full dense grid while reducing compute by up to 10-100x.

### Source excerpt

arXiv:2609.05016v2 Announce Type: replace-cross Abstract: Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to $10\text{--}100\times$.

## When is Warmstarting Effective for Scaling Language Models?

DevFeed: [When is Warmstarting Effective for Scaling Language Models?](<https://devfeed.tech/articles/when-is-warmstarting-effective-for-scaling-language-models-141560.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.13405>)

Author: Neeratyoy Mallik, Maciej Janowski, Johannes Hog, Herilalaina Rakotoarison, Josif Grabocka, Frank Hutter, Aaron Klein

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [ablation](<https://devfeed.tech/tags/ablation.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [accelerate](<https://devfeed.tech/tags/accelerate.md>), [adoption](<https://devfeed.tech/tags/adoption.md>), [also](<https://devfeed.tech/tags/also.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [base](<https://devfeed.tech/tags/base.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [budget](<https://devfeed.tech/tags/budget.md>), [but](<https://devfeed.tech/tags/but.md>), [can](<https://devfeed.tech/tags/can.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [effective](<https://devfeed.tech/tags/effective.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [models](<https://devfeed.tech/tags/models.md>), [scaling](<https://devfeed.tech/tags/scaling.md>)

### AI overview

The abstract reports that warmstarting can speed convergence, but only up to a useful growth limit: beyond an empirically observed upper bound on growth factor g, training from scratch is more efficient. Across experiments with dense MLPs and dense language models, 2x growth was the most reliable for convergence speedups. Gains were strongest at budgets below 20 tokens per parameter and diminished as budgets increased. The authors also find that preserving the smaller model's initial post-growth performance is unnecessary for strong final results, and that simple architecture-agnostic growth strategies can outperform more complex operators. They fit scaling laws to guide choices of when and how much to grow; the abstract does not give the numerical upper bound.

### Source excerpt

arXiv:2605.13405v2 Announce Type: replace Abstract: Model growth from a given checkpoint aims to accelerate training of a larger model, offering potential resource savings. Despite recent interest, warmstarting has seen limited practical adoption in large-scale training. We attribute this to two underexplored factors: (1) an overemphasis on preserving the smaller model's performance at initialization, which constrains operator design for new architectures, and (2) insufficient analysis of how growth interacts with hyperparameters and scaling behavior, compounded by inconsistent growth factors across the literature. We show that preserving the base model's initial post-growth performance is not necessary for strong final performance, and that simple, architecture-agnostic growth strategies can outperform more complex warmstarting operators. Crucially, we empirically identify an upper bound on the growth factor $g$ beyond which training from scratch is more efficient. We observe this across multiple ablation setups. Notably, this limit is also present, but unreported, in prior published results. Across our experiments on dense MLPs and dense language models, we find that a $2\times$ growth factor is the most reliable in yielding convergence speedups, with gains most pronounced under 20 tokens/parameter budgets and diminishing as budget increases. We fit scaling laws over these observations to provide predictive guidance for practitioners deciding when and how much to grow. Together, our analysis provides practical guidelines and empirical limits for model growth.

## K2-Horizon: A Frontier Fleet of Open Source Language Models

DevFeed: [K2-Horizon: A Frontier Fleet of Open Source Language Models](<https://devfeed.tech/articles/k2-horizon-a-frontier-fleet-of-open-source-language-models-137645.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21003>)

Author: K2 Team, Zhengzhong Liu, Haonan Li, Xuezhe Ma, Alexander Moreno, Subham Sekhar Sahoo, Rupesh Kumar Srivastava, Mikhail Yurochkin, Hadi Abdine, Yash Akhauri, Adharsh Babu, David Bellamy, Jalaj Bhandari

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [adapters](<https://devfeed.tech/tags/adapters.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [attention](<https://devfeed.tech/tags/attention.md>), [automation](<https://devfeed.tech/tags/automation.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [deep-research](<https://devfeed.tech/tags/deep-research.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [math](<https://devfeed.tech/tags/math.md>), [moe](<https://devfeed.tech/tags/moe.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reproducibility](<https://devfeed.tech/tags/reproducibility.md>), [scale](<https://devfeed.tech/tags/scale.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>)

### AI overview

K2-Horizon is a six-model open language-model fleet, ranging from 0.9B dense to 375B-A23B MoE, aimed at reasoning and agentic tasks. Its authors report leading or competitive results against open models of similar size on agentic and coding benchmarks. The 36B-A4B model combines Mixture-of-Value Attention and softplus attention gating; on the same training data, it nearly matches a 32B dense model with one-seventh the active parameters. Uno diffusion adapters are reported to speed decoding by up to 2.2x without a separate draft model. The release includes intermediate checkpoints, pretraining data, logs, code, and documented data pipelines, including those that generated roughly 10T synthetic pretraining tokens. Four models share a 22T-token corpus and recipe, enabling controlled studies of scale and sparsity.

### Source excerpt

arXiv:2611.21003v1 Announce Type: new Abstract: We present K2-Horizon, a fleet of six fully open language models, from a 0.9B dense model to a 375B-A23B MoE, built for general-purpose reasoning and agentic work across math and science, software engineering, deep research, office automation, and beyond. The models lead or are competitive with strong open models of their size on agentic and coding benchmarks. On the modeling side, to strengthen reasoning, we introduce Mixture-of-Value Attention (MoVA), applying MoE to attention value projections, and softplus attention gating, a more flexible, theoretically motivated gate. Our 36B-A4B model uses both and nearly matches a 32B dense model trained on the same data, with one-seventh the active parameters. We also introduce Uno, lossless diffusion adapters that speed up decoding by up to 2.2x without a separate draft model. Our release follows the LLM360 definition of open source for transparency and reproducibility: the full development history, including intermediate checkpoints from every training stage, pre-training data, training logs, and all training, evaluation, and infrastructure code, notably our production-scale xLLM framework for dense and MoE training and the RL360 reinforcement-learning stack. Beyond the data itself, we release and document in detail every pipeline that built it, the part of model development that open releases most rarely disclose. These pipelines produced roughly 10T synthetic pre-training tokens, showing that synthetic data can be a major ingredient of pre-training at scale. Four of the models share one 22T-token pre-training corpus and recipe, revealing consistent scaling behavior and enabling controlled studies of scale and sparsity. All artifacts are available at https://ifm.ai/k2/

## Multi-Branch Transformer

DevFeed: [Multi-Branch Transformer](<https://devfeed.tech/articles/multi-branch-transformer-138937.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19800>)

Author: Gaotang Li, Lingjie Chen, Chunyuan Deng, Zhiwei Xu, Hongzheng Chen, Christopher De Sa, Jingrui He, Yingheng Wang, Hanghang Tong

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Transformers](<https://devfeed.tech/topics/transformers.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Cache](<https://devfeed.tech/topics/cache.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [blocks](<https://devfeed.tech/tags/blocks.md>), [branch](<https://devfeed.tech/tags/branch.md>), [cache](<https://devfeed.tech/tags/cache.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [effective](<https://devfeed.tech/tags/effective.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [engine](<https://devfeed.tech/tags/engine.md>), [exploits](<https://devfeed.tech/tags/exploits.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>), [quality](<https://devfeed.tech/tags/quality.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

The Multi-Branch Transformer (MBT) adds parallel, independently parameterized Transformer branches between shared initial and final layers. In controlled pretraining at four MoE scales (0.8B-2.1B parameters), two-branch Wide-Shallow models outperform sequential Transformers at matched parameter and training-compute budgets; Balanced variants improve further and retain comparable quality as branch count increases. The proposed Joint-KV method builds a shared KV cache from branch representations, reducing full-model cache storage by up to 34%. With an inference engine using branch grouping and cache reuse, the final Balanced MBT recipe reports up to 51.4% higher decoding throughput on one H200 GPU. Fitted scaling curves estimate that standard Transformers need up to 1.26x the training body compute to match its validation loss. These results are reported for the studied settings; the supplied text is an abstract, not full paper evidence.

### Source excerpt

arXiv:2611.19800v1 Announce Type: new Abstract: Transformer language models are commonly scaled by making their blocks wider or stacking them deeper. In this work, we explore a complementary path: branching, the parallel composition of full Transformer blocks. We introduce the Multi-Branch Transformer (MBT), which processes the input through shared initial layers, executes independently parameterized branches of sequential blocks in parallel, and fuses their outputs before shared final layers. At comparable training compute, we study three MBT variants: Wide-Shallow preserves block width, Narrow-Deep preserves sequential depth, and Balanced interpolates between them. In controlled pretraining experiments at four mixture-of-experts scales (0.8B to 2.1B total parameters), 2-branch Wide-Shallow MBTs consistently outperform sequential Transformers under matched parameter and training-compute budgets. Balanced MBTs improve further and retain comparable quality as the number of branches increases. To improve inference efficiency at long contexts, we introduce Joint-KV, which constructs a shared KV cache from branch representations at each depth. Our final recipe combines Balanced MBTs with Joint-KV, supported by an inference engine that exploits branch grouping and shared-cache reuse. Fitted scaling curves indicate that standard Transformers require up to 1.26x as much training body compute to match our final models' validation loss. These models reduce full-model KV-cache storage by up to 34% and deliver up to 51.4% higher decoding throughput on a single H200 GPU. Ablation studies show that branching yields gains beyond changes in width and depth, enables an effective KV-sharing strategy (Joint-KV), remains robust across fusion functions, and performs best in the middle layers. These findings show that branching offers a practical path for scaling language models toward both better modeling quality and faster inference.

## Escaping Alignment: A Physical Trap Model of Best-of-N Jailbreaking

DevFeed: [Escaping Alignment: A Physical Trap Model of Best-of-N Jailbreaking](<https://devfeed.tech/articles/escaping-alignment-a-physical-trap-model-of-best-of-n-jailbreaking-137284.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.32116>)

Author: Marco Biroli

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [prompt](<https://devfeed.tech/topics/prompt.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [adversarial](<https://devfeed.tech/tags/adversarial.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [attack](<https://devfeed.tech/tags/attack.md>), [attack-surface](<https://devfeed.tech/tags/attack-surface.md>), [computer-science](<https://devfeed.tech/tags/computer-science.md>), [cond-mat-stat-mech](<https://devfeed.tech/tags/cond-mat-stat-mech.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [predictions](<https://devfeed.tech/tags/predictions.md>), [random](<https://devfeed.tech/tags/random.md>), [safety](<https://devfeed.tech/tags/safety.md>), [safety-mechanism](<https://devfeed.tech/tags/safety-mechanism.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [temperature](<https://devfeed.tech/tags/temperature.md>)

### AI overview

The paper challenges the view that Best-of-N jailbreak success follows a fixed power law: it reports that the exponent drifts with N and that an apparent exponential crossover may reflect a finite adversarial dataset. Its physical barrier model assigns each prompt a baseline safety level and each augmentation a random, thermally activated barrier. Five interpretable parameters describe the attack surface across augmentation count N, completions per augmentation M, and generation temperature T. The authors report extrapolations from N ≤ 100 to N = 10⁴, a shared scaling function across five models, and temperature predictions from a fit at one temperature; they say these results hold with two automated judges and on held-out prompts.

### Source excerpt

arXiv:2609.32116v2 Announce Type: replace Abstract: Best-of-$N$ jailbreaking (BoN) bypasses safeguards of aligned models by drawing $N$ independent augmentations of an unsafe prompt and sampling $M$ completions of each. Previous works have shown that the attack success rate (ASR) seems to follow a power-law in $N$, which we challenge. The exponent drifts with $N$, and the reported exponential crossover is consistent with a finite-size effect of the adversarial dataset. Little work has been done to explore the entire two-budget ($N, M$) attack surface as well as its dependence on the generation temperature $T$. We introduce a simple barrier model where each prompt has a baseline safety level and each augmentation a random thermally activated barrier. Then five numbers, each backed by an interpretable safety mechanism, determine the entire ($N, M$) attack surface. They extrapolate predictions from $N \leq 100$ to $N = 10^4$, collapse five distinct models on the same scaling function and predict ASR at different temperatures from the one they were fitted at. The extrapolation holds under two automated judges and on held-out prompts.

## Rethinking Supervision Scaling in On-Policy Distillation: From Dense Supervision to Stochastic Horizons

DevFeed: [Rethinking Supervision Scaling in On-Policy Distillation: From Dense Supervision to Stochastic Horizons](<https://devfeed.tech/articles/rethinking-supervision-scaling-in-on-policy-distillation-from-dense-supervision-to-stochastic-horizons-139083.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20687>)

Author: Xinhao Yao, Zhenxing Dou, Yong Liu

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [batch](<https://devfeed.tech/tags/batch.md>), [bias](<https://devfeed.tech/tags/bias.md>), [code](<https://devfeed.tech/tags/code.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [compute](<https://devfeed.tech/tags/compute.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [generation](<https://devfeed.tech/tags/generation.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [horizon](<https://devfeed.tech/tags/horizon.md>), [learning](<https://devfeed.tech/tags/learning.md>), [mathematical-reasoning](<https://devfeed.tech/tags/mathematical-reasoning.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [overhead](<https://devfeed.tech/tags/overhead.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [supervision](<https://devfeed.tech/tags/supervision.md>)

### AI overview

On-policy distillation (OPD) gains less from supervising additional trajectory tokens as trajectories grow longer, though later positions remain learnable and receive supervision transferred across positions. The proposed Stochastic Horizons On-Policy Distillation (SH-OPD) randomly varies rollout horizons using one shared horizon per training batch, reducing redundant supervision while retaining access to deeper states and avoiding per-sequence scheduling overhead. The authors characterize a resulting reweighting bias and compute trade-off. In reported experiments, SH-OPD achieved slightly higher mathematical reasoning accuracy at about 45% of standard OPD's per-update cost, and competitive code generation performance at about 60% of baseline cost.

### Source excerpt

arXiv:2611.20687v1 Announce Type: new Abstract: On-policy distillation (OPD) provides dense token-level supervision along student-generated trajectories, yet the learning benefits of longer trajectories may not scale proportionally with their computational cost. We investigate how supervision utility evolves with trajectory length. Controlled experiments reveal diminishing returns from additional supervised tokens, while showing that later positions remain learnable and exhibit substantial cross-position supervision transfer. Gradient analysis further identifies shared update components across response positions and diverse position-specific residuals whose cross-rollout consistency decreases at greater depths. Motivated by these findings, we propose Stochastic Horizons On-Policy Distillation (SH-OPD), which reduces redundant supervision by randomly varying rollout horizons while preserving access to deeper response states. SH-OPD samples a shared horizon for each training batch, enabling efficient parallel generation without per-sequence horizon scheduling overhead. We characterize the resulting reweighting bias and derive a compute-reweighting trade-off under standard optimization assumptions. Experiments on mathematical reasoning and code generation demonstrate that SH-OPD achieves slightly higher mathematical reasoning accuracy at approximately 45% of the per-update cost of standard OPD, while maintaining competitive code generation performance at around 60% of the baseline cost. These results highlight the benefits of efficient supervision allocation across response depths.

## Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws

DevFeed: [Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws](<https://devfeed.tech/articles/fast-catch-up-late-switching-optimal-batch-size-scheduling-via-functional-scaling-laws-137240.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.14208>)

Author: Jinbo Wang, Binghui Li, Zhanpeng Zhou, Mingze Wang, Yuxuan Sun, Jiaqi Zhang, Xunliang Cai, Lei Wu

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [batch](<https://devfeed.tech/tags/batch.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [effect](<https://devfeed.tech/tags/effect.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [fast](<https://devfeed.tech/tags/fast.md>), [functional](<https://devfeed.tech/tags/functional.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [llm](<https://devfeed.tech/tags/llm.md>), [math-oc](<https://devfeed.tech/tags/math-oc.md>), [moe](<https://devfeed.tech/tags/moe.md>), [noise](<https://devfeed.tech/tags/noise.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>), [size](<https://devfeed.tech/tags/size.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [switching](<https://devfeed.tech/tags/switching.md>)

### AI overview

The paper uses functional scaling laws to characterize batch-size schedules under a fixed data budget. It reports that easy tasks favor steadily increasing batch sizes, while hard tasks favor small batches through most of training followed by a late switch to large batches. The proposed "fast catch-up" mechanism attributes rapid post-switch loss alignment with the constant-large-batch trajectory to forgetting accumulated gradient noise; catch-up speed depends on task difficulty. Experiments on Dense and MoE language models up to 1.1B parameters and 1T tokens reportedly found late-switch schedules outperformed constant-batch and early-switch baselines across the tested settings.

### Source excerpt

arXiv:2602.14208v2 Announce Type: cross Abstract: Batch size scheduling (BSS) plays a critical role in large-scale deep learning training, influencing both optimization dynamics and computational efficiency. Yet, its theoretical foundations remain poorly understood. In this work, we show that the functional scaling law (FSL) framework introduced in Li et al. (2025a) provides a principled lens for analyzing BSS. Specifically, we characterize the optimal BSS under a fixed data budget and show that its structure depends sharply on task difficulty. For easy tasks, optimal schedules keep increasing batch size throughout. In contrast, for hard tasks, the optimal schedule maintains small batch sizes for most of training and switches to large batches only in a late stage. To explain the emergence of late switching, we uncover a dynamical mechanism -- the fast catch-up effect -- which also manifests in large language model (LLM) pretraining. After switching from small to large batches, the loss rapidly aligns with the constant large-batch trajectory. Using FSL, we show that this effect stems from rapid forgetting of accumulated gradient noise, with the catch-up speed determined by task difficulty. Crucially, this effect implies that large batches can be safely deferred to late training without sacrificing performance, while substantially reducing data consumption. Finally, extensive LLM pretraining experiments -- covering both Dense and MoE architectures with up to 1.1B parameters and 1T tokens -- validate our theoretical predictions. Across all settings, late-switch schedules consistently outperform constant-batch and early-switch baselines.

## A phase diagram for training and inference in language models

DevFeed: [A phase diagram for training and inference in language models](<https://devfeed.tech/articles/a-phase-diagram-for-training-and-inference-in-language-models-138948.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19848>)

Author: Zhuo-Yang Song, Zeyu Li, Qing-Hong Cao, Ming-xing Luo, Hua Xing Zhu

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [cond-mat-dis-nn](<https://devfeed.tech/tags/cond-mat-dis-nn.md>), [cond-mat-stat-mech](<https://devfeed.tech/tags/cond-mat-stat-mech.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dimension](<https://devfeed.tech/tags/dimension.md>), [flow](<https://devfeed.tech/tags/flow.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [training](<https://devfeed.tech/tags/training.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

The study maps how document representations change across Transformer depth using document distinguishability and document dimension. On this phase diagram, most of 156 untrained inference flows move toward collapse or noise, depending on initialization scale, while flows from 8 trained models across 5 architecture families enter and remain in a central region. A discrete Langevin description attributes the contrast to whether the effective force there behaves like a saddle or a stable node. For Pythia-1.4B, the authors interpret training as a phase transition. These abstract-level findings connect representation geometry with model behavior.

### Source excerpt

arXiv:2611.19848v1 Announce Type: new Abstract: A language model is a complex system whose behavior is determined jointly by the Transformer and the corpus, yet how the representation geometry in the Transformer relates to the actual behavior of language models remains open. We analyze how the representation geometry changes along the forward propagation of actual language models and characterize the representation geometry of a document at a given layer by two collective coordinates, document distinguishability and document dimension. On the phase diagram spanned by these collective coordinates, inference is represented as a flow along depth. We find that, for untrained models, depending on their initialization scale, most of the 156 inference flows move toward the collapse and noise regions of the phase diagram. By contrast, the inference flows of 8 trained models from 5 architecture families all enter and reside in the central region of the phase diagram. Describing the evolution of the flows with a discrete Langevin equation, we find that the difference between the inference flows of untrained runs and those of trained models reduces to whether the effective force on the flows in this region is of saddle or stable-node type. On Pythia-1.4B, we interpret the training process as a phase transition. These results give a unified gateway to the relation between representation geometry and language-model behavior.

## Latent Feedback Is a Compute and Data Amplifier

DevFeed: [Latent Feedback Is a Compute and Data Amplifier](<https://devfeed.tech/articles/latent-feedback-is-a-compute-and-data-amplifier-139042.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20404>)

Author: Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, Hector Liu

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [amplifier](<https://devfeed.tech/tags/amplifier.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>)

### AI overview

In the reported experiments, latent feedback--feeding each token's top-layer state into the next token's input--amplifies compute and data use compared with a vanilla Transformer trained for more epochs. Across model sizes and unique-data amounts, it reached the Transformer's loss with less compute at fixed data, or less unique data at fixed epochs; the savings generally grew with data per parameter and repetition. The reported maximum savings were 48% compute and 47% unique data. The advantage did not hold when unique data was scarce relative to model size, a regime the authors say current models are far from. A scaling law fitted on 60M-271M models predicted held-out 778M losses within 0.3% and the model gap within 0.05%. These findings are limited to the experiments and scaling-law predictions described in the abstract.

### Source excerpt

arXiv:2611.20404v1 Announce Type: new Abstract: When high-quality data runs out, a common practice is to train on it for several epochs, but each repeat is worth less than the last. Latent feedback, which feeds each token's top-layer state into the next token's input, is another way to spend that compute, yet it has not been tested against a vanilla Transformer that spends the same compute on more epochs. We run this test across a ladder of model sizes, unique data and epochs, tuning both models at every size and amount of unique data. Latent feedback makes every FLOP and every token count for more except when unique data is scarce relative to model size, a regime that today's models are far from. We find that latent feedback serves as an amplifier: with unique data fixed, it reaches a Transformer's loss with less compute; with epochs fixed, it reaches that loss with less unique data. Furthermore, both savings grow with unique data per parameter and with repetition, and downstream tasks agree. To go beyond our grid, we fit one scaling law to both models, in which latent feedback enters as a Transformer with more parameters; it predicts that both savings keep growing with unique data and repetition. Fitted on models from 60M to 271M parameters, the law predicts the losses at a held-out 778M size to within 0.3% and the gap between the two models to within 0.05%. At most, latent feedback saves 48% of the compute and 47% of the unique data. When data, not compute, is the limit, latent feedback is the better use of a FLOP for a model of a given size.

[Next page](<https://devfeed.tech/topics/scaling-laws.md?cursor=WyIyMDI2LTExLTE3VDA1OjAwOjAwKzAwOjAwIiwgIjBhMDZiZWRjLWZiYmQtNGMxNy1hNGNiLTYwYWQzNmYwMGVjNSJd>)