# Chinchilla scaling law

A compute-optimal training scaling law for transformer language models that relates model size and training-token count under a fixed compute budget.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## What Does a Bias Head Measure? Non-Identifiability in Head-Level Bias Attribution

DevFeed: [What Does a Bias Head Measure? Non-Identifiability in Head-Level Bias Attribution](<https://devfeed.tech/articles/what-does-a-bias-head-measure-non-identifiability-in-head-level-bias-attribution-141863.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23218>)

Author: S. M Seefat Alam

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [qwen](<https://devfeed.tech/topics/qwen.md>), [llama](<https://devfeed.tech/topics/llama.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attention](<https://devfeed.tech/tags/attention.md>), [attribution](<https://devfeed.tech/tags/attribution.md>), [bias](<https://devfeed.tech/tags/bias.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [fairness](<https://devfeed.tech/tags/fairness.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [llama-3-1](<https://devfeed.tech/tags/llama-3-1.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [qwen](<https://devfeed.tech/tags/qwen.md>)

### AI overview

On Qwen-2.5-7B-Instruct and Llama-3.1-8B-Instruct, the audited head-level bias map could not be identified as measuring bias: although it strongly correlated with a label-blind response-length map, breaking the length-label coupling showed that length explained only 1% of the statistic's variance, while prompt identity explained 64%. Within a fixed prompt, fairness label, response length and protected-group-token presence were highly collinear, leaving observational attribution unresolved. The authors also report that bias selectivity was not concentrated in late layers and that none of the tested head sets reduced unfairness when ablated. Ablating the complementary set instead reliably damaged refusal behaviour, suggesting the pipeline detects effects but not specifically bias.

### Source excerpt

arXiv:2611.23218v1 Announce Type: cross Abstract: Methods that localise social bias in large language models to individual attention heads and remove it by ablation are validated by their effect on a fairness metric, never by evidence that the statistic selecting the heads measures bias. We audit this on Qwen-2.5-7B-Instruct and Llama-3.1-8B-Instruct. The head map correlates at $\rho = -0.950$ and $-0.768$ with a label-blind response-length map, apparently identifying a surface confound. An intervention that breaks the length-label coupling rejects this reading: with the prompt held fixed, response length explains 1% of the statistic's variance. What does explain it is prompt identity, at 64% of variance, in an ordering that is not monotonic in length. Within any fixed prompt the fairness label, response length and the presence of a protected-group token are collinear at $\phi = 0.95$, partly by definition of the labelling rule, so no observational analysis can attribute the map to any of them, and no method in this family performs an experiment that could. The substantive claims fail independently: bias selectivity is not late-layer-concentrated on either model under either reference reading, and no head set we tested reduces the unfairness rate when ablated. Its complement, scored on the opposite label, reliably destroys refusal behaviour ($p = 2.7\times10^{-5}$), demonstrating that the pipeline detects effects at this scale.

## Latent Feedback Is a Compute and Data Amplifier

DevFeed: [Latent Feedback Is a Compute and Data Amplifier](<https://devfeed.tech/articles/latent-feedback-is-a-compute-and-data-amplifier-139042.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20404>)

Author: Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, Hector Liu

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [amplifier](<https://devfeed.tech/tags/amplifier.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>)

### AI overview

In the reported experiments, latent feedback--feeding each token's top-layer state into the next token's input--amplifies compute and data use compared with a vanilla Transformer trained for more epochs. Across model sizes and unique-data amounts, it reached the Transformer's loss with less compute at fixed data, or less unique data at fixed epochs; the savings generally grew with data per parameter and repetition. The reported maximum savings were 48% compute and 47% unique data. The advantage did not hold when unique data was scarce relative to model size, a regime the authors say current models are far from. A scaling law fitted on 60M-271M models predicted held-out 778M losses within 0.3% and the model gap within 0.05%. These findings are limited to the experiments and scaling-law predictions described in the abstract.

### Source excerpt

arXiv:2611.20404v1 Announce Type: new Abstract: When high-quality data runs out, a common practice is to train on it for several epochs, but each repeat is worth less than the last. Latent feedback, which feeds each token's top-layer state into the next token's input, is another way to spend that compute, yet it has not been tested against a vanilla Transformer that spends the same compute on more epochs. We run this test across a ladder of model sizes, unique data and epochs, tuning both models at every size and amount of unique data. Latent feedback makes every FLOP and every token count for more except when unique data is scarce relative to model size, a regime that today's models are far from. We find that latent feedback serves as an amplifier: with unique data fixed, it reaches a Transformer's loss with less compute; with epochs fixed, it reaches that loss with less unique data. Furthermore, both savings grow with unique data per parameter and with repetition, and downstream tasks agree. To go beyond our grid, we fit one scaling law to both models, in which latent feedback enters as a Transformer with more parameters; it predicts that both savings keep growing with unique data and repetition. Fitted on models from 60M to 271M parameters, the law predicts the losses at a held-out 778M size to within 0.3% and the gap between the two models to within 0.05%. At most, latent feedback saves 48% of the compute and 47% of the unique data. When data, not compute, is the limit, latent feedback is the better use of a FLOP for a model of a given size.

## Training Language Models via Neural Cellular Automata

DevFeed: [Training Language Models via Neural Cellular Automata](<https://devfeed.tech/articles/training-language-models-via-neural-cellular-automata-131890.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2603.10055>)

Author: Dan Lee, Seungwook Han, Akarsh Kumar, Pulkit Agrawal

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Cellular automaton](<https://devfeed.tech/topics/cellular-automaton.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>)

Tags: [cellular-automata](<https://devfeed.tech/tags/cellular-automata.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>)

### AI overview

The paper proposes pre-pre-training language models on synthetic, non-linguistic data generated by neural cellular automata (NCA), then training on natural language. In 400M-3B models using Chinchilla-optimal data budgets, 164M NCA tokens improved downstream language modeling by up to 6% and accelerated convergence by up to 1.6x; the authors report this outperformed pre-pre-training on 1.6B high-quality natural-language tokens with more compute. They identify attention layers as the most transferable and report that useful NCA complexity varies by domain: simpler dynamics benefit code, while math and web text favor more complex dynamics. These findings suggest synthetic data distributions can be tuned to target domains; the evidence here is limited to the supplied abstract.

### Source excerpt

arXiv:2603.10055v2 Announce Type: replace-cross Abstract: Pre-training is crucial for large language models (LLMs), as it is when most representations and capabilities are acquired. However, natural language pre-training has problems: high-quality text is finite, contains human biases, and entangles knowledge with reasoning. This raises a fundamental question: is natural language the only path to intelligence? We propose using neural cellular automata (NCA) to generate synthetic, non-linguistic data for pre-pre-training LLMs--training on synthetic-then-natural language. NCA data exhibits rich spatiotemporal structure and statistics resembling natural language while being controllable and cheap to generate at scale. We find that pre-pre-training on only 164M NCA tokens improves downstream language modeling by up to 6% and accelerates convergence by up to 1.6x in 400M-3B models trained with Chinchilla optimal data budgets. Surprisingly, this even outperforms pre-pre-training on 1.6B tokens of high-quality natural language data with more compute. Investigating what drives transfer, we find that attention layers are the most transferable, and that optimal NCA complexity varies by domain: code benefits from simpler dynamics, while math and web text favor more complex ones. These results enable systematic tuning of the synthetic distribution to target domains. More broadly, our work opens a path toward more efficient models with fully synthetic pre-training.

## Curiosity-0826-4B: Generative Medical Event Sequence Modeling Across Pediatric and Adult Populations

DevFeed: [Curiosity-0826-4B: Generative Medical Event Sequence Modeling Across Pediatric and Adult Populations](<https://devfeed.tech/articles/curiosity-0826-4b-generative-medical-event-sequence-modeling-across-pediatric-and-adult-populations-132161.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15710>)

Author: Mani Nagarathnam, Luke Benham, Paul Blazek, John Cashy, Kevin Chung, John Muenning, Hank Voeller, Shane Waxler, Davis White, Alison Callahan, Kevin Dysart, Andrew Loza, Michal A. Mankowski, Mattheus R

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [Language models](<https://devfeed.tech/topics/language-models.md>)

Tags: [compute](<https://devfeed.tech/tags/compute.md>), [context](<https://devfeed.tech/tags/context.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cosmos](<https://devfeed.tech/tags/cosmos.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [curiosity](<https://devfeed.tech/tags/curiosity.md>), [death](<https://devfeed.tech/tags/death.md>), [diagnosis](<https://devfeed.tech/tags/diagnosis.md>), [disease](<https://devfeed.tech/tags/disease.md>), [events](<https://devfeed.tech/tags/events.md>), [fewer](<https://devfeed.tech/tags/fewer.md>), [floating-point](<https://devfeed.tech/tags/floating-point.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [foundation-model](<https://devfeed.tech/tags/foundation-model.md>), [generative](<https://devfeed.tech/tags/generative.md>), [laboratory](<https://devfeed.tech/tags/laboratory.md>), [medical](<https://devfeed.tech/tags/medical.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [sequence](<https://devfeed.tech/tags/sequence.md>)

### AI overview

Curiosity-0826-4B extends zero-shot medical timeline prediction to pediatric cohorts and adult mortality outcomes. On tasks evaluated for both models, it generally matched or exceeded Curiosity-0226-1B; laboratory prediction MAE was 18-46% lower. Pediatric AUROCs for asthma diagnosis (0.742-0.785) and exacerbation (0.829-0.852) were comparable to the adult reference, while sickle cell crisis prediction (0.820-0.847) was lower than the adult result (0.915). Mortality prediction across adult hospice, inpatient, and ambulatory settings achieved AUROCs of 0.821-0.942. The 4.3B-parameter model has a 16K context window, versus 1B parameters and 8K for its predecessor; it used 56% fewer training tokens but 2.2 times the training FLOPs. The abstract calls for further validation of calibration, subgroup and event-frequency performance, timeline completion, generalization, and prospective clinical and operational utility.

### Source excerpt

arXiv:2611.15710v1 Announce Type: cross Abstract: We introduce Curiosity-0826-4B, an updated version of the generative medical event foundation model trained on Epic Cosmos that simulates patient timelines to produce zero-shot clinical predictions. Curiosity-0826-4B incorporates a broader patient population and richer clinical event representations, while generally matching or exceeding Curiosity-0226-1B, the largest model in the previous Curiosity-0226 series, on evaluated tasks supported by both models. Laboratory value prediction improved, with MAE 18--46% lower than that of Curiosity-0226-1B. We evaluated prediction of disease events across pediatric cohorts aged 2--17 years. AUROCs for new asthma diagnosis (0.742--0.785) and asthma exacerbation (0.829--0.852) were comparable to those in the adult reference cohort, while AUROCs for sickle cell crisis (0.820--0.847) were lower than in adults (0.915). Explicit representation of documented death enabled new mortality prediction tasks among adults across hospice, inpatient, and ambulatory settings (AUROC 0.821--0.942). Scaling-law analysis guided compute-optimal initial pretraining followed by continued pretraining with context extension. The resulting Curiosity-0826-4B model has more parameters (4.3B versus 1B) and a longer context window (16K versus 8K). It processes 56% fewer training tokens and requires 2.2 times the training floating point operations (FLOPs) compared with its predecessor. By extending a common zero-shot prediction strategy to pediatric patients and additional outcomes, Curiosity-0826-4B broadens opportunities to develop clinical and operational applications without training separate task-specific models. Further validation should assess local calibration, performance across patient subgroups and event frequencies, timeline completion rates, and generalization to new settings, alongside prospective evaluation of clinical and operational utility.

## On the Origin of Algorithmic Progress in AI: Evidence from Language Model Pre-Training

DevFeed: [On the Origin of Algorithmic Progress in AI: Evidence from Language Model Pre-Training](<https://devfeed.tech/articles/on-the-origin-of-algorithmic-progress-in-ai-evidence-from-language-model-pre-training-128075.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2511.21622>)

Author: Hans Gundlach, Alex Fogelson, Jayson Lynch, Ana Trisovic, Jonathan Rosenfeld, Anmol Sandhu, Neil Thompson

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [Language models](<https://devfeed.tech/topics/language-models.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [ai](<https://devfeed.tech/tags/ai.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [algorithms](<https://devfeed.tech/tags/algorithms.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [experimental](<https://devfeed.tech/tags/experimental.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [gap](<https://devfeed.tech/tags/gap.md>), [improvements](<https://devfeed.tech/tags/improvements.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [law](<https://devfeed.tech/tags/law.md>), [literature](<https://devfeed.tech/tags/literature.md>), [progress](<https://devfeed.tech/tags/progress.md>), [scale](<https://devfeed.tech/tags/scale.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [small](<https://devfeed.tech/tags/small.md>), [training](<https://devfeed.tech/tags/training.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

The paper argues that algorithmic efficiency gains in language model pretraining depend on training compute scale. Ablations of key innovations account for less than 10x improvement at small compute scales; estimates for additional innovations raise the total to less than 100x, far below the estimated 22,000x gain from 2012 to 2023. Scaling experiments suggest that differences between LSTMs and Transformers in compute-optimal scaling-law exponents help explain the gap, while many other innovations show little scaling difference. The authors use scaling extrapolations and literature estimates to account for most of the historical gains, concluding that progress at small training scales was slower than commonly assumed and that efficiency estimates depend strongly on the chosen reference scale.

### Source excerpt

arXiv:2511.21622v2 Announce Type: replace-cross Abstract: Algorithms have been estimated to increase AI pretraining FLOP efficiency by a factor of 22,000 between 2012 and 2023 (Ho et al., 2024). Running ablation experiments on key innovations from this time period, we are able to account for less than 10x of these gains at small compute scales. Surveying the broader literature, we estimate that additional innovations not included in our ablations also account for less than 10x, yielding less than 100x efficiency gains total in low-compute regimes. This leads us to conduct scaling experiments, which provide evidence that the efficiency gap can be explained by algorithms with scale-dependent efficiency improvements. In particular, we conduct scaling experiments between LSTMs and Transformers, finding evidence for exponent differences in their compute-optimal scaling law while finding little scaling difference for many other innovations. These experiments demonstrate that, contrary to standard assumptions, an algorithm's efficiency gains are tied to compute scale. Using experimental scaling extrapolations and literature estimates, we can account for the majority of efficiency gains over the same time period. Our results indicate that algorithmic progress at smaller training compute levels has been far slower than previously assumed, and that measures of algorithmic efficiency are strongly reference-dependent.

## Beyond the Log-Linear Frontier: Pre-Asymptotic Analysis of Finite-Budget Neural Scaling Laws

DevFeed: [Beyond the Log-Linear Frontier: Pre-Asymptotic Analysis of Finite-Budget Neural Scaling Laws](<https://devfeed.tech/articles/beyond-the-log-linear-frontier-pre-asymptotic-analysis-of-finite-budget-neural-scaling-laws-128724.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13534>)

Author: Tianyu Zhao, Zhixiang Ren

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>)

Tags: [capacity](<https://devfeed.tech/tags/capacity.md>), [compute-allocation](<https://devfeed.tech/tags/compute-allocation.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [domain](<https://devfeed.tech/tags/domain.md>), [duality](<https://devfeed.tech/tags/duality.md>), [intrinsic](<https://devfeed.tech/tags/intrinsic.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [operator](<https://devfeed.tech/tags/operator.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>)

### AI overview

The paper analyzes why finite-budget neural learning curves and compute-optimal allocations depart from separable power-law scaling. For model capacity, it derives a three-term expansion in which spectral and regularization effects create curvature and logarithmic transitions, and uses first-order Aitken differencing to initialize fits when narrow pilot grids make multi-power-law fitting ill-conditioned. For data allocation across domains, KKT duality reduces mixing to scalar water-filling and supports bottleneck bounds, an online sampling curriculum, and a simplex Hessian metric for transfer. On the iso-compute frontier C=6ND, the optimal allocation exponent drifts with capacity and data exponent variances; a Schur-complement reduction separates intrinsic drift from frozen-hyperparameter bias. Across reported tests on 45 operator spectra, Pythia, Chinchilla, and RegMix, the estimators achieve 5.89% holdout MAPE, reduce high-compute allocation regret by up to 76.4%, and lower cross-scale mixture RMSE by 59.4% with zero selection regret.

### Source excerpt

arXiv:2611.13534v1 Announce Type: new Abstract: Classical neural scaling laws model excess risk as a separable power law in model capacity $N$ and dataset size $D$. At finite budgets, however, empirical learning curves and compute-optimal allocations depart from this log-linear frontier. This paper analyzes how spectral and domain heterogeneity govern these finite-scale departures. Along the capacity axis, Mellin--Barnes analysis of covariance resolvents yields a three-term expansion where competing spectral and regularization modes produce finite-scale curvature and logarithmic transitions. Because fitting multiple power laws on narrow pilot grids is ill-conditioned, we eliminate the loss floor via first-order Aitken differencing to obtain a well-conditioned closed-form initializer. Along the data axis, KKT duality reduces $K$-domain mixing to scalar water-filling, yielding finite-budget bottleneck bounds, an online sampling curriculum, and a simplex Hessian metric for cross-domain transfer. Along the iso-compute frontier $C=6ND$, we prove that the optimal allocation exponent drifts in proportion to capacity and data exponent variances, while Schur-complement reduction separates intrinsic drift from frozen-hyperparameter bias. Across 45 operator spectra, Pythia (70M--12B), Chinchilla (245 runs), and RegMix (17 domains), our estimators achieve $5.89\%$ holdout MAPE, cut high-compute allocation regret by up to $76.4\%$, and reduce cross-scale mixture RMSE by $59.4\%$ with zero selection regret.

## BARS: Budget-Aware Sequential Sampling with Refusal for Majority Identification under Unknown Distributions

DevFeed: [BARS: Budget-Aware Sequential Sampling with Refusal for Majority Identification under Unknown Distributions](<https://devfeed.tech/articles/bars-budget-aware-sequential-sampling-with-refusal-for-majority-identification-under-unknown-distributions-128403.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.14171>)

Author: Shenxi Liu, Yiwei Li, Mingyang Zhao, Yuhang Tian, Kan Li

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [Parallel reasoning](<https://devfeed.tech/topics/parallel-reasoning.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [also](<https://devfeed.tech/tags/also.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [budget](<https://devfeed.tech/tags/budget.md>), [but](<https://devfeed.tech/tags/but.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [cost](<https://devfeed.tech/tags/cost.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [distributions](<https://devfeed.tech/tags/distributions.md>), [effective](<https://devfeed.tech/tags/effective.md>), [error](<https://devfeed.tech/tags/error.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [fewer](<https://devfeed.tech/tags/fewer.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [under](<https://devfeed.tech/tags/under.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

BARS uses sequential sampling to certify whether the leading answer has a sufficient gap over the runner-up, then returns an early stop, a budgeted majority vote, or a refusal if feasibility cannot be established. Refusal signals a limit of the rule and available budget, not that the question is unanswerable. The proposed certified mode provides proven bounds on joint answering error and on refusing cases the budget could handle; the practical mode used in experiments substitutes empirical calibration. On GSM8K, practical BARS used 22.3% fewer samples than fixed-8 sampling, including refused questions, while declining 18.8% of questions. On MATH, output accuracy reached 82.5%, while the method refused about half of the difficult questions the model could not solve.

### Source excerpt

arXiv:2611.14171v1 Announce Type: new Abstract: Repeated sampling with majority selection is an effective means of enforcing self-consistency in large language models. Recently, a variant has also been introduced that terminates early to save costs when the output distribution is favorable. But it still cannot yield a correct confidence estimate from the answer distribution. For this issue, we formalize this as majority identification under an unknown distribution with a budget and a target error rate, and propose BARS, a sequential rule certifying the top-two gap and returning an early stop, a budgeted majority vote, or a refusal when feasibility cannot be established. A refusal is a statement about the rule and its budget, not a claim that a question is unanswerable, and it tells practitioners when to widen the budget or switch models. A certified mode carries proven bounds on the joint answering error and on refusing an instance the budget could handle, while a practical mode, used in our experiments, replaces them with empirical calibration. On synthetic instances drawn from known distributions over thousands of seeds, the difficulty-driven behavior is recovered and the price of the certified mode is measured.On GSM8K the practical mode draws 22.3\% fewer samples than fixed-8 sampling over the same pool, refused questions included, at the cost of declining 18.8\% of the questions. On MATH, it greatly improves the accuracy of its output answers into 82.5\% at the cost of honestly refusing to answer half of the difficult questions that the model itself is unable to solve.

## Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability

DevFeed: [Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability](<https://devfeed.tech/articles/subspace-aware-sparse-autoencoders-for-effective-mechanistic-interpretability-123177.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.06333>)

Author: Seyed Arshan Dalili, Mehrdad Mahdavi

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [absorption](<https://devfeed.tech/tags/absorption.md>), [autoencoders](<https://devfeed.tech/tags/autoencoders.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [interpretability](<https://devfeed.tech/tags/interpretability.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [mechanistic-interpretability](<https://devfeed.tech/tags/mechanistic-interpretability.md>)

### AI overview

Standard sparse autoencoders represent each feature with one decoder direction, which can split multidimensional features across many directions and increase absorption. Subspace-Aware Sparse Autoencoders (SASA) instead learn feature subspaces, use Top-s group gating for sparsity, and adapt group rank with nuclear-norm regularization. The abstract reports an idealized result in which one group recovers the feature subspace, along with polynomial sample-complexity bounds for within-group estimation. Experiments on GPT-2 and Mistral-7B report reduced absorption versus SAEs; matched-budget GPT-2 comparisons also report better prediction of concept-family members from few latent coordinates, at a reconstruction cost.

### Source excerpt

arXiv:2606.06333v2 Announce Type: replace Abstract: Sparse Autoencoders (SAEs) are widely used for mechanistic interpretability in large language models, yet their standard formulation assigns each latent feature a single decoder direction, implicitly assuming features to be one-dimensional. We show that this assumption can induce feature splitting when model features are multi-dimensional. Geometrically, accurately reconstructing a feature under a fixed co-activation budget can require exponentially many decoder directions as its dimension grows. From an optimization perspective, adding decoder directions can strictly lower the $\ell_1$-regularized population risk, even when a basis can already reconstruct the feature exactly. Motivated by these results, we introduce Subspace-Aware Sparse Autoencoders (SASA), which replace single-vector decoders with learned subspaces, enforce sparsity through Top-$s$ group gating, and adapt group rank with nuclear-norm regularization. Under an idealized single-feature model, we prove that the SASA objective is minimized by a single group whose reconstruction map recovers the feature subspace. We also derive polynomial sample complexity bounds for subspace estimation within a group and a lower bound on the samples needed to observe every atom in a fixed fragmented dictionary. Experiments on GPT-2 and Mistral-7B show reduced absorption compared with SAEs. Matched-budget GPT-2 comparisons show lower absorption for concept families and more accurate prediction of their members using a small number of latent coordinates, at a reconstruction cost. The code is available at https://github.com/arshandalili/sasa.

## Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models

DevFeed: [Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models](<https://devfeed.tech/articles/scratchpad-patching-decoupling-compute-from-patch-size-in-byte-level-language-models-123141.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.09630>)

Author: Lin Zheng, Vasilisa Bashlovkina, Timothy Dozat, Dan Garrette, Laura Rimell, Joshua Maynez

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [cache](<https://devfeed.tech/tags/cache.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [inference](<https://devfeed.tech/tags/inference.md>), [lag](<https://devfeed.tech/tags/lag.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [patching](<https://devfeed.tech/tags/patching.md>), [quality](<https://devfeed.tech/tags/quality.md>), [size](<https://devfeed.tech/tags/size.md>)

### AI overview

Scratchpad Patching (SP) addresses patch lag in byte-level language models: predictions within a patch otherwise rely on a stale representation until that patch is fully observed, and larger patches widen this lag. SP inserts transient scratchpads to aggregate observed bytes and refresh patch-level context. It triggers them using next-byte prediction entropy, allocating compute selectively to information-dense regions and allowing inference-time compute to be adjusted after training. In experiments on natural language and code, SP improved quality at the same patch size. At 16 bytes per patch, SP models matched or closely approached the byte-level baseline on downstream evaluations, with a 16x smaller KV-cache over patches and roughly 3-4x less inference compute than the baseline.

### Source excerpt

arXiv:2605.09630v2 Announce Type: replace-cross Abstract: Tokenizer-free language models eliminate the tokenizer step of the language modeling pipeline by operating directly on bytes; patch-based variants further aggregate contiguous byte spans into patches for efficiency. However, the average patch size chosen at the model design stage governs a tight trade-off: larger patches reduce compute and KV-cache footprint, but degrade modeling quality. We trace this trade-off to patch lag: until a patch is fully observed, byte predictions within it must rely on a stale representation from the previous patch to preserve causality; this lag widens as patches grow larger. We introduce Scratchpad Patching (SP), which inserts transient scratchpads inside each patch to aggregate the bytes seen so far and refresh patch-level context for subsequent predictions. SP triggers scratchpads using next-byte prediction entropy, selectively allocating compute to information-dense regions and enabling post-hoc adjustment of inference-time compute. Across experiments on natural language and code, SP improves model quality at the same patch size; for example, even at $16$ bytes per patch, SP-augmented models match or closely approach the byte-level baseline on downstream evaluations while using a $16\times$ smaller KV-cache over patches and requiring roughly $3$-$4\times$ less inference compute than the byte-level baseline.

## Using Transformers to Generate 3-Manifold Triangulations

DevFeed: [Using Transformers to Generate 3-Manifold Triangulations](<https://devfeed.tech/articles/using-transformers-to-generate-3-manifold-triangulations-120627.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07514>)

Author: Daniel Bruwel, Ashvni Narayanan, Jonathan Spreer

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [alpha](<https://devfeed.tech/tags/alpha.md>), [behaviour](<https://devfeed.tech/tags/behaviour.md>), [blocks](<https://devfeed.tech/tags/blocks.md>), [compute](<https://devfeed.tech/tags/compute.md>), [count](<https://devfeed.tech/tags/count.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [generate](<https://devfeed.tech/tags/generate.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [learn](<https://devfeed.tech/tags/learn.md>), [math-gt](<https://devfeed.tech/tags/math-gt.md>), [model](<https://devfeed.tech/tags/model.md>), [neural](<https://devfeed.tech/tags/neural.md>), [number](<https://devfeed.tech/tags/number.md>), [one](<https://devfeed.tech/tags/one.md>), [over](<https://devfeed.tech/tags/over.md>), [parameter](<https://devfeed.tech/tags/parameter.md>), [range](<https://devfeed.tech/tags/range.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [using](<https://devfeed.tech/tags/using.md>)

### AI overview

The study trains GPT-2-style transformers to generate 3-manifold triangulations encoded as dehydration signatures. In the main runs, models trained on sphere triangulations generated signatures for closed manifolds of various topological types at a 6.9% rate; 82% of those outputs were spheres. IsoFLOP experiments on 15-tetrahedron 3-sphere triangulations, spanning 6 to 3000 petaFLOPs, showed neural scaling behavior, with a reported parameter scaling exponent of 0.49 and an optimal training ratio of 234 tokens per parameter. Smaller experiments found that transformers counted degree-one edges near-perfectly, with performance worsening for higher-degree edges. These findings are reported in the supplied abstract.

### Source excerpt

arXiv:2611.07514v1 Announce Type: cross Abstract: Using dehydration signatures as an encoding, we train GPT-2-style transformers to generate triangulations of 3-dimensional manifolds. Our main runs each use approximately 3 exaFLOPs, spanning a 200 million parameter model trained on 60 million sphere triangulations to an 11 million parameter model trained on 1.1 billion sphere triangulations. The optimal model has 10 blocks, width 640 and generates dehydration signatures of closed manifolds of various topological types at a rate of 6.9%, 82% of which are spheres. To investigate how our method scales, we use the technique of IsoFLOP curves on 15-tetrahedron triangulations of the 3-sphere, over a compute range of 6 to 3000 petaFLOPs, and observe neural scaling behaviour. We extract a parameter scaling exponent of $\alpha = 0.49$ ($[0.48, 0.50]$ stat, $\pm 0.02$ syst), and an optimal ratio of 234 tokens per parameter for training. In addition, we present a number of smaller experiments, including evidence that transformers are able to learn to count degree-one edges near-perfectly, but get progressively worse when counting edges with higher degrees.

## Prescriptive Scaling Laws for Data Constrained Training

DevFeed: [Prescriptive Scaling Laws for Data Constrained Training](<https://devfeed.tech/articles/prescriptive-scaling-laws-for-data-constrained-training-120410.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.01640>)

Author: Justin Lovelace, Christian Belardi, Srivatsa Kundurthy, Shriya Sudhakar, Kilian Q. Weinberger

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [machine learning overfitting](<https://devfeed.tech/topics/machine-learning-overfitting.md>)

Tags: [advice](<https://devfeed.tech/tags/advice.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [availability](<https://devfeed.tech/tags/availability.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [case-study](<https://devfeed.tech/tags/case-study.md>), [chinchilla-scaling-law](<https://devfeed.tech/tags/chinchilla-scaling-law.md>), [compute](<https://devfeed.tech/tags/compute.md>), [compute-allocation](<https://devfeed.tech/tags/compute-allocation.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [high-quality-data](<https://devfeed.tech/tags/high-quality-data.md>), [lambda](<https://devfeed.tech/tags/lambda.md>), [law](<https://devfeed.tech/tags/law.md>), [laws](<https://devfeed.tech/tags/laws.md>), [model](<https://devfeed.tech/tags/model.md>), [overfitting](<https://devfeed.tech/tags/overfitting.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

The proposed scaling law extends Chinchilla's unique-token assumption by adding an overfitting penalty for repeated data. It indicates that beyond a point, repeating tokens is counterproductive and compute is better spent increasing model capacity; the authors report improved performance when following its recommended configuration in data-constrained regimes. The law's single overfitting coefficient also enables comparisons across training configurations. In a case study, strong weight decay (λ=1.0) reduced that coefficient by about 70%, offering a scaling-law explanation for why optimal weight decay in data-constrained training may be much larger than standard practice. This overview is based on the abstract.

### Source excerpt

arXiv:2605.01640v2 Announce Type: replace Abstract: Training compute is increasingly outpacing the availability of high-quality data. This shifts the central challenge from optimal compute allocation to extracting maximum value from limited data. The widely adopted Chinchilla scaling law assumes every training token is unique. This limits its ability to guide pretraining decisions in data-constrained regimes. We model the excess loss under repetition with a simple additive overfitting penalty and find that it accurately describes model behavior. Our scaling law yields qualitatively new compute-optimal allocation advice. Beyond a point, further repetition is counterproductive and compute is better spent on model capacity. We show that following our law's recommended configuration improves performance in data-constrained regimes. Finally, because our one-parameter form isolates overfitting in a single coefficient, it enables direct comparison across training configurations. As a case study, we show that strong weight decay ($\lambda=1.0$) reduces this coefficient by approximately 70\%, providing a scaling-law explanation for recent findings that optimal weight decay in data-constrained regimes is an order of magnitude larger than standard practice.

## First-Token Broadcasters: Mechanistic Origins of Language Identity and Distributed Robustness in Transformers

DevFeed: [First-Token Broadcasters: Mechanistic Origins of Language Identity and Distributed Robustness in Transformers](<https://devfeed.tech/articles/first-token-broadcasters-mechanistic-origins-of-language-identity-and-distributed-robustness-in-transformers-117251.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.22361>)

Author: Arjun Pillai, Christian Hoang, Anjelo Jann Laroza

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [identity](<https://devfeed.tech/tags/identity.md>), [language](<https://devfeed.tech/tags/language.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

Language identity appears to be maintained by different attention-head patterns across the tested models and training regimes. The proposed Language Identity Head Ablation (LIHA) zeros each attention head and measures how often output switches language. In GPT-2, some heads persistently attend to the prompt's first token; ablating them has outsized effects, led by L6H1 (switch rate 0.32), suggesting a role in maintaining output language. First-token attention rises in heads above ablated layers in the tested cases, but the study does not establish functional compensation. In matched Qwen2.5-1.5B models, the Base model shows little concentration of causal influence, while the Instruct model's effects concentrate at layer 0, led by L0H5 (switch rate 0.224). GPT-2 heads found in European-language tests show no observed switch effects on tested Chinese and Russian prompts, though other early heads do; the comparison cannot isolate script from typology or tokenization.

### Source excerpt

arXiv:2606.22361v3 Announce Type: replace-cross Abstract: How is language identity represented and causally maintained in transformers, and to what extent do the mechanisms identified in one language set transfer to other languages and training regimes? We introduce Language Identity Head Ablation (LIHA), a causal intervention that zeros each attention head individually and measures the resulting language switch rate. Across our multilingual evaluation sets, LIHA identifies heads whose ablation has disproportionate effects on output language. In GPT-2, a small set of first-token broadcaster heads---led by L6H1 (switch rate 0.32, 3.23$\sigma$ above the population mean)---attend persistently to the first prompt token across query positions. The combination of persistent first-token attention and causal sensitivity suggests that these heads participate in maintaining output language. Following ablation, first-token attention increases in heads above the ablated layer for the ablations tested (permutation test against random five-head sets, $p < 10^{-5}$). We do not test whether these heads functionally compensate for the ablated head. We also compare Qwen2.5-1.5B-Base and Qwen2.5-1.5B-Instruct, which are matched in architecture and parameter count but differ in training history. The base model is nearly flat (max SR$=$0.016, 200/336 heads at SR$=$0.0), whereas the instruct model concentrates causal influence at layer~0, led by L0H5 (SR$=$0.224, 8.93$\sigma$ above the mean). Finally, GPT-2 heads identified from the European-language experiments show no observed switch effects on the tested Chinese and Russian prompts, while other early-layer heads do show effects. Because script, typology, and tokenization vary together in this comparison, these results do not isolate a specifically script-driven mechanism. Repository is available at https://github.com/ajlrza/First-Token-Broadcasters.

## GatedV: Self-Gated Values for Transformer Language Models

DevFeed: [GatedV: Self-Gated Values for Transformer Language Models](<https://devfeed.tech/articles/gatedv-self-gated-values-for-transformer-language-models-114654.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.02145>)

Author: Li-Ming Yu

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [DGX Spark](<https://devfeed.tech/topics/dgx-spark.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoder](<https://devfeed.tech/tags/decoder.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [latency](<https://devfeed.tech/tags/latency.md>), [nvidia-dgx](<https://devfeed.tech/tags/nvidia-dgx.md>), [performance](<https://devfeed.tech/tags/performance.md>), [perplexity](<https://devfeed.tech/tags/perplexity.md>), [training](<https://devfeed.tech/tags/training.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [value](<https://devfeed.tech/tags/value.md>), [values](<https://devfeed.tech/tags/values.md>)

### AI overview

GatedV applies SiLU to the attention values before aggregation, adding no trainable parameters and leaving the attention-weighting mechanism unchanged. In matched decoder-only Transformer comparisons with paired initializations and equal token budgets, it lowered mean held-out perplexity from 59.1 to 57.9 at 2.33M parameters, improving across all five seeds. At 26.16M parameters, after an equal-budget learning-rate search, perplexity fell from 20.2 to 19.7 across all three seeds. A small-model ablation found tested affine value rescaling did not fully reproduce the gains. A short interleaved NVIDIA DGX Spark benchmark measured about 0.7% higher full-context forward latency for the larger model; the reported results are limited to these experiments.

### Source excerpt

arXiv:2611.02145v1 Announce Type: cross Abstract: Standard attention forms query (Q), key (K), and value (V) matrices through learned linear projections of input hidden states. Architectures such as GAU and Mega incorporate nonlinear transformations into attention. In this paper, we propose GatedV, a simple attention variant that applies SiLU only to V before aggregation without introducing additional trainable parameters. We evaluate GatedV against a standard decoder-only Transformer using matched parameter counts, paired initializations, and equal training-token budgets. At 2.33M parameters, GatedV reduces mean held-out perplexity from 59.1 to 57.9 (2.0%), with improvements across all five paired seeds. At 26.16M parameters, following an equal-budget learning-rate search, mean held-out perplexity falls from 20.2 to 19.7 (2.4%), with improvements across all three paired seeds. Small-model ablations show that the tested affine rescaling of values does not fully reproduce GatedV's gains. A short interleaved benchmark on NVIDIA DGX Spark measures approximately 0.7% higher full-context forward latency for the larger model, based on synchronized wall-clock timings. These findings suggest that value-only self-gating can improve language modeling performance without increasing parameter count or redesigning the attention-weighting mechanism.

## A Scaling Theory of Multi-Epoch Pretraining: Late-Time Overfitting and Ensembling

DevFeed: [A Scaling Theory of Multi-Epoch Pretraining: Late-Time Overfitting and Ensembling](<https://devfeed.tech/articles/a-scaling-theory-of-multi-epoch-pretraining-late-time-overfitting-and-ensembling-114812.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.03590>)

Author: Zhen Yang, Yidi Miao, Blake Bordelon

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [machine learning overfitting](<https://devfeed.tech/topics/machine-learning-overfitting.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>)

Tags: [compute](<https://devfeed.tech/tags/compute.md>), [compute-allocation](<https://devfeed.tech/tags/compute-allocation.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [gradient-descent](<https://devfeed.tech/tags/gradient-descent.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [laws](<https://devfeed.tech/tags/laws.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [overfitting](<https://devfeed.tech/tags/overfitting.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [theory](<https://devfeed.tech/tags/theory.md>), [training-data](<https://devfeed.tech/tags/training-data.md>)

### AI overview

With fixed training data, logit averaging can reduce late-training deterioration and delay the optimal stopping point. In FineWeb language-model experiments, it also outperformed wider or deeper single models at matched parameter-token budgets over a shared training horizon. A dynamical mean-field theory for multi-epoch SGD in power-law random-feature ensembles attributes the effect to reduced prediction variance, which otherwise drives late-time growth in expected test error. Using the resulting stopping times, the authors derive compute-allocation laws for fixed data and individual model sizes: ensemble size grows with compute at least as fast as training time per member, while excess test loss above its limiting value falls as a power of compute.

### Source excerpt

arXiv:2611.03590v1 Announce Type: new Abstract: In the near future language model pretraining will become bound by total data rather than total compute. Such a regime requires revisiting scaling laws and scaling law theory. Motivated by prior works that have found utility in ensembling, we study how the number of training epochs and ensemble size jointly shape generalization when training data are fixed. In language-model experiments on FineWeb, logit averaging reduces late-training deterioration, delays optimal stopping, and can outperform wider or deeper single models at matched parameter-token budgets over a common training horizon. To analyze this interaction between data reuse and ensembling, we derive a dynamical mean-field theory (DMFT) for multi-epoch stochastic gradient descent in power-law random-feature ensembles, tracking error correlations across epochs and models. Ensembling reduces the prediction variance responsible for late-time growth in expected test error, changing the optimal stopping time. We use these stopping times to derive compute-allocation laws at fixed data and individual model sizes. Under this allocation, ensemble size grows with compute at least as fast as training time per member, while excess test loss above the limiting loss decreases as a power of compute.

## Causal Optimal-Transport Attention: Positional Priors and Length Extrapolation

DevFeed: [Causal Optimal-Transport Attention: Positional Priors and Length Extrapolation](<https://devfeed.tech/articles/causal-optimal-transport-attention-positional-priors-and-length-extrapolation-114552.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.01472>)

Author: Ming Lian

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [context window](<https://devfeed.tech/topics/context-window.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [connect](<https://devfeed.tech/tags/connect.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [optimal-transport](<https://devfeed.tech/tags/optimal-transport.md>)

### AI overview

The paper analyzes how causal optimal-transport attention induces positional attention decay and reports better loss extrapolation than softmax in preregistered experiments with 25M- and 100M-parameter models. With bounded scores and fixed balancing strength below one, every fixed-size window's attention share vanishes as context grows; for constant scores, the induced prior has an exact scale-invariant form. The authors classify positional encodings by distant-mass behavior and give finite-length bounds for summable log-distance priors near exponent one. In the tested settings, CABLE beats RoPE with evaluation-time YaRN across seeds at three training lengths, and changing only the bias beyond training length shifts long-context loss as predicted on all 30 checkpoints. Prior-only margin selection controls distant-mass growth for a specified window, length range and tolerance.

### Source excerpt

arXiv:2611.01472v1 Announce Type: new Abstract: Causal masking obstructs the column balancing used by optimal-transport (OT) attention. We study a prefix-consistent unbalanced operator and characterize the positional prior it induces. Under bounded scores, for any fixed balancing strength $\rho<1$, its fixed point dilutes attention: every fixed window receives a vanishing share as the context grows. For constant scores, we derive an exact scale-invariant prior. We then classify positional encodings by whether their distant attention mass remains bounded away from one. Discrete window-mass bounds quantify the finite-length dilution of summable log-distance priors near exponent one. Preregistered experiments with 25M- and 100M-parameter language models establish improved loss extrapolation over softmax; exploratory comparisons favor an explicit distance bias at a fraction of the cost. CABLE outperforms RoPE with evaluation-time YaRN on every seed at three training lengths. Changing only the bias beyond the training length moves long-context loss in the predicted direction on all 30 tested checkpoints. Even with exponents constrained above one, many heads approach the summability boundary during training. A margin selected by prior-only calculations for a specified window, length range and growth tolerance controls distant-mass growth in the tested setting. These results connect loss extrapolation to positional decay over the lengths actually evaluated.

## CogBattery: Human-Referenced Evaluation of Cognitive Task Behavior and Structure in Language Models

DevFeed: [CogBattery: Human-Referenced Evaluation of Cognitive Task Behavior and Structure in Language Models](<https://devfeed.tech/articles/cogbattery-human-referenced-evaluation-of-cognitive-task-behavior-and-structure-in-language-models-111910.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.24999>)

Author: Dengzhe Hou, Lingyu Jiang, Fangzhou Lin, Kazunori D Yamada

Published: 2026-11-02T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [AI Models](<https://devfeed.tech/topics/ai-models.md>)

Tags: [behavior](<https://devfeed.tech/tags/behavior.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [human-data](<https://devfeed.tech/tags/human-data.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [structure](<https://devfeed.tech/tags/structure.md>)

### AI overview

CogBattery evaluates 54 open-weight language models on text versions of 13 cognitive tasks against human results from the original studies. Models reproduce a human n-back lure effect but make many more false alarms on ordinary non-targets. Most also show weaker confidence discrimination than humans, and tend to choose the same wager even when they classify accurately. Across tasks, model scores share two dimensions; one is closely tied to scale, but larger models are not consistently more human-like. Predefined cognitive groupings do not predict scores more reliably than random groupings of equal size. The results caution that matching the direction of a human effect does not establish human-like task behavior. The benchmark provides condition-level human references for evaluating errors and confidence in new models.

### Source excerpt

arXiv:2607.24999v2 Announce Type: replace-cross Abstract: Human cognitive tasks let us compare language models with humans under controlled conditions. Yet a model can reproduce the direction of a human effect while making different errors. CogBattery evaluates 54 open-weight language models on text adaptations of 13 cognitive paradigms, comparing each with human data from its source study. Like humans, models respond more to n-back lures than to controls, yet they make far more false alarms on ordinary non-targets. Most models' reported confidence separates correct from incorrect answers less well than humans' does, and most always choose the same wager, including models that classify accurately. These patterns span many model families. Across tasks, model scores share two dimensions. The first is closely tied to scale, yet larger models are not consistently closer to humans. Predefined cognitive groupings show no reliable predictive advantage over random groupings of the same size. These results show that matching human effect directions does not establish human-like behavior. CogBattery provides condition-level human references for testing errors and confidence reports in new models.

## FARO: Function-Space Adaptive Return-to-Risk Optimization via Cross-Block Equal-Risk-Contribution Budget Allocation

DevFeed: [FARO: Function-Space Adaptive Return-to-Risk Optimization via Cross-Block Equal-Risk-Contribution Budget Allocation](<https://devfeed.tech/articles/faro-function-space-adaptive-return-to-risk-optimization-via-cross-block-equal-risk-contribution-budget-allocation-111507.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.40858>)

Author: Xiaojun Wu, Cehao Yang, Xuhui Jiang, Chengjin Xu, Jia Li, Jian Guo

Published: 2026-11-02T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Operations research and optimization](<https://devfeed.tech/topics/operations-research-and-optimization.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [block](<https://devfeed.tech/tags/block.md>), [budget](<https://devfeed.tech/tags/budget.md>), [contribution](<https://devfeed.tech/tags/contribution.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [function](<https://devfeed.tech/tags/function.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [return](<https://devfeed.tech/tags/return.md>), [risk](<https://devfeed.tech/tags/risk.md>), [space](<https://devfeed.tech/tags/space.md>)

### AI overview

FARO treats parameter blocks as the unit for optimizer decisions: it selects each block's geometry using realized counterfactual return, then reallocates a fixed per-block update budget across blocks using equal risk contribution. In GPT-style training on FineWeb, FARO-Adaptive reached 3.2780 nats, close to the hand-designed NorMuonAndAdam result of 3.2794. FARO-ERC further lowered clean-data loss at all six reported seeds, averaging a 0.0136-nat improvement over FARO-Adaptive; the abstract says several statistical checks agreed and the gain exceeded measured run-to-run noise without added time or memory. The reported mechanism was trimming the riskiest blocks. Under a corrupted stream, it still reduced mean loss and won a majority of seeds.

### Source excerpt

arXiv:2610.40858v1 Announce Type: new Abstract: Training a large language model is compute-bound, and current recipes divide the work between optimizers by hand: orthogonalized momentum for attention and MLP matrices, Adam-style updates for the rest. This division rests on human priors about which layer type deserves which geometry, and it fixes a single global update budget that every parameter block shares. We introduce FARO, a function-space adaptive optimizer that makes the partition of parameters into blocks the unit of decision. FARO gives each block a geometry by its realized counterfactual return, then freezes the routes and reallocates the frozen per-block budget by cross-block equal risk contribution (ERC), the risk-parity principle from portfolio theory. On a GPT-style model trained on FineWeb, the framework gives two results. First, its FARO-Adaptive arm reaches 3.2780 nats at the full schedule, matching NorMuonAndAdam (3.2794), the best hand-designed recipe: it recovers the human prior without hand-setting per-layer geometry. Second, FARO-ERC improves on that arm: it lowers the clean-data loss at every one of 6 seeds, by 0.0136 nats on average, a modest but robust gain. Its sign test, paired test, and confidence interval all agree, and it exceeds the measured run-to-run noise at no added time or memory. It works by trimming the riskiest blocks rather than expanding the safest, and it holds up under a corrupted stream, still lowering the mean loss and winning the seed majority. The source code is available at https://github.com/wxj630/FARO.

## Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta Rule

DevFeed: [Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta Rule](<https://devfeed.tech/articles/activation-keyed-momentum-an-anisotropic-momentum-update-via-the-delta-rule-108526.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.19491>)

Author: Euijin Hong, Guannan Qu

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [activation](<https://devfeed.tech/tags/activation.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [awareness](<https://devfeed.tech/tags/awareness.md>), [better](<https://devfeed.tech/tags/better.md>), [block](<https://devfeed.tech/tags/block.md>), [budget](<https://devfeed.tech/tags/budget.md>), [buffer](<https://devfeed.tech/tags/buffer.md>), [builds](<https://devfeed.tech/tags/builds.md>), [can](<https://devfeed.tech/tags/can.md>), [canonical](<https://devfeed.tech/tags/canonical.md>), [cifar-10](<https://devfeed.tech/tags/cifar-10.md>), [code](<https://devfeed.tech/tags/code.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deep](<https://devfeed.tech/tags/deep.md>), [delta](<https://devfeed.tech/tags/delta.md>), [diagnostics](<https://devfeed.tech/tags/diagnostics.md>), [math-oc](<https://devfeed.tech/tags/math-oc.md>), [momentum](<https://devfeed.tech/tags/momentum.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [update](<https://devfeed.tech/tags/update.md>)

### AI overview

Activation-Keyed Momentum (AK-Momentum) makes momentum direction-aware by using each linear layer's input activation as a key and its output-side error as a value in a delta-rule update. The authors say this forgets directions according to how often they appear, corrects for input-side curvature without matrix inversion, and clears stale directions faster than EMA for fixed and drifting optima. They present it as a drop-in momentum-buffer replacement, with no persistent memory and added compute of 22.2-25.0% of a gated-MLP block's linear cost. In FineWeb-Edu pretraining, AK-AdamW reportedly matched AdamW's validation loss in up to 46.64 ± 4.52% fewer steps at 67M and 22.14 ± 0.81% fewer at 370M across three seeds; gains are also reported at 1B, over a tuned Muon baseline, and for SGD, ResNet-18, and ViT-Tiny on CIFAR-10. These are claims from the supplied abstract.

### Source excerpt

arXiv:2608.19491v3 Announce Type: replace Abstract: Most modern optimizers form their momentum as an exponential moving average (EMA) of past gradients, forgetting every direction at one fixed rate. However, the inputs a deep network sees during training can be highly anisotropic, with a few directions queried frequently while most are seen rarely. Preconditioning methods address this anisotropy by wrapping extra processing around this buffer and leave the momentum update itself unchanged. We propose Activation-Keyed Momentum (AK-Momentum), which builds direction-awareness into the momentum update rule. The gradient of a linear layer splits into an input activation that acts as a $\textit{key}$ and an output-side error that acts as a $\textit{value}$. Keying on that activation, AK-Momentum updates the momentum buffer by the canonical delta rule, so each direction is forgotten at a rate set by how often it appears. We prove that it is a valid momentum, that it applies the input-side curvature correction without matrix inversion, and that it clears stale directions faster than EMA under both a fixed and a drifting optimum. It is a drop-in replacement for the momentum buffer of any optimizer, its coefficient transfers across widths under $\mu$P, and its extra compute stays between $22.2\%$ and $25.0\%$ of a gated-MLP block's linear cost with no persistent memory. In FineWeb-Edu pretraining, AdamW with AK-Momentum (AK-AdamW) reaches AdamW's validation loss in up to $46.64 \pm 4.52\%$ fewer steps at 67M and $22.14 \pm 0.81\%$ at 370M over three seeds, and the gain persists at 1B on a Chinchilla-optimal budget. A Muon baseline tuned under the same protocol sits above AK-AdamW at both language-model scales, and the gain holds for SGD, ResNet-18, and ViT-Tiny on CIFAR-10. Training-time diagnostics confirm the predicted mechanism, better gradient tracking and healthier input directions. Code is available at https://github.com/euijinh/AK-Momentum.

## TreeWY: Speculative Verification for Gated DeltaNet Hybrids

DevFeed: [TreeWY: Speculative Verification for Gated DeltaNet Hybrids](<https://devfeed.tech/articles/treewy-speculative-verification-for-gated-deltanet-hybrids-99441.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.20961>)

Author: Sneha Murthy Ghantasala

Published: 2026-10-26T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Model Optimizer](<https://devfeed.tech/topics/model-optimizer.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [infini-attention](<https://devfeed.tech/topics/infini-attention.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [memory](<https://devfeed.tech/tags/memory.md>), [recurrent](<https://devfeed.tech/tags/recurrent.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [time-to-first-token](<https://devfeed.tech/tags/time-to-first-token.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

TreeWY replaces per-draft-position recurrent-state snapshots in Gated DeltaNet layers with a small pseudo-value matrix. A tree-structured WY transform computes draft-node outputs using one triangular solve, then reconstructs only the accepted state when committing; the derivation depends on the gated delta rule. In serving benchmarks on Qwen3.5 35B and 397B, it reduces recurrent-state memory and KV-cache pressure with essentially unchanged acceptance length. Where memory is a bottleneck, the freed HBM enables higher throughput and much lower time-to-first-token; where it is not, the method costs a few percent. It also makes wider, higher-acceptance draft trees affordable, though the abstract reports no throughput gain from that change yet.

### Source excerpt

arXiv:2608.20961v2 Announce Type: replace-cross Abstract: Modern open models are hybrids: most layers are linear-attention (Gated DeltaNet, GDN) layers carrying a small fixed-size recurrent state instead of a growing key-value (KV) cache. This makes ordinary decoding memory-efficient, but hurts speculative decoding. To verify a batch of draft tokens and then roll back the rejected ones, today's systems snapshot the full recurrent state at every draft position for GDN layers, and those snapshots cannot be shared across branches of a draft tree, so a wide, high-acceptance tree becomes memory-infeasible. We remove the snapshots. Using a tree-structured WY transform of the gated delta rule, we compute every draft node's output with a single triangular solve and reconstruct only the one accepted state on commit, storing a small pseudo-value matrix instead of per-node states; the derivation depends only on the gated delta rule, not on any other architectural detail. In serving benchmarks on two scales of one hybrid model family (Qwen3.5 35B and 397B) this cuts speculative recurrent-state memory and KV-cache pressure at essentially identical acceptance length, turning the freed HBM into higher throughput and much lower time-to-first-token (TTFT) wherever memory binds, and costing a few percent where it does not. For tree width the same memory buys affordability: a wider, higher-acceptance draft becomes possible, though not yet a throughput win.

## Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping

DevFeed: [Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping](<https://devfeed.tech/articles/accordion-elastic-depth-inference-via-fixed-prefix-layer-looping-99621.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.32492>)

Author: Yudong Zhang, Wei Ding, Kaili Zheng, Yuchen Ding, Zhenyu Zhang

Published: 2026-10-26T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [post-training](<https://devfeed.tech/topics/post-training.md>)

Tags: [accordion](<https://devfeed.tech/tags/accordion.md>), [budget](<https://devfeed.tech/tags/budget.md>), [code](<https://devfeed.tech/tags/code.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [domain](<https://devfeed.tech/tags/domain.md>), [elastic](<https://devfeed.tech/tags/elastic.md>), [execution](<https://devfeed.tech/tags/execution.md>), [experts](<https://devfeed.tech/tags/experts.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [flexible](<https://devfeed.tech/tags/flexible.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [layer](<https://devfeed.tech/tags/layer.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [model](<https://devfeed.tech/tags/model.md>), [next-token-prediction](<https://devfeed.tech/tags/next-token-prediction.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>)

### AI overview

Accordion is a mixture-of-experts model offering five inference budgets through fixed-prefix layer looping; its minimum budget uses the unlooped model. Pretraining samples one budget per micro-batch and optimizes next-token prediction without a cross-budget consistency objective. Fine-tuning and reinforcement learning preserve the budgets while merging three domain experts through parameter-group merging and budget-matched multi-teacher on-policy distillation. After 3.6T pretraining tokens, greater inference depth raised the model's five-category average across 37 tasks by 3.01 percentage points; the middle budget captured 89% of that gain with half as many extra layer executions. The final model exceeded each expert on its corresponding task-and-budget average, with depth-dependent gains in math reasoning and code generation.

### Source excerpt

arXiv:2610.32492v1 Announce Type: new Abstract: Large language models serve applications with diverse inference budgets, but training and maintaining separate models for each budget incurs substantial costs. Recent Looped Transformers enable flexible inference computation through parameter reuse, but maintaining strong performance across different inference budgets remains an open challenge. We introduce Accordion, a mixture-of-experts-based model that supports five inference budgets through fixed-prefix layer looping. Each budget follows a pre-defined execution path, while the minimum budget retains the unlooped model. Joint pretraining samples one budget per micro-batch and optimizes its next-token prediction loss without additional cross-budget consistency objectives. We further preserve these inference budgets through supervised fine-tuning and reinforcement learning, where parameter-group merging and budget-matched multi-teacher on-policy distillation integrate three domain experts into one model. After 3.6T tokens of pretraining, increasing inference depth improves Accordion-6B-A0.5B's five-category macro-average across 37 tasks by 3.01 percentage points, with the middle budget capturing 89% of this gain using half as many additional layer executions. The final model surpasses each domain expert on its corresponding task-and-budget average, while achieving strong instruction-following performance across all budgets and retaining depth-dependent gains in mathematical reasoning and code generation.

## An Optimal Control Approach To Transformer Training

DevFeed: [An Optimal Control Approach To Transformer Training](<https://devfeed.tech/articles/an-optimal-control-approach-to-transformer-training-96823.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2603.09571>)

Author: Ka\\u{g}an Akman, Naci Saldi, Serdar Y\\"uksel

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>)

Tags: [consistency](<https://devfeed.tech/tags/consistency.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [dynamic-programming](<https://devfeed.tech/tags/dynamic-programming.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [markov](<https://devfeed.tech/tags/markov.md>), [math-oc](<https://devfeed.tech/tags/math-oc.md>), [optimal-control](<https://devfeed.tech/tags/optimal-control.md>), [robust](<https://devfeed.tech/tags/robust.md>), [theoretical](<https://devfeed.tech/tags/theoretical.md>), [training](<https://devfeed.tech/tags/training.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

The paper models Transformer training as a discrete-time controlled particle system with shared actions, then lifts it to a fully observed MDP over probability measures. Positional encodings are included in the state to preserve sequence order. The authors establish globally optimal policies under mild compactness assumptions and show the lifted closed-loop policies correspond to initial-distribution-dependent open-loop policies that are realized-input-independent. Their proposed training method quantizes the state, probability-measure, and action spaces; optimal policies for the quantized model are shown to be near-optimal for the original problem. The abstract also claims value-function stability to perturbations in initial empirical measures and policy convergence as data size grows. These are theoretical results; the supplied text does not describe empirical comparisons with gradient-based training.

### Source excerpt

arXiv:2603.09571v2 Announce Type: replace Abstract: In this paper, we develop a rigorous optimal control-theoretic approach to Transformer training that respects key structural constraints such as (i) realized-input-independence during execution, (ii) the ensemble control nature of the problem, and (iii) positional dependence. We model the Transformer architecture as a discrete-time controlled particle system with shared actions, exhibiting noise-free controlled McKean--Vlasov dynamics, and we show that lifting the dynamics to probability measures produces a fully-observed Markov decision process (MDP). Positional encodings are incorporated into the state space to preserve the sequence order under lifting. Using the dynamic programming principle, we establish the existence of globally optimal policies under mild assumptions of compactness. We further prove that closed-loop policies in the lifted problem is equivalent to an initial-distribution dependent open-loop policy, which are realized-input-independent and compatible with standard Transformer training. To train a Transformer, we propose a triply quantized training procedure for the lifted MDP by quantizing the state space, the space of probability measures, and the action space, and show that any optimal policy for the triply quantized model is near-optimal for the original training problem. Finally, we establish stability and empirical consistency properties of the lifted model by showing that the value function is continuous with respect to the perturbations of the initial empirical measures and convergence of policies as the data size increases. This approach provides a globally optimal and robust theoretical alternative to gradient-based training without requiring smoothness or convexity.

## Reader--Writer Coupling in Transformer MLPs

DevFeed: [Reader--Writer Coupling in Transformer MLPs](<https://devfeed.tech/articles/reader-writer-coupling-in-transformer-mlps-94910.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.29340>)

Author: Giyeong Oh, Sunghee Ahn, Youngjae Yu

Published: 2026-10-22T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [infini-attention](<https://devfeed.tech/topics/infini-attention.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>)

Tags: [block](<https://devfeed.tech/tags/block.md>), [channel](<https://devfeed.tech/tags/channel.md>), [core](<https://devfeed.tech/tags/core.md>), [coupling](<https://devfeed.tech/tags/coupling.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [layer](<https://devfeed.tech/tags/layer.md>), [memory](<https://devfeed.tech/tags/memory.md>), [mlp](<https://devfeed.tech/tags/mlp.md>), [models](<https://devfeed.tech/tags/models.md>), [pairing](<https://devfeed.tech/tags/pairing.md>), [stream](<https://devfeed.tech/tags/stream.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [training](<https://devfeed.tech/tags/training.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [writer](<https://devfeed.tech/tags/writer.md>)

### AI overview

The authors measure reader-writer coupling in transformer MLP channels using the exact Jacobian of an MLP's residual-stream update. A channel's signed pairing coordinate comes from the inner product between its reader and writer terms. Across the measured blocks, strongly coupled channels form a small core that is nearly all opposed where initialization allows either sign equally: writers point against the directions their readers respond to. This pattern appears in small models trained from scratch and released checkpoints, emerges within a few percent of training, and strengthens thereafter. Most channels' contributions are taken up by readers in later blocks, while few appear in logits. Flipping opposed writers moves outputs toward common tokens; permuting a writer onto another reader leads the writer to regrow to fit that reader while the reader stays.

### Source excerpt

arXiv:2610.29340v1 Announce Type: new Abstract: A transformer MLP gives each hidden channel two roles: a reader that decides when the channel responds, and a writer that decides what it adds back to the residual stream. Read as a key--value memory, the layer can be measured one factor set at a time: the pairing of readers to writers is carried only by their shared index, invisible to any quantity computed from one factor set alone. Yet training couples the two factors through their joint update, so we ask what reading them together shows. The block supplies the joint instrument: the MLP writes back into the very stream it reads, so the exact Jacobian of its update is square, and its agreement with its own transpose resolves into reader--writer terms whose inner products give each channel, read as a couple, a signed pairing coordinate. Ranked by this coordinate, each block we measure has a small core of strongly coupled channels; where initialization gives either sign equally, the core is nearly all opposed: each writer points against the direction its reader responds to. This opposed core recurs from the smallest models trained from scratch to released checkpoints; its sign is in place within a few percent of training and holds while the coupling grows. Read as couples, most channels return their contribution against the reader that fired it, and the readers of later blocks take it up; few show in the logits. Training holds the couples in place: flipping the opposed writers moves the output toward common tokens, and a writer permuted onto another reader regrows to fit that reader while the reader stays.

## Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation

DevFeed: [Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation](<https://devfeed.tech/articles/truncate-or-not-a-scaling-study-of-looped-transformers-with-truncated-backpropagation-93590.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.26030>)

Author: Xuyuan Xiong, Siyu Chen, Zihan Shao, Yue Su, Weiyang Jin, Ping Luo, Zhuoran Yang, Xihui Liu

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [backpropagation](<https://devfeed.tech/tags/backpropagation.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [loops](<https://devfeed.tech/tags/loops.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [model](<https://devfeed.tech/tags/model.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [study](<https://devfeed.tech/tags/study.md>), [training](<https://devfeed.tech/tags/training.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [validation](<https://devfeed.tech/tags/validation.md>)

### AI overview

Looped transformers reuse a shared layer block to add computational depth, but truncating backpropagation through time saves compute at the cost of biased gradients. Across nearly 600 runs, the fitted scaling law favors full backpropagation when forward and backward loop counts are optimized together; at matched compute, full backpropagation through fewer loops consistently beats truncated settings. At fixed model size, the compute-optimal loop count and token budget rise with compute. When model size is also optimized, two fully trained loops perform best among looped settings: they achieve lower validation loss than the compute-optimal non-looped model with about 27% fewer non-embedding parameters and 8% more training compute.

### Source excerpt

arXiv:2610.26030v2 Announce Type: new Abstract: Looped transformers reuse a shared block of layers to increase computational depth without increasing the number of trainable parameters. They are trained with backpropagation through time (BPTT). To save training compute, BPTT is often truncated to the last few loops, but truncation yields biased gradients that may weaken the benefit of looping. In this work, we study the value of truncation in looped transformer training in a systematic way: we jointly vary model size, training tokens, and forward and backward loop counts across nearly 600 training runs and fit a unified scaling law that captures the effects of looping and truncation through effective parameters. Despite the training-compute savings of truncation, our fitted law favors full BPTT when forward and backward loop counts are jointly optimized. In matched-compute runs, full BPTT through fewer loops consistently outperforms truncated settings of the same compute. The law further shows that at fixed model size, the compute-optimal loop count and training-token budget both grow with compute. When model size is also free, the law identifies two fully trained loops as the best looped setting. Experimentally, this setting achieves lower validation loss than the compute-optimal non-looped model with about 27% fewer non-embedding parameters and only 8% more training compute.

## Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation

DevFeed: [Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation](<https://devfeed.tech/articles/tokens-per-parameter-coverage-is-critical-for-robust-llm-scaling-law-extrapolation-94123.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.08541>)

Author: Joshua Shay Kricheli, Alexander Lawrence Reid, Soumajyoti Sarkar, Venkata Gandikota, Paulo Shakarian

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>)

Tags: [3](<https://devfeed.tech/tags/3.md>), [compute](<https://devfeed.tech/tags/compute.md>), [condition](<https://devfeed.tech/tags/condition.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [design](<https://devfeed.tech/tags/design.md>), [diversity](<https://devfeed.tech/tags/diversity.md>), [gap](<https://devfeed.tech/tags/gap.md>), [llm](<https://devfeed.tech/tags/llm.md>), [newton](<https://devfeed.tech/tags/newton.md>), [parameter](<https://devfeed.tech/tags/parameter.md>), [point](<https://devfeed.tech/tags/point.md>), [ratio](<https://devfeed.tech/tags/ratio.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>), [square](<https://devfeed.tech/tags/square.md>), [sufficient](<https://devfeed.tech/tags/sufficient.md>), [token](<https://devfeed.tech/tags/token.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

Fitting scaling laws from runs at a single tokens-per-parameter ratio can make parameter and token effects difficult to distinguish, especially when their exponents are nearly equal. The abstract reports that this collinear design makes Gauss-Newton estimation ill-conditioned, inflates coefficient confidence intervals by an order of magnitude or more, and harms extrapolation away from the training ray. The authors prove the result for four scaling-law formalisms and give a necessary-and-sufficient TPP-diversity threshold for well-conditioned estimation. In held-out comparisons, non-collinear designs won 97.3% of the time across four laws, five corpora, and multiple floating-point precision modes. The abstract attributes the issue to Jacobian geometry, extending it to smooth objectives whose curvature involves the Jacobian.

### Source excerpt

arXiv:2605.08541v3 Announce Type: replace Abstract: Neural scaling laws approximate a language model's loss as a power-law function of parameter count $N$ and token count $D$. Following Chinchilla-style compute-optimal training, many studies fit scaling laws from runs performed under a fixed tokens-per-parameter (TPP) ratio $k$ and set $D = kN$. We show that this collinear design, combined with the empirically common near-equality of the exponents governing $N$ and $D$, induces an inherent ill-conditioning in the Gauss-Newton least-squares problem: the condition number of the design grows as the inverse square of the gap between the $N$ and $D$-exponents. The scale coefficients become practically unidentifiable, with confidence intervals inflating by an order of magnitude or more, yielding a ``sloppy'' model whose extrapolations degrade sharply off the training ray. We prove this for four scaling-law formalisms and derive a closed-form TPP-diversity threshold that is necessary and sufficient for well-conditioned estimation. Empirically, non-collinear designs outperform collinear ones on held-out splits with a 97.3\% win rate across four laws, five corpora, multiple floating point precision modes. We further show the degeneracy is rooted in Jacobian geometry and is not an artifact of the loss function: any smooth estimation objective whose curvature involves the Jacobian inherits the same ill-conditioning.

[Next page](<https://devfeed.tech/topics/chinchilla-scaling-law.md?cursor=WyIyMDI2LTEwLTIxVDA0OjAwOjAwKzAwOjAwIiwgIjMxODllY2RiLTg5ZDUtNGU5MS04OWE1LWU2NWI2NTcyZWQwYSJd>)