# Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation

DevFeed: [Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation](<https://devfeed.tech/articles/tokens-per-parameter-coverage-is-critical-for-robust-llm-scaling-law-extrapolation-94123.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.08541>)

Author: Joshua Shay Kricheli, Alexander Lawrence Reid, Soumajyoti Sarkar, Venkata Gandikota, Paulo Shakarian

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>)

Tags: [3](<https://devfeed.tech/tags/3.md>), [compute](<https://devfeed.tech/tags/compute.md>), [condition](<https://devfeed.tech/tags/condition.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [design](<https://devfeed.tech/tags/design.md>), [diversity](<https://devfeed.tech/tags/diversity.md>), [gap](<https://devfeed.tech/tags/gap.md>), [llm](<https://devfeed.tech/tags/llm.md>), [newton](<https://devfeed.tech/tags/newton.md>), [parameter](<https://devfeed.tech/tags/parameter.md>), [point](<https://devfeed.tech/tags/point.md>), [ratio](<https://devfeed.tech/tags/ratio.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>), [square](<https://devfeed.tech/tags/square.md>), [sufficient](<https://devfeed.tech/tags/sufficient.md>), [token](<https://devfeed.tech/tags/token.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

Fitting scaling laws from runs at a single tokens-per-parameter ratio can make parameter and token effects difficult to distinguish, especially when their exponents are nearly equal. The abstract reports that this collinear design makes Gauss-Newton estimation ill-conditioned, inflates coefficient confidence intervals by an order of magnitude or more, and harms extrapolation away from the training ray. The authors prove the result for four scaling-law formalisms and give a necessary-and-sufficient TPP-diversity threshold for well-conditioned estimation. In held-out comparisons, non-collinear designs won 97.3% of the time across four laws, five corpora, and multiple floating-point precision modes. The abstract attributes the issue to Jacobian geometry, extending it to smooth objectives whose curvature involves the Jacobian.

## Source excerpt

arXiv:2605.08541v3 Announce Type: replace Abstract: Neural scaling laws approximate a language model's loss as a power-law function of parameter count $N$ and token count $D$. Following Chinchilla-style compute-optimal training, many studies fit scaling laws from runs performed under a fixed tokens-per-parameter (TPP) ratio $k$ and set $D = kN$. We show that this collinear design, combined with the empirically common near-equality of the exponents governing $N$ and $D$, induces an inherent ill-conditioning in the Gauss-Newton least-squares problem: the condition number of the design grows as the inverse square of the gap between the $N$ and $D$-exponents. The scale coefficients become practically unidentifiable, with confidence intervals inflating by an order of magnitude or more, yielding a ``sloppy'' model whose extrapolations degrade sharply off the training ray. We prove this for four scaling-law formalisms and derive a closed-form TPP-diversity threshold that is necessary and sufficient for well-conditioned estimation. Empirically, non-collinear designs outperform collinear ones on held-out splits with a 97.3\% win rate across four laws, five corpora, multiple floating point precision modes. We further show the degeneracy is rooted in Jacobian geometry and is not an artifact of the loss function: any smooth estimation objective whose curvature involves the Jacobian inherits the same ill-conditioning.