# scaling laws Scaling laws describe how neural language model performance and training behavior vary with model size, dataset size, and compute. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## A study of sequence weighting at scale DevFeed: [A study of sequence weighting at scale]() Original publisher: [Read original article]() Author: Alex Renda Published: 2026-09-14T00:00:00Z Content type: article Language: en Sources: [Jane Street]() Topics: [scaling laws](), [datasets](), [Deep neural networks](), [Chinchilla scaling law]() Tags: [data](), [machine-learning](), [model](), [scaling-laws](), [training]() ### AI overview The article studies how sequence weighting affects language-model training across model scales. It reports non-monotonic behavior: small-to-medium models increasingly learn data-specific patterns in proportion to sequence weights, while large models can learn all patterns in the data more independently of those weights. ### Source excerpt TL;DR: We study the scaling laws of data weighting across in-house and open-weight LMs, finding non-monotonic behavior across scales. We vary the weight assigned to sequences during training and measure how strongly the model's loss reduction on a sequence depends on the sequence's weight. Taken together, our results are consistent with a general trend: as models transition from small to medium scale, they transition from learning general patterns independent of data weight to learning data-specific patterns proportional to the data weights. As models then transition from medium to large scale they are able to learn all patterns present in the data, once again independent of data weight. ## ATLAS: Practical scaling laws for multilingual models DevFeed: [ATLAS: Practical scaling laws for multilingual models]() Original publisher: [Read original article]() Published: 2026-01-27T18:58:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [scaling laws](), [Language models](), [Training AI Models](), [data](), [Google](), [datasets](), [Artificial Intelligence]() Tags: [ai](), [atlas](), [data](), [generative-ai](), [global](), [google](), [language-models](), [machine-intelligence](), [models](), [natural-language-processing](), [research](), [scaling-laws](), [train]() ### AI overview ATLAS introduces adaptive scaling laws for training multilingual language models. Based on 774 runs across 400+ languages and evaluations in 48 languages, it helps practitioners choose model size, data volume, and language mixtures while optimizing performance for a target language. ### Source excerpt Generative AI ## S1 paper explains inference-time scaling by extending an LLM's reasoning DevFeed: [S1 paper explains inference-time scaling by extending an LLM's reasoning]() Original publisher: [Read original article]() Published: 2025-02-03T00:00:00Z Content type: opinion Language: en Sources: [Tim Kellogg]() Topics: [Artificial Intelligence](), [Large Language Model](), [Inference](), [scaling laws](), [OpenAI]() Tags: [ai](), [inference](), [llm](), [openai](), [paper](), [scaling-laws](), [tokens](), [training]() ### AI overview This commentary discusses the S1 paper, which describes a model that is slightly below state of the art but can run on the author's laptop. It focuses on inference-time scaling and the paper's explanation of extending an LLM's reasoning by forcing it to continue after it tries to stop, along with connections to token selection and Entropix. ### Source excerpt A new paper released on Friday is making waves in the AI community, not because of the model it describes, but because it shows how close we are to some very large breakthroughs in AI. The model is just below state of the art, but it can run on my laptop. More important, it sheds light on how all this stuff works, and it's not complicated. ## A failed experiment: Infini-Attention, and why we should keep trying? DevFeed: [A failed experiment: Infini-Attention, and why we should keep trying?]() Original publisher: [Read original article]() Author: Phuc Nguyen; Leandro von Werra; Thomas Wolf Published: 2024-08-14T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [infini-attention](), [long-context](), [Language models](), [scaling laws](), [AI Research]() Tags: [complexity](), [compute](), [context](), [gpu](), [infini-attention](), [language-models](), [llama](), [long-context](), [memory-compression](), [scaling-laws]() ### AI overview This article examines Infini-attention as an alternative approach for extending language-model context lengths. It explains the quadratic memory cost of standard self-attention, the limitations of Flash Attention and Ring Attention, and the resource demands of training models with million-token contexts. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science.