# scaling laws Published articles for scaling laws. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## A study of sequence weighting at scale DevFeed: [A study of sequence weighting at scale]() Original publisher: [Read original article]() Author: Alex Renda Published: 2026-09-14T00:00:00Z Content type: article Language: en Sources: [Jane Street]() Topics: [scaling laws](), [datasets](), [Deep neural networks](), [Chinchilla scaling law]() Tags: [data](), [machine-learning](), [model](), [scaling-laws](), [training]() ### AI overview The article studies how sequence weighting affects language-model training across model scales. It reports non-monotonic behavior: small-to-medium models increasingly learn data-specific patterns in proportion to sequence weights, while large models can learn all patterns in the data more independently of those weights. ### Source excerpt TL;DR: We study the scaling laws of data weighting across in-house and open-weight LMs, finding non-monotonic behavior across scales. We vary the weight assigned to sequences during training and measure how strongly the model's loss reduction on a sequence depends on the sequence's weight. Taken together, our results are consistent with a general trend: as models transition from small to medium scale, they transition from learning general patterns independent of data weight to learning data-specific patterns proportional to the data weights. As models then transition from medium to large scale they are able to learn all patterns present in the data, once again independent of data weight. ## From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta's Ads Ranking DevFeed: [From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta's Ads Ranking]() Original publisher: [Read original article]() Author: Steven De Gryze; Parshva Doshi; Sean O'Byrne; Arnold Overwijk; Dinesh Ramasamy; Lee Xiong Published: 2026-08-05T19:20:20Z Content type: article Language: en Sources: [Engineering at Meta](), [Meta ML Applications]() Topics: [recommendation systems](), [Temporal data]() Tags: [ads](), [architecture](), [data-infrastructure](), [meta](), [ml-applications](), [production-engineering](), [recommendation-systems](), [scaling-laws](), [tokenization]() ### AI overview Meta describes a multi-stage sequence-model architecture for ads ranking that separates offline user modeling from lightweight online ranking. It also uses dense tokenization and target-aware attention to learn feature interactions, with reported conversion and ad-click lifts across Instagram and Facebook. ### Source excerpt Every day, Meta's recommendation platforms handle billions of user interactions, generating rich temporal signals that capture individual preferences and intent across products, ads, and content. In our 2024 post on sequence learning for ads recommendations, we showed how modeling the order and timing of user actions (rather than relying on static, manually engineered sparse features) [...] Read More... The post From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta's Ads Ranking appeared first on Engineering at Meta. ## Making LLMs faster without sacrificing accuracy DevFeed: [Making LLMs faster without sacrificing accuracy]() Original publisher: [Read original article]() Author: Tao Yu; Youngsuk Park Published: 2026-05-15T13:00:00Z Content type: article Language: en Sources: [Amazon Science homepage]() Topics: [Language models](), [Training AI Models](), [web applications]() Tags: [architecture](), [chinchilla-scaling-law](), [conversational-ai](), [generative-ai](), [grouped-query-attention](), [hyperparameter-optimization](), [iclr-2026](), [inference](), [inference-efficiency](), [large-language-models](), [large-language-models-llms](), [llm-optimization](), [llms](), [machine-learning](), [model-architecture](), [network-architectures](), [scaling-laws](), [training](), [transformer-architecture]() ### AI overview The article presents scaling laws that connect LLM architectural choices to the tradeoff between accuracy and efficiency. It describes how these choices can improve inference throughput without reducing accuracy. ### Source excerpt A new scaling law that relates particular architectural choices to loss helps identify models that improve throughput by up to 47% with no loss of accuracy. ## ATLAS: Practical scaling laws for multilingual models DevFeed: [ATLAS: Practical scaling laws for multilingual models]() Original publisher: [Read original article]() Published: 2026-01-27T18:58:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [scaling laws](), [Language models](), [Training AI Models](), [data](), [Google](), [datasets](), [Artificial Intelligence]() Tags: [ai](), [atlas](), [data](), [generative-ai](), [global](), [google](), [language-models](), [machine-intelligence](), [models](), [natural-language-processing](), [research](), [scaling-laws](), [train]() ### AI overview ATLAS introduces adaptive scaling laws for training multilingual language models. Based on 774 runs across 400+ languages and evaluations in 48 languages, it helps practitioners choose model size, data volume, and language mixtures while optimizing performance for a target language. ### Source excerpt Generative AI ## S1 paper explains inference-time scaling by extending an LLM's reasoning DevFeed: [S1 paper explains inference-time scaling by extending an LLM's reasoning]() Original publisher: [Read original article]() Published: 2025-02-03T00:00:00Z Content type: opinion Language: en Sources: [Tim Kellogg]() Topics: [Artificial Intelligence](), [Large Language Model](), [Inference](), [scaling laws](), [OpenAI]() Tags: [ai](), [inference](), [llm](), [openai](), [paper](), [scaling-laws](), [tokens](), [training]() ### AI overview This commentary discusses the S1 paper, which describes a model that is slightly below state of the art but can run on the author's laptop. It focuses on inference-time scaling and the paper's explanation of extending an LLM's reasoning by forcing it to continue after it tries to stop, along with connections to token selection and Entropix. ### Source excerpt A new paper released on Friday is making waves in the AI community, not because of the model it describes, but because it shows how close we are to some very large breakthroughs in AI. The model is just below state of the art, but it can run on my laptop. More important, it sheds light on how all this stuff works, and it's not complicated. ## Open-R1: a fully open reproduction of DeepSeek-R1 DevFeed: [Open-R1: a fully open reproduction of DeepSeek-R1]() Original publisher: [Read original article]() Author: Elie Bakouch; Leandro von Werra; Lewis Tunstall Published: 2025-01-28T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [deepseek](), [Open Source](), [Reinforcement learning](), [Large Language Model](), [moe](), [Training AI Models](), [Chinchilla scaling law](), [datasets](), [AI Development]() Tags: [artificial-intelligence](), [datasets](), [deepseek](), [llms](), [mixture-of-experts-moe](), [open-source](), [reasoning](), [reinforcement-learning](), [research](), [scaling-laws](), [training]() ### AI overview The article introduces Open-R1, an open-source project to reproduce and extend DeepSeek-R1's reasoning-model training pipeline. It discusses reinforcement learning, reasoning-specific datasets, training-code transparency, scaling laws, and the DeepSeek-V3 Mixture of Experts foundation model. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## A failed experiment: Infini-Attention, and why we should keep trying? DevFeed: [A failed experiment: Infini-Attention, and why we should keep trying?]() Original publisher: [Read original article]() Author: Phuc Nguyen; Leandro von Werra; Thomas Wolf Published: 2024-08-14T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [infini-attention](), [long-context](), [Language models](), [scaling laws](), [AI Research]() Tags: [complexity](), [compute](), [context](), [gpu](), [infini-attention](), [language-models](), [llama](), [long-context](), [memory-compression](), [scaling-laws]() ### AI overview This article examines Infini-attention as an alternative approach for extending language-model context lengths. It explains the quadratic memory cost of standard self-attention, the limitations of Flash Attention and Ring Attention, and the resource demands of training models with million-token contexts. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science.