# Chinchilla scaling law

A compute-optimal training scaling law for transformer language models that relates model size and training-token count under a fixed compute budget.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## A study of sequence weighting at scale

DevFeed: [A study of sequence weighting at scale](<https://devfeed.tech/articles/a-study-of-sequence-weighting-at-scale-20145.md>)

Original publisher: [Read original article](<https://blog.janestreet.com/a-study-of-sequence-weighting-at-scale/>)

Author: Alex Renda

Published: 2026-09-14T00:00:00Z

Content type: article

Language: en

Sources: [Jane Street](<https://devfeed.tech/sources/jane-street.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>)

Tags: [data](<https://devfeed.tech/tags/data.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [model](<https://devfeed.tech/tags/model.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

The article studies how sequence weighting affects language-model training across model scales. It reports non-monotonic behavior: small-to-medium models increasingly learn data-specific patterns in proportion to sequence weights, while large models can learn all patterns in the data more independently of those weights.

### Source excerpt

TL;DR: We study the scaling laws of data weighting across in-house and open-weight LMs, finding non-monotonic behavior across scales. We vary the weight assigned to sequences during training and measure how strongly the model's loss reduction on a sequence depends on the sequence's weight. Taken together, our results are consistent with a general trend: as models transition from small to medium scale, they transition from learning general patterns independent of data weight to learning data-specific patterns proportional to the data weights. As models then transition from medium to large scale they are able to learn all patterns present in the data, once again independent of data weight.

## Open-R1: a fully open reproduction of DeepSeek-R1

DevFeed: [Open-R1: a fully open reproduction of DeepSeek-R1](<https://devfeed.tech/articles/open-r1-a-fully-open-reproduction-of-deepseek-r1-7416.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/open-r1>)

Author: Elie Bakouch; Leandro von Werra; Lewis Tunstall

Published: 2025-01-28T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [deepseek](<https://devfeed.tech/topics/deepseek.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [moe](<https://devfeed.tech/topics/moe.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [AI Development](<https://devfeed.tech/topics/ai-development.md>)

Tags: [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [llms](<https://devfeed.tech/tags/llms.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [research](<https://devfeed.tech/tags/research.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

The article introduces Open-R1, an open-source project to reproduce and extend DeepSeek-R1's reasoning-model training pipeline. It discusses reinforcement learning, reasoning-specific datasets, training-code transparency, scaling laws, and the DeepSeek-V3 Mixture of Experts foundation model.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.