# A study of sequence weighting at scale

DevFeed: [A study of sequence weighting at scale](<https://devfeed.tech/articles/a-study-of-sequence-weighting-at-scale-20145.md>)

Original publisher: [Read original article](<https://blog.janestreet.com/a-study-of-sequence-weighting-at-scale/>)

Author: Alex Renda

Published: 2026-09-14T00:00:00Z

Content type: article

Language: en

Sources: [Jane Street](<https://devfeed.tech/sources/jane-street.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>)

Tags: [data](<https://devfeed.tech/tags/data.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [model](<https://devfeed.tech/tags/model.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

The article studies how sequence weighting affects language-model training across model scales. It reports non-monotonic behavior: small-to-medium models increasingly learn data-specific patterns in proportion to sequence weights, while large models can learn all patterns in the data more independently of those weights.

## Source excerpt

TL;DR: We study the scaling laws of data weighting across in-house and open-weight LMs, finding non-monotonic behavior across scales. We vary the weight assigned to sequences during training and measure how strongly the model's loss reduction on a sequence depends on the sequence's weight. Taken together, our results are consistent with a general trend: as models transition from small to medium scale, they transition from learning general patterns independent of data weight to learning data-specific patterns proportional to the data weights. As models then transition from medium to large scale they are able to learn all patterns present in the data, once again independent of data weight.