# How to shuffle a big dataset

DevFeed: [How to shuffle a big dataset](<https://devfeed.tech/articles/how-to-shuffle-a-big-dataset-20178.md>)

Original publisher: [Read original article](<https://blog.janestreet.com/how-to-shuffle-a-big-dataset/>)

Author: Chris Hardin

Published: 2018-09-26T00:00:00Z

Content type: tutorial

Language: en

Sources: [Jane Street](<https://devfeed.tech/sources/jane-street.md>)

Topics: [datasets](<https://devfeed.tech/topics/datasets.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>)

## AI overview

A Jane Street engineering tutorial explains how to shuffle datasets too large to fit in RAM. It presents a two-pass algorithm that partitions data into piles, shuffles each pile in memory, and appends the results, with linear time under stated assumptions.

## Source excerpt

At Jane Street, we often work with data that has a very low signal-to-noise ratio, but fortunately we also have a lot of data. Where practitioners in many fields might be accustomed to having tens or hundreds of thousands of correctly labeled examples, some of our problems are more like having a billion training examples whose labels have only a slight tendency to be correct. These large datasets present a number of interesting engineering challenges. The one we address here: How do you shuffle a really large dataset? (If you're not familiar with why one might need this, jump to the section Why shuffle below.)