# Streaming datasets: 100x More Efficient

DevFeed: [Streaming datasets: 100x More Efficient](<https://devfeed.tech/articles/streaming-datasets-100x-more-efficient-7493.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/streaming-datasets>)

Author: Andres Marafioti; Quentin Lhoest; ben burtenshaw; Pedro Cuenca; merve

Published: 2025-10-27T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [datasets](<https://devfeed.tech/topics/datasets.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Caching](<https://devfeed.tech/topics/caching.md>), [Library](<https://devfeed.tech/topics/library.md>)

Tags: [caching](<https://devfeed.tech/tags/caching.md>), [dataloaders](<https://devfeed.tech/tags/dataloaders.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [developers](<https://devfeed.tech/tags/developers.md>), [hub](<https://devfeed.tech/tags/hub.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [performance](<https://devfeed.tech/tags/performance.md>), [s3](<https://devfeed.tech/tags/s3.md>), [scale](<https://devfeed.tech/tags/scale.md>), [storage](<https://devfeed.tech/tags/storage.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [xet](<https://devfeed.tech/tags/xet.md>)

## AI overview

The article presents major improvements to streaming datasets at terabyte scale, allowing training to begin without downloading data locally. The changes reduce startup requests by up to 100x, resolve data files 10x faster, double streaming speed, and improve efficiency for highly concurrent workers.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.