# From Files to Chunks: Improving HF Storage Efficiency

DevFeed: [From Files to Chunks: Improving HF Storage Efficiency](<https://devfeed.tech/articles/from-files-to-chunks-improving-hf-storage-efficiency-7206.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/from-files-to-chunks>)

Author: Jared Sulzdorf; Ann Huang

Published: 2024-11-20T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [xet](<https://devfeed.tech/topics/xet.md>), [content defined chunking](<https://devfeed.tech/topics/content-defined-chunking.md>), [content addressed store](<https://devfeed.tech/topics/content-addressed-store.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [Git](<https://devfeed.tech/topics/git.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [content-addressed-store](<https://devfeed.tech/tags/content-addressed-store.md>), [content-defined-chunking](<https://devfeed.tech/tags/content-defined-chunking.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [dedupe](<https://devfeed.tech/tags/dedupe.md>), [git](<https://devfeed.tech/tags/git.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [performance](<https://devfeed.tech/tags/performance.md>), [storage](<https://devfeed.tech/tags/storage.md>), [xet](<https://devfeed.tech/tags/xet.md>)

## AI overview

Hugging Face's Xet team describes a storage system that splits files into variable-sized chunks using content-defined chunking and a rolling hash. Chunks are stored in a content-addressed store with deduplication, so updates upload only new chunks. The article reports a consistent 50% improvement in storage and transfer performance compared with Git LFS across three iterative development use cases, including the CORD-19 dataset.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.