# From Chunks to Blocks: Accelerating Uploads and Downloads on the Hub

DevFeed: [From Chunks to Blocks: Accelerating Uploads and Downloads on the Hub](<https://devfeed.tech/articles/from-chunks-to-blocks-accelerating-uploads-and-downloads-on-the-hub-7205.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/from-chunks-to-blocks>)

Author: Jared Sulzdorf; yuchenglow; Zach Nation; saba noorassa

Published: 2025-02-12T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [xet](<https://devfeed.tech/topics/xet.md>), [content addressed store](<https://devfeed.tech/topics/content-addressed-store.md>), [AI, ML & Data Engineering](<https://devfeed.tech/topics/ai-ml-data-engineering.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [cas](<https://devfeed.tech/tags/cas.md>), [collaboration](<https://devfeed.tech/tags/collaboration.md>), [content-addressed-store](<https://devfeed.tech/tags/content-addressed-store.md>), [content-defined-chunking](<https://devfeed.tech/tags/content-defined-chunking.md>), [dedupe](<https://devfeed.tech/tags/dedupe.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [integration](<https://devfeed.tech/tags/integration.md>), [network](<https://devfeed.tech/tags/network.md>), [performance](<https://devfeed.tech/tags/performance.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [rust](<https://devfeed.tech/tags/rust.md>), [s3](<https://devfeed.tech/tags/s3.md>), [storage](<https://devfeed.tech/tags/storage.md>), [xet](<https://devfeed.tech/tags/xet.md>)

## AI overview

Hugging Face's Xet team explains how content-defined chunking is being adapted for production to accelerate uploads and downloads on the Hub. The article describes the trade-offs of fine-grained deduplication, including network, infrastructure, metadata, and storage costs, and introduces a Rust-based chunk-oriented integration designed to improve experimentation and collaboration on models and datasets.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.