# Rearchitecting Hugging Face Uploads and Downloads

DevFeed: [Rearchitecting Hugging Face Uploads and Downloads](<https://devfeed.tech/articles/rearchitecting-hugging-face-uploads-and-downloads-7455.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/rearchitecting-uploads-and-downloads>)

Author: Banerjee; Jared Sulzdorf; Ann Huang

Published: 2024-11-26T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [content addressed store](<https://devfeed.tech/topics/content-addressed-store.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [huggingface\_hub](<https://devfeed.tech/topics/huggingface-hub.md>), [Compression](<https://devfeed.tech/topics/compression.md>), [dataset](<https://devfeed.tech/topics/dataset.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [Network](<https://devfeed.tech/topics/network.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [compression](<https://devfeed.tech/tags/compression.md>), [content-addressed-store](<https://devfeed.tech/tags/content-addressed-store.md>), [dedupe](<https://devfeed.tech/tags/dedupe.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [huggingface-hub](<https://devfeed.tech/tags/huggingface-hub.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [internet](<https://devfeed.tech/tags/internet.md>), [latency](<https://devfeed.tech/tags/latency.md>), [s3](<https://devfeed.tech/tags/s3.md>), [speed](<https://devfeed.tech/tags/speed.md>), [storage](<https://devfeed.tech/tags/storage.md>), [xet](<https://devfeed.tech/tags/xet.md>)

## AI overview

Hugging Face is redesigning the Hub's upload and download architecture to handle increasingly large model and dataset files. The proposed design introduces a content-addressed store, custom chunk-aware transfer protocols, byte-level processing, deduplication, and compression while continuing to use S3 for data storage. The read path emphasizes high throughput and low latency, while the write path validates and uploads only necessary chunks to improve speed and security.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.