# content defined chunking Content-defined chunking is an algorithm that splits file data into variable-length chunks with deterministic boundaries to support deduplication. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Parquet Content-Defined Chunking DevFeed: [Parquet Content-Defined Chunking]() Original publisher: [Read original article]() Author: Krisztian Szucs Published: 2025-07-25T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [parquet](), [content defined chunking](), [xet](), [data-engineering](), [datasets]() Tags: [content-defined-chunking](), [data](), [data-engineering](), [datasets](), [dedupe](), [hub](), [parquet](), [performance](), [storage](), [xet]() ### AI overview This article explains how Parquet Content-Defined Chunking (CDC) in PyArrow and Pandas improves deduplication of Parquet files on content-addressable storage systems such as Hugging Face Xet. It describes how CDC reduces data transfer and storage costs by identifying and reusing unchanged data chunks, and demonstrates the behavior across common table modifications. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Migrating the Hub from Git LFS to Xet DevFeed: [Migrating the Hub from Git LFS to Xet]() Original publisher: [Read original article]() Author: Jared Sulzdorf; Joseph Godlewski; Sam Horradarn Published: 2025-07-15T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [xet](), [migration](), [content addressed store](), [content defined chunking](), [Amazon S3](), [Git](), [Open Source]() Tags: [cas](), [content-addressed-store](), [content-defined-chunking](), [git](), [infrastructure](), [integration](), [migration](), [s3](), [storage](), [xet]() ### AI overview The article explains Hugging Face's migration of the Hub from Git LFS to Xet. It describes the Git LFS Bridge, background content migrations, content-defined chunking, the content addressed store, and S3-backed storage that enable gradual, large-scale migration without disrupting users. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Xet is on the Hub DevFeed: [Xet is on the Hub]() Original publisher: [Read original article]() Author: Assaf Vayner; Brian Ronan; Di Xiao; Joseph Godlewski; Sam Horradarn; Jared Sulzdorf Published: 2025-03-18T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [xet](), [content defined chunking](), [hugging face](), [datasets](), [migration](), [parquet](), [SQLite](), [Open Source]() Tags: [content-addressed-store](), [content-defined-chunking](), [datasets](), [hugging-face](), [infrastructure](), [migration](), [parquet](), [performance](), [sqlite](), [storage](), [xet]() ### AI overview Hugging Face describes migrating the first Model and Dataset repositories from LFS to Xet storage. The article explains how content-defined chunking enables byte-level deduplication, reducing the data transferred for small edits to large files and improving upload performance. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## From Files to Chunks: Improving HF Storage Efficiency DevFeed: [From Files to Chunks: Improving HF Storage Efficiency]() Original publisher: [Read original article]() Author: Jared Sulzdorf; Ann Huang Published: 2024-11-20T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [xet](), [content defined chunking](), [content addressed store](), [datasets](), [hugging face](), [Git](), [Open Source]() Tags: [content-addressed-store](), [content-defined-chunking](), [datasets](), [dedupe](), [git](), [open-source](), [performance](), [storage](), [xet]() ### AI overview Hugging Face's Xet team describes a storage system that splits files into variable-sized chunks using content-defined chunking and a rolling hash. Chunks are stored in a content-addressed store with deduplication, so updates upload only new chunks. The article reports a consistent 50% improvement in storage and transfer performance compared with Git LFS across three iterative development use cases, including the CORD-19 dataset. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Improving Parquet Dedupe on Hugging Face Hub DevFeed: [Improving Parquet Dedupe on Hugging Face Hub]() Original publisher: [Read original article]() Author: yuchenglow; Di Xiao Published: 2024-10-05T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [parquet](), [content defined chunking](), [datasets](), [hugging face](), [Data analysis](), [Databases]() Tags: [analysis](), [content-defined-chunking](), [data](), [data-analysis](), [datasets](), [dedupe](), [format](), [hugging-face](), [parquet](), [storage](), [xet]() ### AI overview This article examines how Parquet file layout affects byte-level Content-Defined Chunking deduplication on Hugging Face Hub. Experiments show that appending rows deduplicates 99.1% of a 2GB file, while modifying one row achieves 89% deduplication because rewritten column headers contain absolute file offsets. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science.