# Improving Parquet Dedupe on Hugging Face Hub

DevFeed: [Improving Parquet Dedupe on Hugging Face Hub](<https://devfeed.tech/articles/improving-parquet-dedupe-on-hugging-face-hub-7275.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/improve_parquet_dedupe>)

Author: yuchenglow; Di Xiao

Published: 2024-10-05T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [parquet](<https://devfeed.tech/topics/parquet.md>), [content defined chunking](<https://devfeed.tech/topics/content-defined-chunking.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [Databases](<https://devfeed.tech/topics/databases.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [content-defined-chunking](<https://devfeed.tech/tags/content-defined-chunking.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [dedupe](<https://devfeed.tech/tags/dedupe.md>), [format](<https://devfeed.tech/tags/format.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [storage](<https://devfeed.tech/tags/storage.md>), [xet](<https://devfeed.tech/tags/xet.md>)

## AI overview

This article examines how Parquet file layout affects byte-level Content-Defined Chunking deduplication on Hugging Face Hub. Experiments show that appending rows deduplicates 99.1% of a 2GB file, while modifying one row achieves 89% deduplication because rewritten column headers contain absolute file offsets.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.