# dedupe Published articles for dedupe. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Using Dolt's Versioned SQL Database for Feature-Store Point-in-Time Reads DevFeed: [Using Dolt's Versioned SQL Database for Feature-Store Point-in-Time Reads]() Original publisher: [Read original article]() Published: 2026-05-15T00:00:00Z Content type: opinion Language: en Sources: [Alex Korbonits]() Topics: [feature-store](), [Git](), [reproducibility](), [Databases](), [SQL]() Tags: [dedupe](), [feature-store](), [git](), [reproducibility](), [rfc](), [row-number](), [sql](), [sql-database]() ### AI overview This article describes a prototype Feast offline-store plugin backed by Dolt, a version-controlled SQL database. It argues that Dolt's revision-based reads can provide point-in-time feature retrieval and reproducible training snapshots without the usual append-only log deduplication CTEs. Tests on toy datasets produced identical results with shorter queries, though the article presents this as an early spike rather than a production benchmark. ### Source excerpt A weekend spike asked whether Dolt's AS OF reads could replace the ROW_NUMBER dedupe at the heart of every feature store's point-in-time join. Four weeks later, the RFC is quiet and the plugin's get_historical_features works end-to-end against a live Dolt server. Notes on building anyway. ## Parquet Content-Defined Chunking DevFeed: [Parquet Content-Defined Chunking]() Original publisher: [Read original article]() Author: Krisztian Szucs Published: 2025-07-25T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [parquet](), [content defined chunking](), [xet](), [data-engineering](), [datasets]() Tags: [content-defined-chunking](), [data](), [data-engineering](), [datasets](), [dedupe](), [hub](), [parquet](), [performance](), [storage](), [xet]() ### AI overview This article explains how Parquet Content-Defined Chunking (CDC) in PyArrow and Pandas improves deduplication of Parquet files on content-addressable storage systems such as Hugging Face Xet. It describes how CDC reduces data transfer and storage costs by identifying and reusing unchanged data chunks, and demonstrates the behavior across common table modifications. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## From Chunks to Blocks: Accelerating Uploads and Downloads on the Hub DevFeed: [From Chunks to Blocks: Accelerating Uploads and Downloads on the Hub]() Original publisher: [Read original article]() Author: Jared Sulzdorf; yuchenglow; Zach Nation; saba noorassa Published: 2025-02-12T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [hugging face](), [xet](), [content addressed store](), [AI, ML & Data Engineering](), [Open Source](), [Amazon S3]() Tags: [cas](), [collaboration](), [content-addressed-store](), [content-defined-chunking](), [dedupe](), [hugging-face](), [integration](), [network](), [performance](), [quantization](), [rust](), [s3](), [storage](), [xet]() ### AI overview Hugging Face's Xet team explains how content-defined chunking is being adapted for production to accelerate uploads and downloads on the Hub. The article describes the trade-offs of fine-grained deduplication, including network, infrastructure, metadata, and storage costs, and introduces a Rust-based chunk-oriented integration designed to improve experimentation and collaboration on models and datasets. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Rearchitecting Hugging Face Uploads and Downloads DevFeed: [Rearchitecting Hugging Face Uploads and Downloads]() Original publisher: [Read original article]() Author: Banerjee; Jared Sulzdorf; Ann Huang Published: 2024-11-26T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [content addressed store](), [hugging face](), [Amazon S3](), [Amazon Web Services](), [huggingface\_hub](), [Compression](), [dataset](), [Latency](), [Network]() Tags: [aws](), [compression](), [content-addressed-store](), [dedupe](), [hugging-face](), [huggingface-hub](), [infrastructure](), [internet](), [latency](), [s3](), [speed](), [storage](), [xet]() ### AI overview Hugging Face is redesigning the Hub's upload and download architecture to handle increasingly large model and dataset files. The proposed design introduces a content-addressed store, custom chunk-aware transfer protocols, byte-level processing, deduplication, and compression while continuing to use S3 for data storage. The read path emphasizes high throughput and low latency, while the write path validates and uploads only necessary chunks to improve speed and security. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## From Files to Chunks: Improving HF Storage Efficiency DevFeed: [From Files to Chunks: Improving HF Storage Efficiency]() Original publisher: [Read original article]() Author: Jared Sulzdorf; Ann Huang Published: 2024-11-20T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [xet](), [content defined chunking](), [content addressed store](), [datasets](), [hugging face](), [Git](), [Open Source]() Tags: [content-addressed-store](), [content-defined-chunking](), [datasets](), [dedupe](), [git](), [open-source](), [performance](), [storage](), [xet]() ### AI overview Hugging Face's Xet team describes a storage system that splits files into variable-sized chunks using content-defined chunking and a rolling hash. Chunks are stored in a content-addressed store with deduplication, so updates upload only new chunks. The article reports a consistent 50% improvement in storage and transfer performance compared with Git LFS across three iterative development use cases, including the CORD-19 dataset. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Improving Parquet Dedupe on Hugging Face Hub DevFeed: [Improving Parquet Dedupe on Hugging Face Hub]() Original publisher: [Read original article]() Author: yuchenglow; Di Xiao Published: 2024-10-05T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [parquet](), [content defined chunking](), [datasets](), [hugging face](), [Data analysis](), [Databases]() Tags: [analysis](), [content-defined-chunking](), [data](), [data-analysis](), [datasets](), [dedupe](), [format](), [hugging-face](), [parquet](), [storage](), [xet]() ### AI overview This article examines how Parquet file layout affects byte-level Content-Defined Chunking deduplication on Hugging Face Hub. Experiments show that appending rows deduplicates 99.1% of a 2GB file, while modifying one row achieves 89% deduplication because rewritten column headers contain absolute file offsets. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## How Chrome achieved the highest score ever on Speedometer 3 DevFeed: [How Chrome achieved the highest score ever on Speedometer 3]() Original publisher: [Read original article]() Author: Chromium Blog (noreply@blogger.com) Published: 2024-06-06T16:15:00Z Content type: article Language: en Sources: [Chromium Blog]() Topics: [web applications]() Tags: [apple](), [benchmark](), [benchmarking](), [browser](), [chrome](), [dedupe](), [google](), [intel](), [memory](), [microsoft](), [mozilla](), [none](), [optimization](), [performance](), [the-fast-and-the-curious](), [web-applications]() ### AI overview Chrome describes workload-focused optimizations that increased its Speedometer 3 score by 72% since May 2022. The work targets browser operations including string parsing, stylesheet deduplication, path drawing, form creation, selector handling, DOM parsing, and font rendering. ### Source excerpt Today's The Fast and the Curious post explores how Chrome achieved the highest score on the new Speedometer 3.0, an upgraded browser benchmarking tool to optimize the performance of Web applications. Try out Chrome today! Speedometer 3.0 is a recently published benchmark for measuring browser performance that was created as an industry collaboration between companies like Google, Apple, Mozilla, Intel, and Microsoft. This benchmark helped us identify areas in which we could optimize Chrome to deliver a faster browser experience to all our users. Here's a closer look at how we further optimized Chrome to achieve the highest score ever Speedometer 3, by carefully tracking its recent performance over time as the updated benchmark was being developed. Since the inception of Speedometer 3 in May 2022, we've driven a 72% increase in Chrome's Speedometer score - translating into performance gains for our users: Optimizing workloads By looking at the workloads in Speedometer and in which functions Chrome was spending the most time, we were able to make targeted optimizations to those functions that each drove an increase in Chrome's score. For example, the SpaceSplitString function is used heavily to turn space-separated strings such as those in "class='foo bar' " into a list representation. In this function we removed some unnecessary bound checks. When we detect that there are duplicated stylesheets, we dedupe them and reference a single stylesheet instance. We made an optimization to reduce the cost of drawing paths and arcs by tuning memory allocations. When creating form editors we detected some unnecessary processing that occurs when form elements are created. Within querySelector, we were able to detect what selector was commonly used and create a hot-path for that. We previously shared how we optimized innerHTML using specialized fast paths for parsing, an implementation that also made its way into WebKit. Some workloads in Speedometer 3 use DOMParser so we extended ## Reducing DynamoDB Storage Costs for High-Volume Datapoint Deduplication DevFeed: [Reducing DynamoDB Storage Costs for High-Volume Datapoint Deduplication]() Original publisher: [Read original article]() Author: Joanna Solmon Published: 2017-01-04T14:45:00Z Content type: tutorial Language: en Sources: [Localytics]() Topics: [DynamoDB](), [amazon](), [Database](), [Mobile]() Tags: [amazon](), [aws](), [cost](), [data](), [database](), [dedupe](), [devices](), [distributed](), [dynamodb](), [mobile](), [ops](), [storage]() ### AI overview The article describes how Localytics used DynamoDB to deduplicate billions of daily datapoints and examined schema changes to reduce storage costs. It discusses duplicate UUIDs, a 30-day deduplication history, and the limitations of a naive schema. ### Source excerpt Amazon's DynamoDB is a great technology: it's easy to set up and provides all the benefits of an infinitely scalable distributed database without the operational overhead. Unfortunately, DynamoDB can also be very expensive for high volume, high throughput storage. This post will show how we were