# Finding near-duplicates with Jaccard similarity and MinHash

DevFeed: [Finding near-duplicates with Jaccard similarity and MinHash](<https://devfeed.tech/articles/finding-near-duplicates-with-jaccard-similarity-and-minhash-21952.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/fuzzy-dedup/>)

Author: Nelson Elhage

Published: 2024-07-03T23:00:00Z

Content type: tutorial

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [dataset](<https://devfeed.tech/topics/dataset.md>), [Website](<https://devfeed.tech/topics/website.md>)

Tags: [dataset](<https://devfeed.tech/tags/dataset.md>), [pairs](<https://devfeed.tech/tags/pairs.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [range](<https://devfeed.tech/tags/range.md>), [scale](<https://devfeed.tech/tags/scale.md>)

## AI overview

This article explains approximate document deduplication using Jaccard similarity and the MinHash approximation technique. It discusses defining similarity between document pairs, setting a threshold for approximate duplicates, and why approximate similarity is not transitive.

## Source excerpt

Suppose we have a large collection of documents, and we wish you identify which documents are approximately the same as each other. For instance, we may have crawled the web over some period of time, and expect to have fetched the "same page" several times, but to see slight differences in metadata, or that we have several revisions of a page following small edits. In this post I want to explore the method of approximate deduplication via Jaccard similarity and the MinHash approximation trick.