# How to remove duplicates in large datasets

DevFeed: [How to remove duplicates in large datasets](<https://devfeed.tech/articles/how-to-remove-duplicates-in-large-datasets-27893.md>)

Original publisher: [Read original article](<https://clevertap.com/blog/how-to-remove-duplicates-in-large-datasets/>)

Author: Suresh Kondamudi

Published: 2016-04-19T17:17:05Z

Content type: tutorial

Language: en

Sources: [CleverTap](<https://devfeed.tech/sources/clevertap.md>)

Topics: [datasets](<https://devfeed.tech/topics/datasets.md>), [Data structures](<https://devfeed.tech/topics/data-structures.md>), [hash](<https://devfeed.tech/topics/hash.md>)

Tags: [bloom-filter](<https://devfeed.tech/tags/bloom-filter.md>), [data-and-analytics](<https://devfeed.tech/tags/data-and-analytics.md>), [data-and-analytics-data-insights-engineering-data-science](<https://devfeed.tech/tags/data-and-analytics-data-insights-engineering-data-science.md>), [data-insights](<https://devfeed.tech/tags/data-insights.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [data-structures](<https://devfeed.tech/tags/data-structures.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [hash](<https://devfeed.tech/tags/hash.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [push-notifications](<https://devfeed.tech/tags/push-notifications.md>)

## AI overview

This tutorial explains how probabilistic data structures, especially Bloom filters, can reduce memory requirements when identifying duplicate push notification tokens in datasets ranging from hundreds of millions to billions of records.

## Source excerpt

Dealing with large datasets is often daunting. With limited computing resources, particularly memory, it can be challenging to perform even The post How to remove duplicates in large datasets first appeared on CleverTap.