# Data and Analytics, Data Insights, Engineering, Data Science

Published articles for Data and Analytics, Data Insights, Engineering, Data Science.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## How to remove duplicates in large datasets

DevFeed: [How to remove duplicates in large datasets](<https://devfeed.tech/articles/how-to-remove-duplicates-in-large-datasets-27893.md>)

Original publisher: [Read original article](<https://clevertap.com/blog/how-to-remove-duplicates-in-large-datasets/>)

Author: Suresh Kondamudi

Published: 2016-04-19T17:17:05Z

Content type: tutorial

Language: en

Sources: [CleverTap](<https://devfeed.tech/sources/clevertap.md>)

Topics: [datasets](<https://devfeed.tech/topics/datasets.md>), [Data structures](<https://devfeed.tech/topics/data-structures.md>), [hash](<https://devfeed.tech/topics/hash.md>)

Tags: [bloom-filter](<https://devfeed.tech/tags/bloom-filter.md>), [data-and-analytics](<https://devfeed.tech/tags/data-and-analytics.md>), [data-and-analytics-data-insights-engineering-data-science](<https://devfeed.tech/tags/data-and-analytics-data-insights-engineering-data-science.md>), [data-insights](<https://devfeed.tech/tags/data-insights.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [data-structures](<https://devfeed.tech/tags/data-structures.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [hash](<https://devfeed.tech/tags/hash.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [push-notifications](<https://devfeed.tech/tags/push-notifications.md>)

### AI overview

This tutorial explains how probabilistic data structures, especially Bloom filters, can reduce memory requirements when identifying duplicate push notification tokens in datasets ranging from hundreds of millions to billions of records.

### Source excerpt

Dealing with large datasets is often daunting. With limited computing resources, particularly memory, it can be challenging to perform even The post How to remove duplicates in large datasets first appeared on CleverTap.