# Data and Analytics

Published articles for Data and Analytics.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Tricks and tips for feature engineering

DevFeed: [Tricks and tips for feature engineering](<https://devfeed.tech/articles/tricks-and-tips-for-feature-engineering-27897.md>)

Original publisher: [Read original article](<https://clevertap.com/blog/tricks-and-tips-for-feature-engineering/>)

Author: Mrinal Parekh

Published: 2016-07-20T18:32:44Z

Content type: tutorial

Language: en

Sources: [CleverTap](<https://devfeed.tech/sources/clevertap.md>)

Topics: [Feature Engineering](<https://devfeed.tech/topics/feature-engineering.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [ai-and-ml](<https://devfeed.tech/tags/ai-and-ml.md>), [ai-and-ml-data-and-analytics-data-insights-engineering-data-science](<https://devfeed.tech/tags/ai-and-ml-data-and-analytics-data-insights-engineering-data-science.md>), [data](<https://devfeed.tech/tags/data.md>), [data-and-analytics](<https://devfeed.tech/tags/data-and-analytics.md>), [data-insights](<https://devfeed.tech/tags/data-insights.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [feature-engineering](<https://devfeed.tech/tags/feature-engineering.md>), [tips](<https://devfeed.tech/tags/tips.md>), [tricks](<https://devfeed.tech/tags/tricks.md>)

### AI overview

The article explains feature engineering as the creation of meaningful new input variables from existing fields for predictive modeling. It describes how domain expertise, reasoning, intuition, or automation can improve data insights and predictive model performance.

### Source excerpt

Predictive modeling is a formula that transforms a list of input fields or variables into some output of interest. Feature The post Tricks and tips for feature engineering first appeared on CleverTap.

## How to remove duplicates in large datasets

DevFeed: [How to remove duplicates in large datasets](<https://devfeed.tech/articles/how-to-remove-duplicates-in-large-datasets-27893.md>)

Original publisher: [Read original article](<https://clevertap.com/blog/how-to-remove-duplicates-in-large-datasets/>)

Author: Suresh Kondamudi

Published: 2016-04-19T17:17:05Z

Content type: tutorial

Language: en

Sources: [CleverTap](<https://devfeed.tech/sources/clevertap.md>)

Topics: [datasets](<https://devfeed.tech/topics/datasets.md>), [Data structures](<https://devfeed.tech/topics/data-structures.md>), [hash](<https://devfeed.tech/topics/hash.md>)

Tags: [bloom-filter](<https://devfeed.tech/tags/bloom-filter.md>), [data-and-analytics](<https://devfeed.tech/tags/data-and-analytics.md>), [data-and-analytics-data-insights-engineering-data-science](<https://devfeed.tech/tags/data-and-analytics-data-insights-engineering-data-science.md>), [data-insights](<https://devfeed.tech/tags/data-insights.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [data-structures](<https://devfeed.tech/tags/data-structures.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [hash](<https://devfeed.tech/tags/hash.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [push-notifications](<https://devfeed.tech/tags/push-notifications.md>)

### AI overview

This tutorial explains how probabilistic data structures, especially Bloom filters, can reduce memory requirements when identifying duplicate push notification tokens in datasets ranging from hundreds of millions to billions of records.

### Source excerpt

Dealing with large datasets is often daunting. With limited computing resources, particularly memory, it can be challenging to perform even The post How to remove duplicates in large datasets first appeared on CleverTap.