# Clustering Similar Stories Using LDA

DevFeed: [Clustering Similar Stories Using LDA](<https://devfeed.tech/articles/clustering-similar-stories-using-lda-31894.md>)

Original publisher: [Read original article](<http://engineering.flipboard.com//2017/02/storyclustering>)

Author: https://www.linkedin.com/in/arnab-bhadury-a6304768 (Arnab Bhadury)

Published: 2017-02-08T00:00:00Z

Content type: tutorial

Language: en

Sources: [Flipboard](<https://devfeed.tech/sources/flipboard.md>)

Topics: [Algorithms, Complexity](<https://devfeed.tech/topics/algorithms-complexity.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [algorithms](<https://devfeed.tech/tags/algorithms.md>), [clustering](<https://devfeed.tech/tags/clustering.md>), [trimming](<https://devfeed.tech/tags/trimming.md>)

## AI overview

This article explains how Flipboard uses a clustering algorithm and Latent Dirichlet Allocation (LDA) to group related stories into multi-source story roundups. It discusses challenges including high-dimensional text representations, differing word usage, unknown cluster counts, and the need for fast updates.

## Source excerpt

There is more to a story than meets the eye, and some stories deserve to be presented from more than just one perspective. With Flipboard 4.0, we have released story roundups, a new feature that adds coverage from multiple sources to a story and provides you with a fuller picture of an event. Here's how it looks: With our scale of millions of articles and constant stream of documents, it's impossible to generate these roundups manually. So, we have developed a clustering algorithm that's both fast and scalable, and in this blog post, I will explain how we create these roundups on Flipboard. Why is this difficult? Although there are many sophisticated automatic clustering algorithms, such as K-means or Agglomerative clustering, story clustering is a non-trivial problem. Because each text document can contain any word from our vocabulary, most text document representations are extremely high-dimensional. In high-dimensional spaces, even basic clustering or similarity measures fail or are very slow. Additionally, two very similar documents often have very different word usages. For example, one article may use the term kitten and another may use feline, but both articles could be referring to the same cat. Furthermore, we don't know the number of roundups that we expect to see beforehand. This makes it difficult for us to directly use parameteric algorithms such as K-means. Our clustering algorithm also needs to be fast and easy to update, because there is a constant stream of documents coming into our system. Overview Since even the most basic distance measures fail in high dimensions, the first thing we do is lower the problem's dimensionality. We represent each of our text documents as a bag-of-words, and remove stop-words and rare words from our vocabulary. Even after an aggressive trimming, the documents are still very high-dimensional. We then we use Latent Dirichlet Allocation (LDA) to further lower the documents' dimensionality. We use LDA because this algorith