# How to Reindex One Billion Documents in One Hour at SoundCloud

DevFeed: [How to Reindex One Billion Documents in One Hour at SoundCloud](<https://devfeed.tech/articles/how-to-reindex-one-billion-documents-in-one-hour-at-soundcloud-2050.md>)

Original publisher: [Read original article](<https://developers.soundcloud.com/blog//how-to-reindex-1-billion-documents-in-1-hour-at-soundcloud>)

Published: 2019-03-21T00:00:00Z

Content type: tutorial

Language: en

Sources: [SoundCloud Backstage Blog](<https://devfeed.tech/sources/soundcloud-backstage-blog.md>)

Topics: [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [MySQL](<https://devfeed.tech/topics/mysql.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [availability](<https://devfeed.tech/tags/availability.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [mysql](<https://devfeed.tech/tags/mysql.md>), [performance](<https://devfeed.tech/tags/performance.md>), [scale](<https://devfeed.tech/tags/scale.md>), [search](<https://devfeed.tech/tags/search.md>)

## AI overview

SoundCloud describes Elasticsearch changes that reduced full catalog reindexing time from up to a week to one hour. Its search indexing pipeline streams JSON documents from MySQL through Kafka to Elasticsearch, allowing reindexing by replaying Kafka messages.

## Source excerpt

In the past, the Search Team at SoundCloud had high lead times for making updates to Elasticsearch clusters, either during the implementation of a new feature or simply while fixing a bug. This was because both tasks require us to reindex our catalog from scratch, which means reindexing more than 720 million users, tracks, playlists, and albums. Altogether, this process took up to one week, though there was even one scenario where it almost took one month to roll out a bug fix. In this post, I would like to share the concrete Elasticsearch tweaks we made so that we can now reindex our entire catalog in one hour.