# Rethinking full text search

DevFeed: [Rethinking full text search](<https://devfeed.tech/articles/rethinking-full-text-search-38907.md>)

Original publisher: [Read original article](<https://idea.popcount.org/2012-07-02-rethinking-full-text-search>)

Author: Marek

Published: 2012-07-01T22:00:00Z

Content type: opinion

Language: en

Sources: [Marek Majkowski](<https://devfeed.tech/sources/marek-majkowski.md>)

Topics: [Databases](<https://devfeed.tech/topics/databases.md>), [scaling](<https://devfeed.tech/topics/scaling.md>), [NoSQL](<https://devfeed.tech/topics/nosql.md>), [data](<https://devfeed.tech/topics/data.md>), [coding](<https://devfeed.tech/topics/coding.md>)

Tags: [full-text-search](<https://devfeed.tech/tags/full-text-search.md>), [key-value-store](<https://devfeed.tech/tags/key-value-store.md>), [nosql](<https://devfeed.tech/tags/nosql.md>), [performance](<https://devfeed.tech/tags/performance.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [search-engine](<https://devfeed.tech/tags/search-engine.md>)

## AI overview

The article explores the design of a full-text search engine for Twitter-like workloads, focusing on real-time index updates, horizontal scalability, and simple architecture. It reports experiments using a distributed key-value store and finds that the storage layer, rather than the Python indexer, became the main performance bottleneck as the dataset grew beyond available memory.

## Source excerpt

Rethinking full text search In January 2009 I was wondering whether it is possible to build a full-text search engine that could handle the search for Twitter. At that time the search tool they provided was barely working. Later Twitter acquired Summize to solve this problem. A search engine for Twitter presents a unique set of constraints. Traditional full-text search engines assume that index once created won't be changed, updating is expensive by design.