# Smart suggestions with Django, Elasticsearch and Haystack

DevFeed: [Smart suggestions with Django, Elasticsearch and Haystack](<https://devfeed.tech/articles/smart-suggestions-with-django-elasticsearch-and-haystack-20001.md>)

Original publisher: [Read original article](<http://engineering.hackerearth.com/2016/01/29/smart-sugesstions-with-elasticsearch/>)

Published: 2016-01-29T00:00:00Z

Content type: tutorial

Language: en

Sources: [HackerEarth](<https://devfeed.tech/sources/hackerearth.md>)

Topics: [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [Django](<https://devfeed.tech/topics/django.md>), [data](<https://devfeed.tech/topics/data.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [tokenizers](<https://devfeed.tech/topics/tokenizers.md>), [servers](<https://devfeed.tech/topics/servers.md>)

Tags: [data](<https://devfeed.tech/tags/data.md>), [django](<https://devfeed.tech/tags/django.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [latency](<https://devfeed.tech/tags/latency.md>), [mapping](<https://devfeed.tech/tags/mapping.md>), [search](<https://devfeed.tech/tags/search.md>), [tokenizers](<https://devfeed.tech/tags/tokenizers.md>)

## AI overview

This tutorial explains how HackerEarth uses Django, Elasticsearch, and Haystack to provide smart suggestions over large collections of user data. It introduces inverted indexes, discusses the trade-off between search relevance and latency, and describes how Elasticsearch analyzers, tokenizers, token filters, character filters, and field mappings can be configured for effective search.

## Source excerpt

Introduction One of the primary issues when gathering information from users is suggesting the right options that they are looking for. At HackerEarth, we gather information from all our developers which help us provide them a better experience. So there came a time when we had to suggest very smartly to our users! :D When humongous amounts of data has to be indexed and suggested intelligently, one of the efficient ways to do it is by using an inverted index. An inverted index basically is a map of words that appear in documents to a list of documents the word is found in. Popular Lucene based search servers like Elasticsearch and Solr are tools to maintain large inverted indexes and provide an efficient means to look up documents. Here is an example from the profiles page on HackerEarth. We use Elasticsearch to index millions of documents with various fields. Two hurdles to be crossed while solving this problem are latency and relevance. Relevent documents have to be suggested to the user while keeping the time taken to retrieve them (ie. latency) as low as possible. Elasticsearch uses analyzers that help in achieving good relevence, but only if used in a witty manner. It also allows us to build our own custom analyzers. So by assaying the user input, astute analyzers can be built to increase relevance. A simple example for a document can be something like, { '_id' : 'AVJUN6QaLYvICHZxvYEq', 'username': 'ksrvtsa', 'location': 'Bangalore', 'hobbies': ['music', 'reading', 'hiking'], } So what are analyzers? An analyzer converts the text to be indexed and creates lookups for finding the text when needed using appropriate search terms. An analyzer is composed of a tokenizer that splits your text into multiple tokens which is followed by many token filters which modify, delete or add new tokens. The tokenizer can be preceded by character filters which modify the text before passing it to the tokenizer. Every field in a document has an index analyzer and a search analyzer