# Offline Evaluation of Open-Source Embedding Models for Model Selection

DevFeed: [Offline Evaluation of Open-Source Embedding Models for Model Selection](<https://devfeed.tech/articles/importance-of-offline-evaluation-to-guide-model-choice-20388.md>)

Original publisher: [Read original article](<https://tech.olx.com/importance-of-offline-evaluation-to-guide-model-choice-1c2be1c4599a?source=rss----761b019b483f---4>)

Author: Tiago Cabo

Published: 2025-10-07T15:11:01Z

Content type: article

Language: en

Sources: [OLX](<https://devfeed.tech/sources/olx.md>)

Topics: [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [AI Models](<https://devfeed.tech/topics/ai-models.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [classification](<https://devfeed.tech/tags/classification.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [recommendations](<https://devfeed.tech/tags/recommendations.md>), [search](<https://devfeed.tech/tags/search.md>)

## AI overview

This article explains how OLX evaluated open-source embedding models against its existing internal model. It describes embeddings, the MTEB benchmark, and practical constraints such as memory, latency, modality, and multilingual support when selecting a model for deployment.

## Source excerpt

The Importance of Offline Evaluation to Guide Model ChoiceIntroduction Recent advancements in open-source AI models make it challenging to justify the development of custom models, given the high quality of existing options. This also applies to embedding models, which are available in impressive quality. At OLX, we utilize a model called Item2vec to generate similar item recommendations. For more details, please refer to our blog post. Item2Vec: Neural Item Embeddings to enhance recommendations In this work, we developed an embedding model that not only improved recommendations but was also used by other teams across OLX, such as the search team. For an in-depth look at this application, see our post on Hybrid Search, where we tried to balance the benefits of semantic and lexical search. Hybrid Search -- Where Keywords Meet Vectors, Enabling Classifieds Discovery This article discusses our evaluation of open-source embedding models compared to our existing internal model. Open Source Embeddings An embedding model converts inputs, such as text or images, into vectors. These vectors are then used to compute similarities, often cosine distance, where closer vectors are deemed semantically related. Cosine similarity Many architectures types for embedding models support this, including word2vec, GloVe, RNN, LSTM, and transformers. This leads to hundreds of publicly available models, making selection a challenge. Common benchmarks play a crucial role in this process. After exploring various options, we chose the MTEB: Massive Text Embedding Benchmark. We selected this benchmark due to its widespread community adoption and extensive features, such as: 56 datasets across 8 tasks Support for up to 112 different languages Easy extensibility via the repo MTEB HuggingfaceBenchmark MTEB Hugging Face provides a comprehensive leaderboard showcasing top-performing models across tasks (retrieval, classification, etc), reflecting advancements in machine learning and AI. The higher, t