# From Pre-trained to Fine-tuned: Nextdoor's Path to Effective Embedding Applications

DevFeed: [From Pre-trained to Fine-tuned: Nextdoor's Path to Effective Embedding Applications](<https://devfeed.tech/articles/from-pre-trained-to-fine-tuned-nextdoor-s-path-to-effective-embedding-applications-20338.md>)

Original publisher: [Read original article](<https://engblog.nextdoor.com/from-pre-trained-to-fine-tuned-nextdoors-path-to-effective-embedding-applications-3a13b56d91aa?source=rss----5e54f11cdfdf---4>)

Author: Karthik Jayasurya

Published: 2023-09-07T11:31:32Z

Content type: article

Language: en

Sources: [Nextdoor](<https://devfeed.tech/sources/nextdoor.md>)

Topics: [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Natural language processing](<https://devfeed.tech/topics/nlp.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [Development](<https://devfeed.tech/topics/development.md>), [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>)

Tags: [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [development](<https://devfeed.tech/tags/development.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [knowledge-graph](<https://devfeed.tech/tags/knowledge-graph.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [ml](<https://devfeed.tech/tags/ml.md>), [models](<https://devfeed.tech/tags/models.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [recommendation-system](<https://devfeed.tech/tags/recommendation-system.md>), [systems](<https://devfeed.tech/tags/systems.md>)

## AI overview

Nextdoor describes its evolution from using pre-trained transformer models as embedding feature extractors to fine-tuning embeddings with unlabelled and labeled data. The article covers how embeddings are developed, featurized, and served at scale for applications including notification scoring and feed ranking.

## Source excerpt

Background The majority of ML models at Nextdoor are typically driven by a large number of features that are primarily either continuous or discrete in nature. The personalized features usually stem from historical aggregations or real-time summarization of interaction features, typically captured through logged tracking events. However, representing content through deep understanding using information behind it (text/image) is crucial for modeling nuanced user signals and better personalizing complex user behavior across many of our products. In the rapidly evolving field of NLP, utilizing transformer models to perform representation learning effectively and efficiently has become increasingly important for user understanding and improving their product experience. Towards that, we have built a lot of entity embedding models spanning entities such as posts, comments, users, search queries & classifieds. We first leveraged deep understanding of content and used that to derive embeddings for meta entities like users based on their past interacted content. These powerful representations are found to be very crucial towards extracting meaningful features for some of the biggest ML ranking systems at Nextdoor such as notifications scoring and feed ranking. By making them readily available and building to scale, we can drive adoption of state-of-the-art reliably and put them in the hands of ML Engineers for rapidly building performant models across the company. This blog primarily focuses on how we iterated on the development of embedding models, how they are featurized and served at large scale into various product applications as well as some of the challenges encountered during this process. We summarize the evolution of work across three sections. In section 1, the focus is to leverage state-of-the-art pre-trained models to rapidly evaluate the value of embeddings models as feature extractors. Section 2 describes how to fine-tune embeddings using unlabelled data for