# Nextdoor Nextdoor is the neighborhood hub for trusted connections and the exchange of helpful information, goods, and services. We believe that by bringing neighbors together, we can cultivate a kinder world where everyone has a neighborhood they can rely on. - Medium This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Scaling Nextdoor's Datastores: Part 5 DevFeed: [Scaling Nextdoor's Datastores: Part 5]() Original publisher: [Read original article]() Author: Slava Markeyev Published: 2025-03-19T15:09:15Z Content type: article Language: en Sources: [Nextdoor]() Topics: [Databases](), [Cache](), [consistency](), [Scalability](), [Usability](), [data]() Tags: [cache](), [cache-invalidation](), [caching-strategies](), [consistency](), [database-consistency](), [database-scalability](), [databases](), [rdbms](), [reconciliation](), [scalability](), [stream](), [usability]() ### AI overview The final installment of Nextdoor's datastore-scaling series explains how cache consistency can fail when a database writer misses its cache update. It presents a Change Data Capture stream and a reconciler that uses database changes to repair cache inconsistencies. ### Source excerpt In this final installment of the Scaling Nextdoor's Datastores blog series, we detail how the Core-Services team at Nextdoor solved cache consistency challenges as part of a holistic approach to improve our database and cache scalability and usability. In Part 4: Keeping the cache consistent, we highlighted a class of consistency issues arising from racing cache writes and introduced an approach for forward cache versioning as a mechanism to avoid inconsistencies. The cache is able to decide which write to persist and which to reject because it is aware of the version of data it currently has. However, this is only a partial solution because it assumes writers will always succeed in communicating with the cache in a timely manner, if at all. Missed Writes Let's consider the scenario where Writers A and B both performed an update to the same row in the database and have not yet updated the cache. Writer A holds Version 1 and Writer B holds Version 2. What happens if Writer B with Value 2 fails to talk with the cache? Writer B fails to write to the cache. In this case the result is that the cache becomes inconsistent and we can't rely on the writers to provide that consistency. A process must exist outside of this interaction to fix-up the cache when Version 2 is written to the database but fails to be written to the cache. Change Data Stream To solve this problem we tap into a common feature provided by most modern databases, a Change Data Capture (CDC) Stream. A CDC Stream is a mechanism to subscribe to row level changes in a database. The change stream contains a row's previous column values along with the new values. Here's a visual example of the change stream when the last_name field gets updated in the database. For visual clarity the changed values have been underlined in red. Reconciler Since the database is the source of truth and the CDC Stream emits all changes, a consumer of this stream can clean up any consistency issues in the cache. In our system we ca ## Scaling Nextdoor's Datastores: Part 4 DevFeed: [Scaling Nextdoor's Datastores: Part 4]() Original publisher: [Read original article]() Author: Ronak Shah Published: 2025-03-19T15:08:57Z Content type: article Language: en Sources: [Nextdoor]() Topics: [Cache](), [Caching](), [consistency](), [Concurrency](), [distributed-systems](), [Database]() Tags: [cache](), [cache-control](), [cache-invalidation](), [caching](), [concurrency](), [concurrent](), [consistency](), [database](), [database-consistency](), [distributed-systems](), [examples](), [infrastructure](), [lua]() ### AI overview Part 4 of Nextdoor's datastore series explains how racing database writes can update a cache in a different order, allowing stale writes and leaving the cache inconsistent with the database. It introduces the problem and notes that the full solution will be covered in a later installment. ### Source excerpt In this part of the Scaling Nextdoor's Datastores blog series, we will see how the Core-Services team at Nextdoor keeps its cache consistent with database updates and avoids stale writes to the cache. In this post, we'll focus specifically on inconsistencies caused by racing writes and our solution. We'll discuss other causes and our full solution for consistent caching in the next installment of our blog: Part 5: A time-bounded eventually-consistent cache. Inconsistent Cache Caches can become inconsistent with the database for several reasons, such as: Racing Writes / Concurrent Updates: Multiple writes occurring simultaneously can result in a stale cache. Missed Writes / Failing to Update the Cache: Failure to update or set cache correctly after a database write. Delayed Cache Updates or Deletes: Slow propagation of updates or invalidation can leave the cache out of sync. Application-Level Bugs: Bugs in application side caching logic. Maintaining cache consistency with the database is crucial for data accuracy, especially in distributed systems with concurrent web requests. Consistent caching ensures reliable read-after-write behavior, improving performance and user experience. Without it, applications may face unpredictable behavior and user frustration. A well-designed caching system boosts performance, ensures consistency, and delivers up-to-date data even under high concurrency. Racing Writes Let's look at the scenario where two writers, A and B, update the same user in the database but write to the cache in a different order, causing the cache to become inconsistent with the database. Author's Note: In examples moving forward we'll use "User_1" to mean id=1 in the "users" table.Look-Aside Cache with two writers. Sequence of operations: Writer A updates the name of user_1 to Foo in the database. Writer B updates the name of the same user_1 to Bar in the database. Writer B updates the cache for user_1 with name = Bar. Writer A updates the cache for user_1 with ## Scaling Nextdoor's Datastores: Part 3 DevFeed: [Scaling Nextdoor's Datastores: Part 3]() Original publisher: [Read original article]() Author: Ronak Shah Published: 2025-03-19T15:08:43Z Content type: article Language: en Sources: [Nextdoor]() Topics: [Caching](), [Databases](), [schema-evolution](), [Object-relational mapping](), [Django](), [Python](), [Redis](), [valkey]() Tags: [blog](), [cache](), [cache-invalidation](), [caching](), [compatibility](), [database](), [django](), [lua](), [python](), [redis](), [schema](), [serialization-format](), [thundering-herd](), [valkey]() ### AI overview Part 3 of Nextdoor's datastore-scaling series explains how applications serialize database objects for Redis or Valkey look-aside caches. It describes compatibility problems caused by runtime, package, and schema changes, including cache misses and thundering-herd effects during migrations. ### Source excerpt In this part of the Scaling Nextdoor's Datastores blog series, we'll explore how the Core-Services team at Nextdoor serializes database data for caching while ensuring forward and backward compatibility between the cache and application code. In part 1 of this series we discussed how ORMs, object-relational mapping frameworks, help abstract away database specific schemas and queries from application code. Developers simply utilize objects in their application's language to access database data. Here's a simple example of using Python's Django ORM to define a model: from django.db import models class Users(models.Model): first_name = models.CharField(max_length=30) last_name = models.CharField(max_length=30) The associated SQL create table would look like: CREATE TABLE users ( "id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "first_name" varchar(30) NOT NULL, "last_name" varchar(30) NOT NULL ); Developers would then access database data like this: user_id = 123 user = User.objects.get(id=user_id) print(user.first_name)Object Byte Serialization for Caching An issue arises when adding a look-aside cache such as Redis/Valkey to an application: How do you store what you got from the database in the cache? A common solution to caching complex objects, such as those from ORMs, is object byte serialization. This process converts language objects into bytes before storing them in the cache. When reading from the cache the process is done in reverse where the byte data is turned into language objects. For instance in Python this is often done with the pickle package. The interaction between the application, database, and the cache looks like this: Look-Aside Cacheimport pickle # Try getting from cache ('None' if not in cache) user_bytes = cache.get("user_123") if user_bytes is not None: # Read bytes using pickle user = pickle.loads(user_bytes) else: # Fetch from database user = User.objects.get(id=123) # Convert to bytes user_bytes = pickle.dumps(user) # Sto ## Scaling Nextdoor's Datastores: Part 2 DevFeed: [Scaling Nextdoor's Datastores: Part 2]() Original publisher: [Read original article]() Author: Tushar Singla Published: 2025-03-19T15:08:32Z Content type: article Language: en Sources: [Nextdoor]() Topics: [Database](), [Replication](), [Django](), [Object-relational mapping](), [consistency](), [race-condition](), [Transactions]() Tags: [consistency](), [database](), [database-scalability](), [django](), [orm](), [race-condition](), [rdbms](), [read-replica](), [replication](), [transactions]() ### AI overview The second installment of Nextdoor's datastore-scaling series examines the consistency problems caused by database read replicas. It describes how routing decisions became obscured by Django ORM abstractions, leading to read-after-write races, and explains how transactions were used as a workaround with negative effects on database load. ### Source excerpt In the second installment of Nextdoor's "Scaling Nextdoor's Datastores" blog series, the Core-Services team discusses challenges faced after implementing database read replicas. Adding read replicas to an existing database is a very common pattern as applications or products evolve to handle increased demand. Typically, the implementation details are hand waved and it's assumed that this strategy will work. However, that is rarely the case, and we'll dive into some more of the intricacies around the implementation. Initial Attempt When replicas were first introduced in the Nextdoor stack, we gave the product engineers latitude to choose when they wanted to have their query routed to a read replica or to the primary. This was done by leveraging the existing routing mechanism in our ORM, Django. This seemed like the right idea at the time because the product engineers had the most context around consistency requirements within their changes and load characteristics of their product feature. Therefore, they would have the best ability to judge which node to send their query to. However, as our business logic evolved and became more feature-rich, product engineers began to add abstraction layers to help abstract complex operations away from business logic. In this design evolution there is a high frequency read, followed by a low frequency conditional write, followed by a read. The read performed after the write should be routed to the primary, but that may get buried in abstractions and this requirement regressed. The explicit routing decisions engineers made became buried and subsequently created a serious problem for users of these abstractions. If one abstraction method was performing a write and another a read, they could not safely be used together due to read-after-write consistency issues. Due to replication lag between the primary and replica databases, a race condition arises when the application attempts to read data from a replica after performing a write. W ## Scaling Nextdoor's Datastores: Part 1 DevFeed: [Scaling Nextdoor's Datastores: Part 1]() Original publisher: [Read original article]() Author: Slava Markeyev Published: 2025-03-19T15:08:13Z Content type: article Language: en Sources: [Nextdoor]() Topics: [Scalability](), [Databases](), [Caching](), [Back end](), [Django](), [Python](), [PostgreSQL](), [NoSQL]() Tags: [backend](), [caching](), [database-consistency](), [database-scalability](), [databases](), [development](), [django](), [nosql](), [postgresql](), [python](), [rdbms](), [scalability]() ### AI overview This introductory post in a Nextdoor blog series examines scalability challenges involving database load and cache consistency. It reviews common industry solutions, their caveats, and the difficulty of moving from entrenched relational data models to NoSQL or distributed SQL datastores. ### Source excerpt At Nextdoor, the Core-Services team is responsible for the primary set of databases and caches that power the Nextdoor platform. This blog series explores our 2024 initiatives to enhance the scalability of this critical infrastructure. When we sat down at the whiteboard we sought to address two related problems: How can we reduce load on our primary database(s) and better utilize database read replicas? How can we improve our cache consistency? In this post we'll provide a primer on the common industry-wide solutions we've previously employed along with discussing their caveats and pitfalls. In subsequent posts we'll dive into the technical details of the components of our solution and how they fit together. Table of Contents Background primer (this post) Decreasing database load with dynamic routing Appropriately serializing data for caching Keeping the cache consistent A time-bounded, eventually-consistent cache Background Nextdoor's backend, built using the Python-based Django web framework, powers the core product experience for neighbors, government agencies, and local businesses. The power of Django and similar frameworks (Rails, Spring, etc) is that they allow development teams to focus on implementing business logic rather than getting caught up in the details like learning and writing SQL. The Object Relational Mapping, ORMs, included in these frameworks provide a lever that allows developers to define data models and relationships between them in the application's language without ever needing to worry about SQL. As some readers are all too aware, relational data modeling comes at a cost. Without careful data modeling, performant access to relational data largely depends on that data residing on monolithic databases. NoSQL or distributed SQL datastores are often advertised as solutions to the scalability challenges of relational databases like PostgreSQL. However, many companies face significant obstacles in transitioning to these modern datastores. Their ## Improving Nextdoor Notification Email Engagement with Generative AI and Rejection Sampling DevFeed: [Improving Nextdoor Notification Email Engagement with Generative AI and Rejection Sampling]() Original publisher: [Read original article]() Author: Jaewon Yang Published: 2023-10-16T17:03:52Z Content type: article Language: en Sources: [Nextdoor]() Topics: [Artificial Intelligence](), [Generative AI](), [Reinforcement learning](), [email]() Tags: [a-b-testing](), [ai](), [building](), [chatgpt](), [email](), [generative-ai](), [metrics](), [model-training](), [notifications](), [platform](), [reinforcement-learning]() ### AI overview Nextdoor describes using Generative AI and rejection sampling to improve Notification email subject lines. The approach uses user engagement feedback to produce more informative subject lines intended to increase email opens, clicks, and sessions. ### Source excerpt Generative AI (Gen AI) has demonstrated proficiency in content generation but does not consistently guarantee user engagement, mainly for two reasons. First, Gen AI generates content without considering user engagement feedback. While the content may be informative and well-written, it does not always translate to increased user engagement such as clicks. Second, Gen AI-produced content often remains generic and may not always provide the specific information that users seek. Nextdoor is the neighborhood network where neighbors, businesses, and public agencies connect with each other. Nextdoor is building innovative solutions to enhance the user engagement with AI-Generated Content (AIGC). This post outlines our approach to improving user engagement through user feedback, specifically focusing on Notification email subject lines. Our solutions employ Rejection sampling [1], a technique used in reinforcement learning, to boost the engagement metrics. We believe our work presents a general framework to drive user engagement with AIGC, particularly when off-the-shelf Generative AI falls short in producing engaging content. To the best of our knowledge, this marks an early milestone in the industry's successful use of AIGC to enhance user engagement. Introduction At Nextdoor, one of the ways to drive user growth and engagement on platform is through emails. One of the emails we have is called New and Trending notifications, where we send a single post that we think the user might be interested in and want to engage with. As part of sending an email, we need to determine a subject line of the email for the email audiences. Historically, we simply pick the first few words of the post being sent to be the subject line. However, in certain posts, these initial words are often greetings or introductory remarks and may not provide valuable information to the user. In the provided image example below, we observe a simple greeting, "Hello!" Figure 1. New and Trending email wher ## From Pre-trained to Fine-tuned: Nextdoor's Path to Effective Embedding Applications DevFeed: [From Pre-trained to Fine-tuned: Nextdoor's Path to Effective Embedding Applications]() Original publisher: [Read original article]() Author: Karthik Jayasurya Published: 2023-09-07T11:31:32Z Content type: article Language: en Sources: [Nextdoor]() Topics: [Embeddings](), [Fine-tuning](), [Machine learning](), [Natural language processing](), [Transformer](), [Development](), [recommendation systems]() Tags: [deep-learning](), [development](), [embedding](), [embeddings](), [knowledge-graph](), [machine-learning](), [ml](), [models](), [ranking](), [real-time](), [recommendation-system](), [systems]() ### AI overview Nextdoor describes its evolution from using pre-trained transformer models as embedding feature extractors to fine-tuning embeddings with unlabelled and labeled data. The article covers how embeddings are developed, featurized, and served at scale for applications including notification scoring and feed ranking. ### Source excerpt Background The majority of ML models at Nextdoor are typically driven by a large number of features that are primarily either continuous or discrete in nature. The personalized features usually stem from historical aggregations or real-time summarization of interaction features, typically captured through logged tracking events. However, representing content through deep understanding using information behind it (text/image) is crucial for modeling nuanced user signals and better personalizing complex user behavior across many of our products. In the rapidly evolving field of NLP, utilizing transformer models to perform representation learning effectively and efficiently has become increasingly important for user understanding and improving their product experience. Towards that, we have built a lot of entity embedding models spanning entities such as posts, comments, users, search queries & classifieds. We first leveraged deep understanding of content and used that to derive embeddings for meta entities like users based on their past interacted content. These powerful representations are found to be very crucial towards extracting meaningful features for some of the biggest ML ranking systems at Nextdoor such as notifications scoring and feed ranking. By making them readily available and building to scale, we can drive adoption of state-of-the-art reliably and put them in the hands of ML Engineers for rapidly building performant models across the company. This blog primarily focuses on how we iterated on the development of embedding models, how they are featurized and served at large scale into various product applications as well as some of the challenges encountered during this process. We summarize the evolution of work across three sections. In section 1, the focus is to leverage state-of-the-art pre-trained models to rapidly evaluate the value of embeddings models as feature extractors. Section 2 describes how to fine-tune embeddings using unlabelled data for ## Securing Diversity in Cybersecurity DevFeed: [Securing Diversity in Cybersecurity]() Original publisher: [Read original article]() Author: Kristen Beneduce Published: 2023-05-02T13:01:51Z Content type: opinion Language: en Sources: [Nextdoor]() Topics: [Cybersecurity](), [Security & Privacy]() Tags: [ciso](), [culture](), [cybersecurity](), [diversity](), [engienering](), [events](), [inclusion](), [industry](), [innovation](), [rsa-conference](), [security]() ### AI overview Nextdoor discusses the representation gap in cybersecurity and its partnership with WiCyS Silicon Valley during an RSAC 2023 diversity event. The article highlights barriers affecting women in cybersecurity and argues that diverse teams improve problem-solving and innovation. ### Source excerpt Panelists from Left to Right: Ronit Polak (Moderator), Kathy Wang* , Lea Kissner, Rupa Parameswaran, Olivia Rose, Jameeka Green Aaron *Correction: Kathy Wang is the former, not current CISO of Discord At Nextdoor we build technology that empowers resilient, safe, and kind neighborhoods all over the world. Securing a product that empowers global communities requires diverse and inclusive teams, reflective of the communities we support. Yet hiring and retaining the diverse talent needed to achieve our purpose remains an industry challenge. The gap is particularly evident in the cybersecurity field where 25% of the workforce and 16% of CISOs identify as female. According to the WiCyS State of Inclusion report 2023, women cite lack of respect and limited opportunities for growth in cybersecurity as top challenges accompanying lack of representation. We must keep working on it. That is why Nextdoor welcomed the chance to celebrate diversity, alongside RSAC 2023, in Nextdoor HQ's backyard this week and to partner with our neighborhood Women in Cybersecurity (WiCyS) Silicon Valley chapter. We are committed to building a diverse and inclusive workplace, and we are proud to work with organizations like WiCyS, who share the same values. Nextdoor's CISO TC Niedzialkowski kicked off with a warm welcome. CEO Sarah Friar framed the discussion by sharing how she launched her career by building a network at her first RSA conference as an equity analyst for Security Software at Goldman Sachs. She emphasized that diverse teams bring a variety of perspectives and experiences to the table, which ultimately leads to better problem-solving and innovation. Left to Right: Tanvi Kolte Tiwari (WiCyS Silicon Valley Events Chair) introducing the panel, Attendees soaking into a fantastic intro by Sarah Friar (Nextdoor CEO) , TC Niedzialkowski (Nextdoor CISO) cheering on the panel Moderator Ronit Polak, WiCyS Silicon Valley President, and CISOs Kathy Wang Lea Kissner Rupa Parameswaran Olivia Ros ## Catching Anomalies Early in Mobile App Releases DevFeed: [Catching Anomalies Early in Mobile App Releases]() Original publisher: [Read original article]() Author: Walt Leung Published: 2023-01-11T15:27:14Z Content type: article Language: en Sources: [Nextdoor]() Topics: [Mobile](), [observability](), [Android](), [iOS]() Tags: [android](), [data-science](), [ios](), [metrics](), [mobile](), [mobile-app-development](), [mobile-apps](), [observability](), [regression](), [releases](), [software-development](), [statistics]() ### AI overview This engineering article explains how Nextdoor uses phased rollouts and observability practices to detect anomalies in iOS and Android mobile app releases. It describes why aggregate metrics can make regressions difficult to detect at low adoption levels. ### Source excerpt How Nextdoor catches mobile app release anomalies at 1% adoption At Nextdoor, our mobile applications on iOS and Android serve content to tens of millions of weekly active users. At this scale, we run a weekly release process for both iOS and Android, shipping hundreds of changes across multiple teams and dozens of mobile engineers. Our team uses several observability processes and rollout strategies to keep these deployments safe and scalable. We most notably use phased rollouts to minimize the impact of a potentially bad release. Phased rollouts allow us to gradually increase the adoption of users for a new app version. For example, we can have a new app version be released to only 1% of users on the 1st day, 2% of users on the 2nd day, and so on. That way, if a new release were accidentally shipped with an uncaught regression, having it at 1% rollout means it affects fewer users, reduces its severity level, and gives us more time to react. However, for many of our critical business metrics where a failure can sometimes be silent, most out-of-the-box observability approaches don't work with phased rollouts. This is largely due to two problems: Observability typically happens at an aggregate level. For example, we look at app sessions or revenue on a daily basis, across all users for a platform. The behavior of early adopters on an app version differs from the median behavior of all users. Most importantly, early adopters are more active, almost by definition, to be in an early rollout of the new app version. At Nextdoor, Daily Users are more likely to adopt releases over Weekly Users, Weekly Users over Monthly Users, and so on. For example, consider an app session regression on a hypothetical iOS version v1.234.5 released March 4. If we had unknowingly introduced a regression where we didn't count an app session 5% of the time, at a 1% rollout, our aggregate impact would be expected to be roughly 0.05 x 0.01 = 0.05% of all iOS app sessions, which is practically im ## Typeahead Search at Nextdoor DevFeed: [Typeahead Search at Nextdoor]() Original publisher: [Read original article]() Author: Jerry Tian Published: 2022-07-06T19:49:06Z Content type: tutorial Language: en Sources: [Nextdoor]() Topics: [Latency](), [API](), [Low Latency](), [Network](), [Requirements](), [User Experience](), [Google Search]() Tags: [apis](), [autocomplete](), [engineering](), [geohash](), [google-search](), [latency](), [low-latency](), [network](), [requirements](), [search](), [system-design-project](), [user-experience]() ### AI overview This article explains how Nextdoor built a proximity-based typeahead search service for businesses, users, and keywords. It describes the service's focus on geographic relevance, low latency, horizontal scalability, extensibility, and high-throughput indexing. ### Source excerpt Background In a thriving community, people are connected to their friends and local businesses. Nextdoor is the hyperlocal platform that mirrors these offline relationships. Every day, through active discussions on the platform, new relationships are formed and existing ones strengthened. For example, a Nextdoor user can create a post like "I really like @XYZ cafe. @John is a hard working business owner and we should all support him by buying a cup of delicious latte!" Here, the post is created by at-mentioning (via the @ symbol) nearby businesses and users. From this post, users in the neighborhood can contribute by at-mentioning others to be part of the comment threads. As a result, John's cafe thrives and acts as a neighborhood hub where new friends are made. Every month, millions of these mentions are created in various discussions (including lost dogs!). In addition to posts and comments, a user can type into the search box and see, among other things, nearby users and businesses. All these features are powered by the same autocomplete service -- a set of APIs to ingest data and handle typeahead search of different entity types (businesses, users, keywords etc) on Nextdoor. This post focuses on how we built a proximity-based typeahead service to power typeahead use cases at Nextdoor. Proximity-Based Typeahead Search as a Service Any good search experience can be boiled down to two core components: Relevance: Given a search query, whether the user sees relevant results or not. As a hyperlocal social network, relevancy is heavily weighted by geo proximity. 2. Low latency. Google Search found that a 400 millisecond delay resulted in a -0.59% change in searches/user. What's more, even after the delay was removed, these users still had -0.21% fewer searches, indicating that a slower user experience affects long term behavior. For a good autocomplete experience, as users type, relevant results should show up instantaneously. To meet the product requirements, we set ou