# database-consistency Published articles for database-consistency. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Scaling Nextdoor's Datastores: Part 5 DevFeed: [Scaling Nextdoor's Datastores: Part 5]() Original publisher: [Read original article]() Author: Slava Markeyev Published: 2025-03-19T15:09:15Z Content type: article Language: en Sources: [Nextdoor]() Topics: [Databases](), [Cache](), [consistency](), [Scalability](), [Usability](), [data]() Tags: [cache](), [cache-invalidation](), [caching-strategies](), [consistency](), [database-consistency](), [database-scalability](), [databases](), [rdbms](), [reconciliation](), [scalability](), [stream](), [usability]() ### AI overview The final installment of Nextdoor's datastore-scaling series explains how cache consistency can fail when a database writer misses its cache update. It presents a Change Data Capture stream and a reconciler that uses database changes to repair cache inconsistencies. ### Source excerpt In this final installment of the Scaling Nextdoor's Datastores blog series, we detail how the Core-Services team at Nextdoor solved cache consistency challenges as part of a holistic approach to improve our database and cache scalability and usability. In Part 4: Keeping the cache consistent, we highlighted a class of consistency issues arising from racing cache writes and introduced an approach for forward cache versioning as a mechanism to avoid inconsistencies. The cache is able to decide which write to persist and which to reject because it is aware of the version of data it currently has. However, this is only a partial solution because it assumes writers will always succeed in communicating with the cache in a timely manner, if at all. Missed Writes Let's consider the scenario where Writers A and B both performed an update to the same row in the database and have not yet updated the cache. Writer A holds Version 1 and Writer B holds Version 2. What happens if Writer B with Value 2 fails to talk with the cache? Writer B fails to write to the cache. In this case the result is that the cache becomes inconsistent and we can't rely on the writers to provide that consistency. A process must exist outside of this interaction to fix-up the cache when Version 2 is written to the database but fails to be written to the cache. Change Data Stream To solve this problem we tap into a common feature provided by most modern databases, a Change Data Capture (CDC) Stream. A CDC Stream is a mechanism to subscribe to row level changes in a database. The change stream contains a row's previous column values along with the new values. Here's a visual example of the change stream when the last_name field gets updated in the database. For visual clarity the changed values have been underlined in red. Reconciler Since the database is the source of truth and the CDC Stream emits all changes, a consumer of this stream can clean up any consistency issues in the cache. In our system we ca ## Scaling Nextdoor's Datastores: Part 4 DevFeed: [Scaling Nextdoor's Datastores: Part 4]() Original publisher: [Read original article]() Author: Ronak Shah Published: 2025-03-19T15:08:57Z Content type: article Language: en Sources: [Nextdoor]() Topics: [Cache](), [Caching](), [consistency](), [Concurrency](), [distributed-systems](), [Database]() Tags: [cache](), [cache-control](), [cache-invalidation](), [caching](), [concurrency](), [concurrent](), [consistency](), [database](), [database-consistency](), [distributed-systems](), [examples](), [infrastructure](), [lua]() ### AI overview Part 4 of Nextdoor's datastore series explains how racing database writes can update a cache in a different order, allowing stale writes and leaving the cache inconsistent with the database. It introduces the problem and notes that the full solution will be covered in a later installment. ### Source excerpt In this part of the Scaling Nextdoor's Datastores blog series, we will see how the Core-Services team at Nextdoor keeps its cache consistent with database updates and avoids stale writes to the cache. In this post, we'll focus specifically on inconsistencies caused by racing writes and our solution. We'll discuss other causes and our full solution for consistent caching in the next installment of our blog: Part 5: A time-bounded eventually-consistent cache. Inconsistent Cache Caches can become inconsistent with the database for several reasons, such as: Racing Writes / Concurrent Updates: Multiple writes occurring simultaneously can result in a stale cache. Missed Writes / Failing to Update the Cache: Failure to update or set cache correctly after a database write. Delayed Cache Updates or Deletes: Slow propagation of updates or invalidation can leave the cache out of sync. Application-Level Bugs: Bugs in application side caching logic. Maintaining cache consistency with the database is crucial for data accuracy, especially in distributed systems with concurrent web requests. Consistent caching ensures reliable read-after-write behavior, improving performance and user experience. Without it, applications may face unpredictable behavior and user frustration. A well-designed caching system boosts performance, ensures consistency, and delivers up-to-date data even under high concurrency. Racing Writes Let's look at the scenario where two writers, A and B, update the same user in the database but write to the cache in a different order, causing the cache to become inconsistent with the database. Author's Note: In examples moving forward we'll use "User_1" to mean id=1 in the "users" table.Look-Aside Cache with two writers. Sequence of operations: Writer A updates the name of user_1 to Foo in the database. Writer B updates the name of the same user_1 to Bar in the database. Writer B updates the cache for user_1 with name = Bar. Writer A updates the cache for user_1 with ## Scaling Nextdoor's Datastores: Part 1 DevFeed: [Scaling Nextdoor's Datastores: Part 1]() Original publisher: [Read original article]() Author: Slava Markeyev Published: 2025-03-19T15:08:13Z Content type: article Language: en Sources: [Nextdoor]() Topics: [Scalability](), [Databases](), [Caching](), [Back end](), [Django](), [Python](), [PostgreSQL](), [NoSQL]() Tags: [backend](), [caching](), [database-consistency](), [database-scalability](), [databases](), [development](), [django](), [nosql](), [postgresql](), [python](), [rdbms](), [scalability]() ### AI overview This introductory post in a Nextdoor blog series examines scalability challenges involving database load and cache consistency. It reviews common industry solutions, their caveats, and the difficulty of moving from entrenched relational data models to NoSQL or distributed SQL datastores. ### Source excerpt At Nextdoor, the Core-Services team is responsible for the primary set of databases and caches that power the Nextdoor platform. This blog series explores our 2024 initiatives to enhance the scalability of this critical infrastructure. When we sat down at the whiteboard we sought to address two related problems: How can we reduce load on our primary database(s) and better utilize database read replicas? How can we improve our cache consistency? In this post we'll provide a primer on the common industry-wide solutions we've previously employed along with discussing their caveats and pitfalls. In subsequent posts we'll dive into the technical details of the components of our solution and how they fit together. Table of Contents Background primer (this post) Decreasing database load with dynamic routing Appropriately serializing data for caching Keeping the cache consistent A time-bounded, eventually-consistent cache Background Nextdoor's backend, built using the Python-based Django web framework, powers the core product experience for neighbors, government agencies, and local businesses. The power of Django and similar frameworks (Rails, Spring, etc) is that they allow development teams to focus on implementing business logic rather than getting caught up in the details like learning and writing SQL. The Object Relational Mapping, ORMs, included in these frameworks provide a lever that allows developers to define data models and relationships between them in the application's language without ever needing to worry about SQL. As some readers are all too aware, relational data modeling comes at a cost. Without careful data modeling, performant access to relational data largely depends on that data residing on monolithic databases. NoSQL or distributed SQL datastores are often advertised as solutions to the scalability challenges of relational databases like PostgreSQL. However, many companies face significant obstacles in transitioning to these modern datastores. Their