# database-scalability Published articles for database-scalability. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## PGKeeper: Building the bouncer we needed for Postgres DevFeed: [PGKeeper: Building the bouncer we needed for Postgres]() Original publisher: [Read original article]() Author: Lihao He; Tim Goh; Mehant Baid Published: 2026-05-04T14:52:00Z Content type: article Language: en Sources: [Figma Blog]() Topics: [PostgreSQL](), [Scalability](), [Architecture & Design](), [benchmarking](), [Databases](), [Network]() Tags: [architecture](), [database-scalability](), [performance](), [postgres](), [postgresql](), [queue](), [routing](), [scalability](), [sharding]() ### AI overview Figma describes why it built PGKeeper, a scalable PostgreSQL connection and load management service that replaced PgBouncer in front of its Postgres fleet. The article covers the database workload pressures, request routing, connection pooling, and rollout of the new service. ### Source excerpt This is the story of why and how we built PGKeeper, a scalable and reliable service to support Figma's rapidly growing products and database workload. ## From a single point of failure to a cell-based architecture: How we scaled Mercado Envíos' stock... DevFeed: [From a single point of failure to a cell-based architecture: How we scaled Mercado Envíos' stock...]() Original publisher: [Read original article]() Author: Rafael Silvestri Published: 2025-12-29T21:09:09Z Content type: article Language: en Sources: [Mercado Libre Tech]() Topics: [distributed-systems](), [Architecture & Design](), [Scalability](), [migration](), [MySQL](), [Resilience]() Tags: [architecture](), [cell-based-architecture](), [database](), [database-scalability](), [distributed-systems](), [fury](), [internaldeveloperplatform](), [migration](), [outages](), [resilience](), [scalability](), [software-architecture]() ### AI overview This article explains how Mercado Libre migrated Mercado Envíos' consolidated MySQL-based stock system into independent cells using a cell-based architecture and Fury. The migration was intended to isolate failures, reduce systemic risk, and provide more predictable scalability after the regional database reached its vertical-scaling and operational limits. ### Source excerpt From a single point of failure to a cell-based architecture: How we scaled Mercado Envíos' stock system How we migrated Mercado Libre's second-largest MySQL instance into independent cells using Fury, reducing risk and achieving predictable scalability in Fulfillment. Abstract What happens when the database supporting a continent's logistics reaches its limits? We reached that point when a critical regional database, used for inventory operations across Latin America (LATAM), could no longer scale vertically. This article describes how we transitioned from that consolidated model to a cell-based architecture -- isolating failures, reducing systemic risk, and improving operational predictability -- while keeping logistics running throughout the migration. When a core system reaches its breaking point Software architecture uses patterns to prevent local failures from causing global outages. One of them is the cell-based architecture, which is conceptually similar to the naval bulkhead mechanism. Ships use watertight compartments, or bulkheads, to divide the hull into separate sections. If one compartment floods, the others remain sealed and the ship keeps moving. In distributed systems, we apply the same idea: each cell operates autonomously, with its own compute, database, and traffic. If one cell fails, the rest continue serving requests. This isolation reduces the blast radius and increases resilience. This pattern became essential at Mercado Libre when the stock system powering Mercado Envíos reached its operational limit. Every inbound, outbound, reservation, and logistics movement depended on a single database that could no longer scale. By early 2024, the question was clear: What do you do when vertical scaling is no longer an option? The problem: One database serving all of LATAM Our initial architecture was simple: multiple stock services connected to a single MySQL cluster. This cluster managed: Stock availability per Fulfillment Center Reservations for Fulfil ## Scaling Nextdoor's Datastores: Part 5 DevFeed: [Scaling Nextdoor's Datastores: Part 5]() Original publisher: [Read original article]() Author: Slava Markeyev Published: 2025-03-19T15:09:15Z Content type: article Language: en Sources: [Nextdoor]() Topics: [Databases](), [Cache](), [consistency](), [Scalability](), [Usability](), [data]() Tags: [cache](), [cache-invalidation](), [caching-strategies](), [consistency](), [database-consistency](), [database-scalability](), [databases](), [rdbms](), [reconciliation](), [scalability](), [stream](), [usability]() ### AI overview The final installment of Nextdoor's datastore-scaling series explains how cache consistency can fail when a database writer misses its cache update. It presents a Change Data Capture stream and a reconciler that uses database changes to repair cache inconsistencies. ### Source excerpt In this final installment of the Scaling Nextdoor's Datastores blog series, we detail how the Core-Services team at Nextdoor solved cache consistency challenges as part of a holistic approach to improve our database and cache scalability and usability. In Part 4: Keeping the cache consistent, we highlighted a class of consistency issues arising from racing cache writes and introduced an approach for forward cache versioning as a mechanism to avoid inconsistencies. The cache is able to decide which write to persist and which to reject because it is aware of the version of data it currently has. However, this is only a partial solution because it assumes writers will always succeed in communicating with the cache in a timely manner, if at all. Missed Writes Let's consider the scenario where Writers A and B both performed an update to the same row in the database and have not yet updated the cache. Writer A holds Version 1 and Writer B holds Version 2. What happens if Writer B with Value 2 fails to talk with the cache? Writer B fails to write to the cache. In this case the result is that the cache becomes inconsistent and we can't rely on the writers to provide that consistency. A process must exist outside of this interaction to fix-up the cache when Version 2 is written to the database but fails to be written to the cache. Change Data Stream To solve this problem we tap into a common feature provided by most modern databases, a Change Data Capture (CDC) Stream. A CDC Stream is a mechanism to subscribe to row level changes in a database. The change stream contains a row's previous column values along with the new values. Here's a visual example of the change stream when the last_name field gets updated in the database. For visual clarity the changed values have been underlined in red. Reconciler Since the database is the source of truth and the CDC Stream emits all changes, a consumer of this stream can clean up any consistency issues in the cache. In our system we ca ## Scaling Nextdoor's Datastores: Part 2 DevFeed: [Scaling Nextdoor's Datastores: Part 2]() Original publisher: [Read original article]() Author: Tushar Singla Published: 2025-03-19T15:08:32Z Content type: article Language: en Sources: [Nextdoor]() Topics: [Database](), [Replication](), [Django](), [Object-relational mapping](), [consistency](), [race-condition](), [Transactions]() Tags: [consistency](), [database](), [database-scalability](), [django](), [orm](), [race-condition](), [rdbms](), [read-replica](), [replication](), [transactions]() ### AI overview The second installment of Nextdoor's datastore-scaling series examines the consistency problems caused by database read replicas. It describes how routing decisions became obscured by Django ORM abstractions, leading to read-after-write races, and explains how transactions were used as a workaround with negative effects on database load. ### Source excerpt In the second installment of Nextdoor's "Scaling Nextdoor's Datastores" blog series, the Core-Services team discusses challenges faced after implementing database read replicas. Adding read replicas to an existing database is a very common pattern as applications or products evolve to handle increased demand. Typically, the implementation details are hand waved and it's assumed that this strategy will work. However, that is rarely the case, and we'll dive into some more of the intricacies around the implementation. Initial Attempt When replicas were first introduced in the Nextdoor stack, we gave the product engineers latitude to choose when they wanted to have their query routed to a read replica or to the primary. This was done by leveraging the existing routing mechanism in our ORM, Django. This seemed like the right idea at the time because the product engineers had the most context around consistency requirements within their changes and load characteristics of their product feature. Therefore, they would have the best ability to judge which node to send their query to. However, as our business logic evolved and became more feature-rich, product engineers began to add abstraction layers to help abstract complex operations away from business logic. In this design evolution there is a high frequency read, followed by a low frequency conditional write, followed by a read. The read performed after the write should be routed to the primary, but that may get buried in abstractions and this requirement regressed. The explicit routing decisions engineers made became buried and subsequently created a serious problem for users of these abstractions. If one abstraction method was performing a write and another a read, they could not safely be used together due to read-after-write consistency issues. Due to replication lag between the primary and replica databases, a race condition arises when the application attempts to read data from a replica after performing a write. W ## Scaling Nextdoor's Datastores: Part 1 DevFeed: [Scaling Nextdoor's Datastores: Part 1]() Original publisher: [Read original article]() Author: Slava Markeyev Published: 2025-03-19T15:08:13Z Content type: article Language: en Sources: [Nextdoor]() Topics: [Scalability](), [Databases](), [Caching](), [Back end](), [Django](), [Python](), [PostgreSQL](), [NoSQL]() Tags: [backend](), [caching](), [database-consistency](), [database-scalability](), [databases](), [development](), [django](), [nosql](), [postgresql](), [python](), [rdbms](), [scalability]() ### AI overview This introductory post in a Nextdoor blog series examines scalability challenges involving database load and cache consistency. It reviews common industry solutions, their caveats, and the difficulty of moving from entrenched relational data models to NoSQL or distributed SQL datastores. ### Source excerpt At Nextdoor, the Core-Services team is responsible for the primary set of databases and caches that power the Nextdoor platform. This blog series explores our 2024 initiatives to enhance the scalability of this critical infrastructure. When we sat down at the whiteboard we sought to address two related problems: How can we reduce load on our primary database(s) and better utilize database read replicas? How can we improve our cache consistency? In this post we'll provide a primer on the common industry-wide solutions we've previously employed along with discussing their caveats and pitfalls. In subsequent posts we'll dive into the technical details of the components of our solution and how they fit together. Table of Contents Background primer (this post) Decreasing database load with dynamic routing Appropriately serializing data for caching Keeping the cache consistent A time-bounded, eventually-consistent cache Background Nextdoor's backend, built using the Python-based Django web framework, powers the core product experience for neighbors, government agencies, and local businesses. The power of Django and similar frameworks (Rails, Spring, etc) is that they allow development teams to focus on implementing business logic rather than getting caught up in the details like learning and writing SQL. The Object Relational Mapping, ORMs, included in these frameworks provide a lever that allows developers to define data models and relationships between them in the application's language without ever needing to worry about SQL. As some readers are all too aware, relational data modeling comes at a cost. Without careful data modeling, performant access to relational data largely depends on that data residing on monolithic databases. NoSQL or distributed SQL datastores are often advertised as solutions to the scalability challenges of relational databases like PostgreSQL. However, many companies face significant obstacles in transitioning to these modern datastores. Their ## SQL vs NoSQL Explained DevFeed: [SQL vs NoSQL Explained]() Original publisher: [Read original article]() Author: Nelson Djalo Published: 2024-07-02T16:00:57Z Content type: comparison Language: en Sources: [Amigoscode Newsletter]() Topics: [NoSQL](), [SQL](), [Databases](), [clickhouse]() Tags: [acid](), [business-intelligence](), [consistency](), [data-analysis](), [database](), [database-scalability](), [durability](), [microsoft](), [mysql](), [nosql](), [olap](), [oracle](), [postgresql](), [rdbms](), [relational-database-management-systems-rdbms](), [relational-databases](), [schema](), [sql]() ### AI overview This comparison explains the differences between SQL and NoSQL databases, including their characteristics, scalability models, and common use cases. It covers SQL database features such as structured schemas and ACID compliance, along with relational and OLAP database categories. ### Source excerpt Choosing the Right Database for Your Needs ## Lessons Learned Running Presto at Meta Scale DevFeed: [Lessons Learned Running Presto at Meta Scale]() Original publisher: [Read original article]() Author: Philip Bell Published: 2023-07-16T16:42:21Z Content type: tutorial Language: en Sources: [High Scalability](), [High Scalability]() Topics: [SQL](), [Deployment](), [gateway](), [Open Source](), [Availability](), [releases](), [data centers](), [Hardware]() Tags: [analytics](), [automation](), [availability](), [data-centers](), [database-scalability](), [ddatabase](), [deployment](), [gateway](), [hardware](), [high-availablilty](), [high-performance](), [high-scalability](), [load-balancer](), [open-source](), [presto](), [releases](), [scalability](), [sql]() ### AI overview This article presents lessons from running Presto, an open source SQL query engine, at Meta scale. It discusses deploying new releases across many clusters, maintaining availability during updates, routing queries through the Gateway, and automating cluster operations. ### Source excerpt Presto is a free, open source SQL query engine. We've been using it at Meta for the past ten years, and learned a lot while doing so. Running anything at scale - tools, processes, services - takes problem solving to overcome unexpected challenges. Here are four things we learned while scaling up Presto to Meta scale, and some advice if you're interested in running your own queries at scale. Scaling Presto rapidly to meet growing demands: What challenges did we face? Deploying new Presto releases ## Lessons Learned Running Presto at Meta Scale DevFeed: [Lessons Learned Running Presto at Meta Scale]() Original publisher: [Read original article]() Author: Philip Bell Published: 2023-07-16T15:42:21Z Content type: article Language: en Sources: [High Scalability]() Topics: [presto](), [Meta](), [Automation](), [Deployment](), [gateway](), [SQL](), [Availability](), [Open Source](), [datacenter]() Tags: [analytics](), [automation](), [availability](), [data-centers](), [database-scalability](), [ddatabase](), [deployment](), [gateway](), [high-availablilty](), [high-performance](), [high-scalability](), [load-balancer](), [open-source](), [presto](), [scalability](), [sql]() ### AI overview The article presents lessons from running Presto, an open source SQL query engine, at Meta scale. It discusses deploying releases across many clusters, maintaining availability for interactive queries, using the Gateway load balancer to route queries, and automating cluster updates and capacity management. ### Source excerpt Presto is a free, open source SQL query engine. We've been using it at Meta for the past ten years, and learned a lot while doing so. Running anything at scale - tools, processes, services - takes problem solving to overcome unexpected challenges. Here are four things we learned while