# sharding Sharding is a computing technique that distributes or partitions data across multiple independent databases or machines to distribute computational and storage workloads and support scalable processing. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Top System Design Performance Metrics DevFeed: [Top System Design Performance Metrics]() Original publisher: [Read original article]() Author: Saurabh Dashora Published: 2026-07-14T08:36:41Z Content type: tutorial Language: en Sources: [System Design Codex]() Topics: [benchmarking](), [Availability](), [Architecture & Design](), [Scalability](), [Load Balancing](), [health checks](), [Concurrency](), [Disaster Recovery](), [Database](), [sharding](), [IO](), [API]() Tags: [api](), [availability](), [blocking](), [concurrency](), [database](), [disaster-recovery](), [health-checks](), [load-balancing](), [performance](), [performance-metrics](), [scalability](), [sharding](), [system-design]() ### AI overview A tutorial on system design performance metrics, focusing on availability and throughput. It explains how these metrics are measured and outlines techniques such as load balancing, health checks, failover, redundancy, disaster recovery, query optimization, sharding, and asynchronous processing. ### Source excerpt Must Know Metrics ## Scaling a distributed cache: Why consistent hashing is mandatory DevFeed: [Scaling a distributed cache: Why consistent hashing is mandatory]() Original publisher: [Read original article]() Author: hello@ankit-rana.com Published: 2026-05-17T00:00:00Z Content type: tutorial Language: en Sources: [Ankit Rana | Mechanical Sympathy]() Topics: [consistent hashing](), [distributed-systems](), [Caching](), [sharding](), [scaling](), [Database](), [Memcached](), [Redis]() Tags: [caching](), [consistent-hashing](), [database](), [distributed](), [distributed-cache](), [distributed-systems](), [memcached](), [redis](), [scaling](), [sharding]() ### AI overview The article explains why modulo-based cache sharding can remap nearly every key when a node fails, causing a large cache miss surge and database overload. It presents consistent hashing on a fixed ring as a way to limit movement to keys assigned to the failed node, and notes that virtual nodes improve distribution across physical cache nodes. ### Source excerpt Sharding a cache with hash(key) modulo N means the denominator changes when a node dies, remapping nearly every key. Dropping from five nodes to four invalidates about 80 percent of the cache instantly and routes that miss storm straight at the primary database. Consistent hashing maps nodes and keys onto a fixed ring so only the dead node's keys move, and virtual nodes fix the uneven distribution a handful of physical nodes would otherwise produce. ## Kubo 0.40.0 adds reproducible CID Profiles, import cleanup, diagnostics, and gateway improvements DevFeed: [Kubo 0.40.0 adds reproducible CID Profiles, import cleanup, diagnostics, and gateway improvements]() Original publisher: [Read original article]() Author: Ipfs Published: 2026-02-25T00:00:00Z Content type: release Language: en Sources: [IPFS]() Topics: [Software](), [sharding](), [Go Language](), [configuration]() Tags: [gc](), [go](), [memory](), [release](), [release-notes](), [sharding]() ### AI overview Kubo 0.40.0 introduces reproducible file imports through CID Profiles, automatic cleanup of interrupted flatfs operations, improved connectivity diagnostics and gateway behavior, and reduced memory usage and garbage-collection overhead through its Go 1.26 toolchain. ### Source excerpt Just released: Kubo 0.40.0! ## Scaling LLM Inference: Innovations in Tensor Parallelism, Context Parallelism, and Expert Parallelism DevFeed: [Scaling LLM Inference: Innovations in Tensor Parallelism, Context Parallelism, and Expert Parallelism]() Original publisher: [Read original article]() Author: Cen Zhao; Xiaodong Wang; Jianyu Huang Published: 2025-10-17T16:00:50Z Content type: article Language: en Sources: [Meta AI Research]() Topics: [Large Language Model](), [AI Inference](), [Inference Performance](), [LLM Techniques](), [GPU](), [sharding](), [long-context]() Tags: [ai-research](), [gpu](), [inference](), [kv-cache](), [large-language-models-llms](), [latency](), [llms](), [parallelism](), [performance](), [sharding]() ### AI overview Meta describes three forms of parallelism--tensor, context, and expert parallelism--for scaling large language model inference across GPUs. The article explains how prefill and decoding differ computationally and how these techniques target resource efficiency, throughput, and latency. ### Source excerpt At Meta, we are constantly pushing the boundaries of LLM inference systems to power applications such as the Meta AI App. We're sharing how we developed and implemented advanced parallelism techniques to optimize key performance metrics related to resource efficiency, throughput, and latency. The rapid evolution of large language models (LLMs) has ushered in a [...] Read More... The post Scaling LLM Inference: Innovations in Tensor Parallelism, Context Parallelism, and Expert Parallelism appeared first on Engineering at Meta. ## How Klarna Migrated the KRED System from Mnesia to Postgres with Zero Downtime DevFeed: [How Klarna Migrated the KRED System from Mnesia to Postgres with Zero Downtime]() Original publisher: [Read original article]() Author: Onno Vos Dev Published: 2025-02-26T09:04:27Z Content type: article Language: en Sources: [Klarna Engineering]() Topics: [Databases](), [Erlang](), [Replication](), [sharding](), [Transactions](), [systems]() Tags: [cluster](), [databases](), [erlang](), [klarna](), [memory](), [postgresql](), [production](), [replication](), [scaling](), [sharding](), [software-development](), [software-engineering](), [transactions]() ### AI overview This article describes Klarna's migration of KRED, an Erlang servicing system, from Mnesia to Postgres with zero downtime. It explains the original replicated seven-node architecture, the system's scaling challenges, and a plan involving sharding across multiple clusters. ### Source excerpt How we migrated from Mnesia to Postgres with zero downtime Back in December 2004, an Erlang application was born called KRED (referring to the freshly-started company called Kreditor, now known as Klarna). KRED is one of the "servicing systems" at Klarna and keeps track of consumer debt (among other things). It was powered by Mnesia and consisted of a cluster of 7 nodes, each holding a full copy of the database on disk. The data was replicated using a custom replication mechanism built in-house by Klarna. One node was elected as the leader and its database was considered the source of truth in the system. All database transactions were executed on the leader and writes were replicated to the rest of the nodes, the so-called followers. The Mnesia database was around 15 TB and at its peak in 2018 around 1.3 TB was held in memory at all times. Considering that few suppliers were selling hardware with such specs, it's easy to claim the crown of one of the biggest Mnesia databases in terms of in-memory storage, that was running in production. The rest of the data was offloaded to disk using mnesia_eleveldb. KRED has been a stable workhorse at Klarna so why change a winning concept? Get ready, for a two part blog post where we'll first go through our journey of how we went about this and secondly, how we made Mnesia behave just like Postgres and implemented our version serializable isolation level on top of Postgres! How the journey started Three engineers, sat down in a bar in Stockholm, Sweden and asked this question: 'When Klarna truly takes off, will KRED survive? Assuming "no", and presented with a blanco check, how would we tackle this problem?' The answer quickly revolved around the issues of running Mnesia on an even larger cluster and with leveldb compaction hitting some hot tables during peak times. One can only imagine how that problem would just continue to get worse over time. Considering the three engineers had worked on KRED for a long time, scaling KRED wa ## Implementing Multi-GPU Distributed Training for Stitch Fix's Personalized Recommendations DevFeed: [Implementing Multi-GPU Distributed Training for Stitch Fix's Personalized Recommendations]() Original publisher: [Read original article]() Published: 2023-06-08T09:00:00Z Content type: article Language: en Sources: [Stitch Fix]() Topics: [distributed-training](), [recommendations](), [GPU](), [sharding](), [Training AI Models](), [Multi-GPU](), [PyTorch]() Tags: [distributed-training](), [gpu](), [model-training](), [multi-gpu](), [parallel](), [performance](), [pytorch](), [recommendations](), [sharding]() ### AI overview This Stitch Fix engineering article explains how the company implemented multi-GPU distributed training for its Client Time Series Model (CTSM), a PyTorch-based model used in personalized recommendations. It describes sharding training data across GPUs and training mini-batches in parallel to reduce training time, along with the surrounding retraining and deployment workflow. ### Source excerpt Stitch Fix uses a cutting-edge multi-tiered recommender system stack to personalize styling recommendations at scale. This stack comprises several critical components, including feature generation, scoring, ranking, and inventory optimization techniques. Our scoring module is based on the Client Time Series Model (CTSM) which is an award winning novel sequence based model that uses temporally masked encoders. CTSM is built using PyTorch, and was initially trained on a single Graphics Processing Unit (GPU) instance. Since we first put this model into production last year, we have launched several updates to the model that improved its performance. Many of these improvements involved adding new features or increasing the time window of our training data. As a result, the model training time increased significantly, making it harder for us to iterate quickly and get feedback on new ideas we want to try for improving the model. We needed a way to reduce the model training time. This blog delves into the steps we followed to overcome this challenge and our journey to implement multi-GPU distributed model training for CTSM. By sharding the training data across multiple GPUs and training multiple mini-batches in parallel, we aimed to achieve significant reductions in training time. We present empirical results showcasing the observed reduction in training time when we scaled up resources from 1 to N GPUs, and share some future directions we are considering in our continued effort to speed up model training. Model Training Workflow The scores generated by CTSM are leveraged by multiple downstream services to get insight into what items a client is likely to purchase. The model is retrained at a regular cadence to ensure that it is using the most updated information about each client when making predictions and does not degrade in its performance. We leverage configuration driven machine learning pipelines to set up a Directed Acyclic Graph (DAG) that automatically retrains ## How trivago Maintains Global Availability During a Google Cloud Regional Outage DevFeed: [How trivago Maintains Global Availability During a Google Cloud Regional Outage]() Original publisher: [Read original article]() Author: Arne Claus Follow Published: 2022-08-15T00:00:00Z Content type: tutorial Language: en Sources: [Trivago]() Topics: [Google Cloud Platform (GCP)](), [Kubernetes](), [Kafka](), [migration](), [Back end](), [sharding](), [Replication](), [systems](), [MySQL]() Tags: [backend](), [cloud](), [datacenters](), [failover](), [gke](), [google-cloud](), [how-to](), [kafka](), [migration](), [mysql](), [replication](), [sharding](), [systems]() ### AI overview This article explains trivago's regional failover testing and its effort to maintain global availability when a Google Cloud region becomes unavailable. It describes the move from locale-based sharding toward services that can handle equivalent traffic and data across regions. ### Source excerpt How trivago manages to stay globally available, even if a Google Cloud region goes down ## Understanding Monarch, Google's Planet-Scale Monitoring System DevFeed: [Understanding Monarch, Google's Planet-Scale Monitoring System]() Original publisher: [Read original article]() Published: 2021-05-20T00:00:00Z Content type: article Language: en Sources: [Gaurav Sarma's Blog]() Topics: [Monitoring](), [Time Series](), [Google](), [Databases](), [sharding](), [systems](), [Latency](), [Network]() Tags: [database](), [google](), [latency](), [load-balancing](), [logs](), [monitoring](), [series](), [sharding](), [spanner](), [time-series]() ### AI overview This article explains Monarch, Google's planet-scale in-memory time series database and monitoring system. It covers its availability and partitioning goals, regional data placement, in-memory leaves, persistent logs, ingestion routers, range assignment, lexicographic sharding, and metric formats. ### Source excerpt Monarch is a planet-scale in-memory time series database developed by Google. It is mainly used by as a reliable monitoring system by most of Google's internal systems like Spanner, BigTable, Colossus, BlobStore... ## Designing a Workflow engine from first principles DevFeed: [Designing a Workflow engine from first principles]() Original publisher: [Read original article]() Author: Shawn Wang Published: 2021-04-05T07:00:00Z Content type: tutorial Language: en Sources: [Temporal Blog]() Topics: [Architecture & Design](), [Scalability](), [Messaging](), [sharding](), [Routing (disambiguation)](), [consistency](), [Availability](), [Programming](), [Open Source]() Tags: [architecture](), [consistency](), [hashicorp](), [messaging](), [microservices](), [open-source](), [queues](), [routing](), [scalability](), [scalable-architecture](), [temporal](), [temporal-concepts](), [workflow](), [workflow-engine]() ### AI overview This article explains workflow engine design principles from first principles, including task queues, timers, consistency, scalability, sharding, routing, system workflows, and external implementations. It describes how Temporal addresses these requirements and outlines Temporal's history through related systems and projects such as SWF, Durable Task Framework, Cherami, and Cadence. ### Source excerpt Learn key workflow engine design principles and how Temporal's scalable architecture solves challenges like task queues and scalability. ## Increasing the scaling limits of the Firebase Realtime Database DevFeed: [Increasing the scaling limits of the Firebase Realtime Database]() Original publisher: [Read original article]() Author: Jamie Niemasik Published: 2019-09-19T00:00:00Z Content type: news Language: en Sources: [Firebase Blog]() Topics: [Realtime Database](), [Scalability](), [sharding](), [Firebase](), [Firestore]() Tags: [firebase](), [firestore](), [iot](), [launch](), [low-latency](), [realtime-database](), [scalability](), [serverless](), [sharding]() ### AI overview Firebase announces that it has doubled the Firebase Realtime Database concurrent connections limit from 100,000 to 200,000 for existing databases and new projects. The article also explains sharding across database instances for higher aggregate capacity and discusses use cases such as collaborative applications and IoT sensors. ### Source excerpt News, tutorials, and updates from the Firebase team. ## Postgres Open Silicon Valley line-up: First take DevFeed: [Postgres Open Silicon Valley line-up: First take]() Original publisher: [Read original article]() Author: Map Published: 2017-07-01T20:55:56Z Content type: opinion Language: en Sources: [Craig Kerstiens]() Topics: [PostgreSQL](), [scaling](), [Concurrency](), [sharding](), [MySQL]() Tags: [concurrency](), [postgres](), [postgresql](), [scale](), [sharding]() ### AI overview A preview of the combined Postgres Open and PGConf SV conference highlights sessions on self-indexing PostgreSQL with Dexter, scaling beyond a single Postgres database with Citus, concurrency and locking, and running PostgreSQL at Instagram's scale. ### Source excerpt This year Postgres open and PGConf SV have combined to great a bigger and better conference right in downtown San Francisco. I'm obviously biased as I'm one of the co-chairs, and I know every conference organizer says picking the talks was hard, but I'm especially excited for the line-up this year. The hard part for me is going to be which talks do I miss out on because I'm sitting in the other session that's ongoing. You can see the full list of talk and tutorial sessions, but I thought it'd be fun to do a rundown of some of my favorites. How Postgres could index itself Postgres indexing itself has long been on my wishlist. Andrew Kane from Instacart, and creator of PgHero has bottled up many learnings into a new tool: Dexter. I suspect we'll get a look at all that went into this, how it works, and how you can leverage it to have a more automatically tuned database. Scaling a SaaS Application Beyond a Single Postgres with Citus Migration talks are all to common, from Postgres to MySQL from MySQL to Postgres, or from Dynamo to Postgres. But this one is a little different flavor from Postgres to sharded Postgres with Citus. Sharding into a distributed system of course brings new things to consider and think about, and here you'll learn about them from first hand experience so hopefully you can avoid mistakes yourself. Concurrency Deep Dive This one looks to be a great under the hood look as well as likely very practical. It'll cover MVCC which is really at so much of the core of how Postgres works, but then bring it up to what it means for things like locks. Best of all, this one like so many others comes with lots of real world experience from Segment. Postgres window magic Running PostgreSQL @ Instagram Instagram is well known as one of the largest apps in the world. They optimized and changed their setup multiple times and probably scaled in about every way possible. Here we get to learn about all the various things you need in running at a truly astonishing scale ## OSCON, Portland, and PDXPUG DevFeed: [OSCON, Portland, and PDXPUG]() Original publisher: [Read original article]() Author: Dimitri Fontaine PostgreSQL Major Contributor; Author Published: 2013-07-29T15:09:00Z Content type: opinion Language: en Sources: [Dimitri Fontaine]() Topics: [PostgreSQL](), [migration](), [MySQL](), [sharding](), [Databases](), [SQL](), [Perl](), [Python]() Tags: [databases](), [migration](), [mysql](), [perl](), [postgresql](), [python](), [sharding](), [sql]() ### AI overview A personal report from OSCON and PDXPUG in Portland, describing a presentation about Fotolog's migration from a 37-database MySQL sharding cluster to PLProxy on PostgreSQL. It also discusses using PostgreSQL for application logic and middleware, including stored procedures implemented in languages such as PL/Python and PL/Perl. ### Source excerpt After spending an awesome week in San Francisco, CA I'm lucky enough to be spending another week in the USA, in Portand, OR. The main excuse for showing up here has been OSCON where I presented a talk about the fotolog migration from MySQL to PostgreSQL. *[Mark Wong](http://markwkm.blogspot.com/) is doing some serious database crochet work!* Fotolog is a photo sharing website having more than 32 millions of users sharing more than a billion of photos, which made for a very interesting migration use case. In particular switching from a hand-made MySQL sharding cluster of 37 databases to a fully integrated PLProxy setup on 16 servers hosting each 16 shards, in less than 6 months including complete code rewrite, made things... interesting. ## Sharding your database DevFeed: [Sharding your database]() Original publisher: [Read original article]() Author: Map Published: 2012-11-30T20:55:56Z Content type: tutorial Language: en Sources: [Craig Kerstiens]() Topics: [sharding](), [Databases](), [scaling](), [Heroku]() Tags: [database](), [databases](), [development](), [hash](), [heroku](), [scaling](), [sharding]() ### AI overview This guide explains database sharding for applications whose data outgrows the storage or performance limits of a single database. It covers logical and physical shards, choosing shard counts, determining a shard without querying the database, and primary-key considerations when distributing data across multiple databases. ### Source excerpt I'm increasingly encountering users on Heroku that are encountering the need to [shard](http://en.wikipedia.org/wiki/Shard_(database_architecture)) their data. For most users this is something you delay as long as possible as you can generally go for sometime before you have to worry about it. Additionally scaling up your database is often a reasonable approach early on and something I encourage as a starting point as scaling up is easy to do with regards to databases. However, for the 1% of users that do need to shard when the time comes many are left wondering where to start, hence the following guide. What and Why Sharding is the process of splitting up your data so it resides in different tables or often different physical databases. Sharding is helpful when you have some specific set of data that outgrows either storage or reasonable performance within a single database. Logical Shards First when initially implementing sharding you'll want to create an arbitrary number of logical shards. This will allow you to change less code later when it comes to adding more shards. You'll also want to define your shards to the power of 2. Generally I'd recommend for most services 1024 can be a good number, I believe Instagram actually used 4096, either can really be an appropriate number. For simplicity sake lets start with an example of using 4 logical shards. First lets look at an example set of users: id | email | name ----+---------------------------+----------------- 1 | craig.kerstiens@gmail.com | Craig Kerstiens 2 | john.doe@gmail.com | John Doe 3 | jane.doe@gmail.com | Jane Doe 4 | user4@gmail.com | User 4 5 | user5@gmail.com | User 5 6 | user6@gmail.com | User 6 7 | user7@gmail.com | User 7 8 | user8@gmail.com | User 8 Dividing these up into logical shards we're going to have something that looks roughly like this: Its important when sharding that you find a mechanism that requires you to not hit the database. As the above example shows its using the ID of the row