# The fellowship of the forgotten

DevFeed: [The fellowship of the forgotten](<https://devfeed.tech/articles/the-fellowship-of-the-forgotten-35655.md>)

Original publisher: [Read original article](<https://engineering.klarna.com/the-fellowship-of-the-forgotten-d341045a6123?source=rss----86090d14ab52---4>)

Author: Onno Vos Dev

Published: 2025-02-26T09:04:27Z

Content type: article

Language: en

Sources: [Klarna Engineering](<https://devfeed.tech/sources/klarna-engineering.md>)

Topics: [Database](<https://devfeed.tech/topics/database.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [zero downtime deployment](<https://devfeed.tech/topics/zero-downtime-deployment.md>), [consistency](<https://devfeed.tech/topics/consistency.md>), [database-consistency](<https://devfeed.tech/topics/database-consistency.md>), [copy](<https://devfeed.tech/topics/copy.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>), [Erlang](<https://devfeed.tech/topics/erlang.md>)

Tags: [2018](<https://devfeed.tech/tags/2018.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [custom](<https://devfeed.tech/tags/custom.md>), [downtime](<https://devfeed.tech/tags/downtime.md>), [erlang](<https://devfeed.tech/tags/erlang.md>), [klarna](<https://devfeed.tech/tags/klarna.md>), [leveldb](<https://devfeed.tech/tags/leveldb.md>), [memory](<https://devfeed.tech/tags/memory.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [replication](<https://devfeed.tech/tags/replication.md>), [serializable](<https://devfeed.tech/tags/serializable.md>), [software-development](<https://devfeed.tech/tags/software-development.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [source-of-truth](<https://devfeed.tech/tags/source-of-truth.md>)

## AI overview

Klarna migrated KRED from a 15 TB Mnesia cluster to Postgres with no observed downtime, using a staged transition that kept rollback possible. Sharding was set aside because surrounding systems and required components were not ready. Instead, the team preserved KRED's kdb API as a compatibility layer, added record types and null handling for Postgres, and replicated data between Mnesia and Postgres so either could lead. Batching helped Postgres replication keep pace; prepared transactions were chosen for Postgres-to-Mnesia replication because they changed less of the stable Mnesia follower logic, despite adding latency. A live key-level comparison process checked that writes and deletes converged across replicas. After the cutover, logical decoding of transaction-log messages in Postgres WAL replaced prepared transactions, requiring an upgrade to Postgres 14 and a decoder change; this reduced transaction time by about 20 ms. The authors emphasize proof of concept work, small useful steps, and extensive testing and monitoring. The excerpt says isolation-level details are reserved for a second post.

## Source excerpt

How we migrated from Mnesia to Postgres with zero downtime Back in December 2004, an Erlang application was born called KRED (referring to the freshly-started company called Kreditor, now known as Klarna). KRED is one of the "servicing systems" at Klarna and keeps track of consumer debt (among other things). It was powered by Mnesia and consisted of a cluster of 7 nodes, each holding a full copy of the database on disk. The data was replicated using a custom replication mechanism built in-house by Klarna. One node was elected as the leader and its database was considered the source of truth in the system. All database transactions were executed on the leader and writes were replicated to the rest of the nodes, the so-called followers. The Mnesia database was around 15 TB and at its peak in 2018 around 1.3 TB was held in memory at all times. Considering that few suppliers were selling hardware with such specs, it's easy to claim the crown of one of the biggest Mnesia databases in terms of in-memory storage, that was running in production. The rest of the data was offloaded to disk using mnesia_eleveldb. KRED has been a stable workhorse at Klarna so why change a winning concept? Get ready, for a two part blog post where we'll first go through our journey of how we went about this and secondly, how we made Mnesia behave just like Postgres and implemented our version serializable isolation level on top of Postgres! How the journey started Three engineers, sat down in a bar in Stockholm, Sweden and asked this question: 'When Klarna truly takes off, will KRED survive? Assuming "no", and presented with a blanco check, how would we tackle this problem?' The answer quickly revolved around the issues of running Mnesia on an even larger cluster and with leveldb compaction hitting some hot tables during peak times. One can only imagine how that problem would just continue to get worse over time. Considering the three engineers had worked on KRED for a long time, scaling KRED wa