# database-corruption

Published articles for database-corruption.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Database corruption in PostgreSQL: our journey to improving our upgrade process

DevFeed: [Database corruption in PostgreSQL: our journey to improving our upgrade process](<https://devfeed.tech/articles/database-corruption-in-postgresql-our-journey-to-improving-our-upgrade-process-26259.md>)

Original publisher: [Read original article](<https://medium.com/adyen/database-corruption-in-postgresql-our-journey-to-improving-our-upgrade-process-d76d39e5b696?source=rss----64941d9fbc09---4>)

Author: Adyen

Published: 2025-01-24T18:42:23Z

Content type: article

Language: en

Sources: [Adyen Tech](<https://devfeed.tech/sources/adyen-tech.md>)

Topics: [PostgreSQL](<https://devfeed.tech/topics/postgresql.md>), [Database](<https://devfeed.tech/topics/database.md>), [upgrade](<https://devfeed.tech/topics/upgrade.md>), [Disk image](<https://devfeed.tech/topics/disk-image.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>)

Tags: [adyen](<https://devfeed.tech/tags/adyen.md>), [backup](<https://devfeed.tech/tags/backup.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [database-corruption](<https://devfeed.tech/tags/database-corruption.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [scale](<https://devfeed.tech/tags/scale.md>), [storage](<https://devfeed.tech/tags/storage.md>), [toast-storage](<https://devfeed.tech/tags/toast-storage.md>), [transactions](<https://devfeed.tech/tags/transactions.md>), [upgrade](<https://devfeed.tech/tags/upgrade.md>)

### AI overview

Adyen engineers describe how corrupted PostgreSQL TOAST data surfaced after a database upgrade from version 9.6 to 13. The corruption affected millions of rows, could not be restored from a pre-upgrade backup, and led to a nearly year-long investigation into data access, detection, and upgrade-process improvements.

### Source excerpt

by Cagri Biroglu & Derk van Veen, Database Engineers, Adyen PostgreSQL is a core component of Adyen's payment platform, enabling high-throughput, low-latency transaction processing on a global scale. Its stability, scalability, and extensibility make it a critical part of our infrastructure, supporting the reliability required for financial operations. A few years ago, we upgraded our PostgreSQL fleet from version 9.6 to 13. Upgrades are always long and tedious, but we prepared well for them, and the upgrade on even the hardest clusters with an extremely high number of TPS(transactions per second) and a size of multiple hundreds of TB(terabytes) went without major difficulties. But a few years later, we suddenly saw occasional errors related to transactions not being visible or invalid x_min or x_max. We tried to analyze the issue but didn't have much time, as transaction wraparound is always just around the corner in this environment. We had some quiet weeks, where no error was seen, and we thought that whatever it was, it was now behind us. We were wrong. TOAST storage The errors became a frequent nuisance, and we had no other option but to figure out what was happening. As it turned out, we had an issue with corrupted TOAST data entered in a table in the two weeks before we did the upgrade from 9.6 to 13 on this table's cluster. After the first analysis, we realized: This corruption affected millions of rows. We couldn't restore this table from a backup before the upgrade. If possible, we want to regain access to our data. We only knew about this corruption because of a clean-up job. It was unclear how many other tables were affected by the same issue. TOAST storage is not something we have understood and mastered yet. Transaction logic on TOAST is even more complicated than transaction logic on main tables. This is the start of a journey that will last almost a year. Background on TOAST storage Let's start with some background on TOAST data, as the corruptions a