# Replicating Database Changes to a Message Queue is Tricky

DevFeed: [Replicating Database Changes to a Message Queue is Tricky](<https://devfeed.tech/articles/replicating-database-changes-to-a-message-queue-is-tricky-20759.md>)

Original publisher: [Read original article](<https://www.evanjones.ca/replicating-db-to-queue.html>)

Published: 2022-12-13T02:01:42Z

Content type: tutorial

Language: en

Sources: [Evan Jones](<https://devfeed.tech/sources/evan-jones.md>)

Topics: [Database](<https://devfeed.tech/topics/database.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [systems](<https://devfeed.tech/topics/systems.md>)

Tags: [database](<https://devfeed.tech/tags/database.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [message-queue](<https://devfeed.tech/tags/message-queue.md>), [systems](<https://devfeed.tech/tags/systems.md>)

## AI overview

This article explains why replicating database changes to a message queue can produce missing or extra updates when components fail. It compares parallel and sequential approaches and recommends using a database transaction to update application state and record pending messages at a single commit point.

## Source excerpt

Let's imagine we have an program that stores its state in a database, and we want other programs to do things when changes occur. For example, we might want to send email notifications if a bank balance drops below a threshold. This is a very common reason applications use message queues like Kafka. Unfortunately, the "trivial" implementation does not work when components fail. I suspect there are many real applications that get this wrong. Most of the time, these applications work correctly, and the changes are replicated across multiple systems. However, when things restart, updates can go missing, or extra updates can appear. In this article, I'm going to try to explain how this can go wrong, and some ways to fix it. Attempt one: Update both in parallel The application performs the following operations: Write the change to the database. At the same time, publish the message to the message queue. The problem in this case is that the database update could fail, but publishing the message succeeds. This means applications consuming the stream receive an "extra" update that does not exist in the database. Attempt two: Update database, then publish message Okay, let's try again, and make sure that updating the database succeeds: Write the change to the database. Wait for the database to confirm the write occurred. Publish the message to the message queue. We fixed the "extra" message update problem! However, we still have a problems: If the application crashes after writing to the database, but before publishing the message, the stream is missing an update. This can be particularly bad if the message queue is unavailable. The application can retry publishing the message for a while. However, if the message queue is down for long enough, it is likely the application will run out of memory, or be restarted. In this case, all the pending updates are lost. So now what? We can't do the operations sequentially, and we can't do them in parallel. The trick is to order the wor