# Data replication across backend services with Kafka and Protobuf

DevFeed: [Data replication across backend services with Kafka and Protobuf](<https://devfeed.tech/articles/data-replication-across-backend-services-with-kafka-and-protobuf-20096.md>)

Original publisher: [Read original article](<https://medium.com/jobteaser-dev-team/data-replication-across-backend-services-with-kafka-and-protobuf-cad1d9fd9f90?source=rss----bd77d16a0035---4>)

Author: Emmanuel Joubaud

Published: 2023-04-14T10:01:57Z

Content type: tutorial

Language: en

Sources: [JobTeaser](<https://devfeed.tech/sources/jobteaser.md>)

Topics: [Back end](<https://devfeed.tech/topics/backend.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Replication](<https://devfeed.tech/topics/replication.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [API](<https://devfeed.tech/topics/api.md>), [Structured-data](<https://devfeed.tech/topics/structured-data.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [backend](<https://devfeed.tech/tags/backend.md>), [data](<https://devfeed.tech/tags/data.md>), [data-replication](<https://devfeed.tech/tags/data-replication.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [microservices](<https://devfeed.tech/tags/microservices.md>), [replication](<https://devfeed.tech/tags/replication.md>), [serialization](<https://devfeed.tech/tags/serialization.md>), [serialization-format](<https://devfeed.tech/tags/serialization-format.md>), [software-architecture](<https://devfeed.tech/tags/software-architecture.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>)

## AI overview

This tutorial explains Jobteaser's Silos architecture for replicating data across backend services. It describes using Kafka topics and Google Protobuf messages so consumer services can maintain local database copies while avoiding synchronous service-to-service API calls.

## Source excerpt

The Jobteaser application contains a lot of different relatively independent modules to help universities provide career guidance to students: a job board, a career event management system, a career advice appointment management system... When we decided to migrate our application's backend from a monolith to a service-oriented architecture, we strived to keep each module as isolated as possible from the others in the event of an incident. If the career appointment system was down, students should still be able to browse and apply to job ads. That isolation is achieved through what we've called our Silos architecture. The gist is we avoid synchronous API calls between backend services, and prefer asynchronous data integration between services. There are a lot of different ways to implement asynchronous communication between services, and few companies share the details of theirs so we had to figure out a lot of stuff on our own. Now that we've refined our system, we thought we'd share the details of our approach, based on simple data replication using Kafka and Protobuf. How it worksWhen a write operation happens, data is changed first on the Source of Truth service, who then publishes it to a Kafka topic that consumers can subscribe to, in order to replicate their own local copy in their databaseSource of Truth and consumers Every table in our data model has an owner service, also called the Source of Truth (SoT) for that data, or the producer. For instance, the job board service may be the SoT for the job ad data. The owner service has 3 main responsibilities: it receives and validates all write requests for the data it owns (Create, Update, Delete) it stores the data to its own database, i.e. the Source of Truth, the authoritative state of the data if it's data that other services might want to access, it publishes its latest state as a message into a Kafka topic Consumer services that are interested in accessing a given table can then subscribe to the Kafka topic