# Integrating Confluent Schema Registry with Apache Spark applications

DevFeed: [Integrating Confluent Schema Registry with Apache Spark applications](<https://devfeed.tech/articles/integrating-confluent-schema-registry-with-apache-spark-applications-24745.md>)

Original publisher: [Read original article](<https://medium.com/yazio-engineering/integrating-confluent-schema-registry-with-apache-spark-applications-d3426e33bc51?source=rss----65bd178b00af---4>)

Author: Dominik Liebler

Published: 2022-01-24T08:04:19Z

Content type: tutorial

Language: en

Sources: [YAZIO Engineering - Medium](<https://devfeed.tech/sources/yazio-engineering-medium.md>)

Topics: [Kafka](<https://devfeed.tech/topics/kafka.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Kotlin](<https://devfeed.tech/topics/kotlin.md>), [Kubernetes](<https://devfeed.tech/topics/kubernetes.md>), [ceph](<https://devfeed.tech/topics/ceph.md>), [JSON Schema](<https://devfeed.tech/topics/json-schema.md>)

Tags: [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [backpressure](<https://devfeed.tech/tags/backpressure.md>), [ceph](<https://devfeed.tech/tags/ceph.md>), [confluent](<https://devfeed.tech/tags/confluent.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [json](<https://devfeed.tech/tags/json.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [kotlin](<https://devfeed.tech/tags/kotlin.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [payload](<https://devfeed.tech/tags/payload.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [schema](<https://devfeed.tech/tags/schema.md>), [schemaregistry](<https://devfeed.tech/tags/schemaregistry.md>), [serialization](<https://devfeed.tech/tags/serialization.md>), [spark](<https://devfeed.tech/tags/spark.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

## AI overview

This engineering article explains YAZIO's data pipeline from mobile and web applications through Kafka and Spark Structured Streaming into a Ceph-based data lake. It discusses why schemas matter and describes replacing JSON with Apache Avro and Confluent Schema Registry to reduce message size while keeping schema information externally stored and cached.

## Source excerpt

At YAZIO, we believe in making decisions backed by data to help people live healthier lives through better nutrition. For each new and existing feature we want to evaluate how well it performs and how our users interact with it. In order to do so, we need a lot of data and we need to handle backpressure in our systems. To cope with that we use a Kafka cluster managed by Strimzi operators running in Kubernetes. The data itself is being ingested from our mobile and web apps via HTTP or TCP endpoints serialized into JSON and stored in Kafka by a small application written in Kotlin/JVM. Overview of our data pipeline architecture At the other end of the pipeline, different Spark Structured Streaming applications (also written in Kotlin) dump this information into our data lake residing in a Ceph bucket. They read data from Kafka, deserialize it, transform some of the fields and write Parquet files into the data lake using a new schema. Why schemas? Schemas play an important role in data pipelines because they give meaning and context to data. In a world without schemas we would still do random interpretations about the context and meaning of data every now and then when using it. As you might have guessed already this would lead to a lot of bugs and misunderstandings. Photo by EJ Strat https://unsplash.com/photos/VjWi56AWQ9k Similar to a legal contract that binds you to certain limits, a schema binds the data to certain limits and meaning which narrow down the need of interpretation. Choice of serialization formats At the time of writing, Confluent Schema Registry supports these three serialization formats: Apache Avro Protocol Buffers (protobuf) JSON Schema From those choices, only two really provide more than just validation of the data that is ingested and transmitted through our data pipelines. Avro and Protobuf also allow us to shrink the sizes of our topics because only the payload is contained in a message, while the repeating schema will not be stored. In the cas