# Capturing Every Change From Shopify's Sharded Monolith

DevFeed: [Capturing Every Change From Shopify's Sharded Monolith](<https://devfeed.tech/articles/capturing-every-change-from-shopify-s-sharded-monolith-1340.md>)

Original publisher: [Read original article](<https://shopify.engineering/capturing-every-change-shopify-sharded-monolith>)

Author: John Martin

Published: 2021-03-12T18:51:39Z

Content type: article

Language: en

Sources: [Shopify Engineering](<https://devfeed.tech/sources/shopify-engineering.md>), [Shopify Engineering - Shopify Engineering](<https://devfeed.tech/sources/shopify-engineering-shopify-engineering.md>)

Topics: [data-platforms](<https://devfeed.tech/topics/data-platforms.md>), [Shopify](<https://devfeed.tech/topics/shopify.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [data](<https://devfeed.tech/topics/data.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Databases](<https://devfeed.tech/topics/databases.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-storage](<https://devfeed.tech/tags/cloud-storage.md>), [data](<https://devfeed.tech/tags/data.md>), [data-platforms](<https://devfeed.tech/tags/data-platforms.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [databases](<https://devfeed.tech/tags/databases.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [latency](<https://devfeed.tech/tags/latency.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [platforms](<https://devfeed.tech/tags/platforms.md>), [shopify](<https://devfeed.tech/tags/shopify.md>), [spark](<https://devfeed.tech/tags/spark.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

## AI overview

Shopify describes how its data architecture evolved from an internal warehouse into a cloud-based data lake and separate platforms for internal workloads and merchant analytics. The article covers batch extraction, streaming change events, Apache Spark and dbt transformations, Kafka-based pipelines, and serving data through systems including BigQuery, Redshift, and Google Bigtable.

## Source excerpt

Shopify is moving from existing query-based CDC (Change-Data Capture) tooling (Longboat) to an immutable, append-only, log-based mechanism. This has required a complete re-platforming, involving a shift to new, open source technology such as Kafka Connect and Debezium.