# Automated Schema Evolution in Pinterest's Next-Generation DB Ingestion Framework

DevFeed: [Automated Schema Evolution in Pinterest's Next-Generation DB Ingestion Framework](<https://devfeed.tech/articles/automated-schema-evolution-in-pinterest-s-next-generation-db-ingestion-framework-1227.md>)

Original publisher: [Read original article](<https://medium.com/pinterest-engineering/automated-schema-evolution-in-pinterests-next-generation-db-ingestion-framework-36c5c07070de?source=rss----4c5a5f6279b6---4>)

Author: Pinterest Engineering

Published: 2026-06-24T18:01:01Z

Content type: article

Language: en

Sources: [Pinterest Engineering Blog - Medium](<https://devfeed.tech/sources/pinterest-engineering-blog-medium.md>)

Topics: [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Automation](<https://devfeed.tech/topics/automation.md>), [Databases](<https://devfeed.tech/topics/databases.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Pull Request](<https://devfeed.tech/topics/pull-request.md>), [Code generation](<https://devfeed.tech/topics/code-generation.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [monitor](<https://devfeed.tech/topics/monitor.md>), [Provisioning](<https://devfeed.tech/topics/provisioning.md>)

Tags: [automation](<https://devfeed.tech/tags/automation.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [icebergs](<https://devfeed.tech/tags/icebergs.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [monitor](<https://devfeed.tech/tags/monitor.md>), [pinterest](<https://devfeed.tech/tags/pinterest.md>), [process](<https://devfeed.tech/tags/process.md>), [schema-evolution](<https://devfeed.tech/tags/schema-evolution.md>), [software](<https://devfeed.tech/tags/software.md>), [spark](<https://devfeed.tech/tags/spark.md>), [storage](<https://devfeed.tech/tags/storage.md>)

## AI overview

Pinterest describes an automated schema-evolution framework for its CDC-based database ingestion platform, which uses Kafka, Flink, Spark, and Iceberg. The framework propagates supported schema changes across the stack, uses pull-request-based versioning and auditing, targets SLA-based consistency between online and offline schemas, and provides recovery paths for unsupported or ambiguous changes.

## Source excerpt

Yisheng Zhou | Software Engineer II Liang Mou | Sr Staff Software Engineer Gabriel Raphael Garcia Montoya | Staff Software Engineer Istvan Podor | Staff Software Engineer Introduction In the first post of this series, we introduced Pinterest's next-generation CDC-based ingestion platform built on Kafka, Flink, Spark, and Iceberg. In production, upstream schemas are constantly evolving, and in a distributed CDC pipeline, schema is not just metadata -- it is a cross-system contract spanning ingestion, transformation, storage, and historical backfill. A schema change that is not handled carefully can break Flink jobs, block Spark upserts, or create inconsistencies between online and offline representations. This post walks through how we make schema evolution safe in practice: the onboarding model it builds on, the changes we support and the tradeoffs we accept, how updates propagate across the stack, and how we monitor and recover when things go wrong. We end with where we're heading next -- zero-gap schema evolution. Background & Motivation Schema evolution was particularly challenging in our environment because our ingestion pipeline is composed of multiple tightly related stages, each of which depends on schema in a different way. A single table onboarding flow spans CDC source configuration, Kafka provisioning, Flink and Spark code generation, Iceberg CDC and base table creation, and historical bootstrap -- all driven by the same schema. Once a source schema changes, that change must be reflected across code, metadata, and storage in a controlled and auditable way. Without a unified automation framework, manual updates across these layers increase the risk of drift, failed deployments, and inconsistent datasets. We therefore needed a way to make schema evolution safe, repeatable, and scalable. Our Solution: SLA-Based Automated Schema Evolution To address this problem, we built an automated schema evolution framework on top of our CDC ingestion platform, providing: au