# From Batch Snapshots to Near-Real-Time Data

DevFeed: [From Batch Snapshots to Near-Real-Time Data](<https://devfeed.tech/articles/from-batch-snapshots-to-near-real-time-data-20029.md>)

Original publisher: [Read original article](<https://technology.doximity.com/articles/from-batch-snapshots-to-near-real-time-data>)

Author: Doximity

Published: 2026-08-04T16:29:00Z

Content type: article

Language: en

Sources: [Doximity](<https://devfeed.tech/sources/doximity.md>)

Topics: [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [transaction-log](<https://devfeed.tech/topics/transaction-log.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [application](<https://devfeed.tech/tags/application.md>), [apply](<https://devfeed.tech/tags/apply.md>), [article](<https://devfeed.tech/tags/article.md>), [base](<https://devfeed.tech/tags/base.md>), [batch](<https://devfeed.tech/tags/batch.md>), [bootstrap](<https://devfeed.tech/tags/bootstrap.md>), [but](<https://devfeed.tech/tags/but.md>), [can](<https://devfeed.tech/tags/can.md>), [capture](<https://devfeed.tech/tags/capture.md>), [cascade](<https://devfeed.tech/tags/cascade.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [change](<https://devfeed.tech/tags/change.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [changes](<https://devfeed.tech/tags/changes.md>), [data](<https://devfeed.tech/tags/data.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [real-time-data](<https://devfeed.tech/tags/real-time-data.md>), [snapshots](<https://devfeed.tech/tags/snapshots.md>), [time](<https://devfeed.tech/tags/time.md>)

## AI overview

Doximity added CDC to make application database changes available in minutes while retaining its batch pipeline for transactionally consistent snapshots and recovery. In a 12-day measurement window, 95% of sampled events from active tables reached the queryable intermediate layer within eight minutes of publication to Kafka. The excerpt identifies four design decisions: use trusted batch snapshots to bootstrap CDC, preserve metadata while routing changes across source databases, expose new changes through a base-plus-delta view before warehouse merging, and synthesize cascade deletes that MySQL does not emit as individual binary-log events. The excerpt begins describing point-in-time snapshots using Amazon Aurora, but does not provide the full implementation details or tradeoffs for the four decisions.

## Source excerpt

Change Data Capture (CDC) is often presented as a straightforward pipeline: read a database transaction log, publish each change, and apply those changes to another system. That description is accurate, but it leaves out many of the decisions that determine whether the resulting data can be trusted. At Doximity, we already had a batch pipeline that periodically copied snapshots of application databases into our data warehouse. Those snapshots were reliable, but their freshness was measured in hours. We introduced CDC to make changes available in minutes so downstream transformations and operational analytics would not have to wait for the next batch snapshot. We continued using the batch pipeline for the consistency and recovery guarantees it already provided. Over a 12-day measurement window, 95% of events from a stratified sample of active tables reached the queryable intermediate layer within eight minutes of being published to Kafka. The most interesting parts of the project were not the connections from a source database to Kafka or from Kafka to Snowflake, but four questions we had to answer around them: How could we reuse our existing, transactionally consistent batch snapshots as an on-demand starting point for CDC, without reprocessing every existing row? How could we onboard new tables and absorb schema changes from many source databases across our products without growing operational overhead for each one? How could new changes become queryable without waiting for the warehouse to merge them into place? How could we handle cascading child deletes that MySQL performs but never emits as individual binary-log events? Our answers are the four design decisions in this article: a trusted batch snapshot, metadata-preserving routing, a base-plus-delta view, and synthetic cascade deletes. Together, they turned a stream of row changes into a system we could bootstrap, scale, validate, and recover. The sections that follow explain the tradeoffs and guardrails so rea