# From Batch Snapshots to Near-Real-Time Data

DevFeed: [From Batch Snapshots to Near-Real-Time Data](<https://devfeed.tech/articles/from-batch-snapshots-to-near-real-time-data-20029.md>)

Original publisher: [Read original article](<https://technology.doximity.com/articles/from-batch-snapshots-to-near-real-time-data>)

Author: Doximity

Published: 2026-08-04T16:29:00Z

Content type: article

Language: en

Sources: [Doximity](<https://devfeed.tech/sources/doximity.md>)

Topics: [data](<https://devfeed.tech/topics/data.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [MySQL](<https://devfeed.tech/topics/mysql.md>), [Amazon Aurora](<https://devfeed.tech/topics/amazon-aurora.md>)

Tags: [amazon-aurora](<https://devfeed.tech/tags/amazon-aurora.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [data](<https://devfeed.tech/tags/data.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [mysql](<https://devfeed.tech/tags/mysql.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

## AI overview

Doximity describes combining batch database snapshots with Change Data Capture to make data available in minutes while retaining batch-based consistency and recovery guarantees. The design uses Kafka and includes a trusted snapshot, metadata-preserving routing, a base-plus-delta view, and synthetic cascade deletes. In a 12-day measurement window, 95% of sampled events reached the queryable intermediate layer within eight minutes of publication to Kafka.

## Source excerpt

Change Data Capture (CDC) is often presented as a straightforward pipeline: read a database transaction log, publish each change, and apply those changes to another system. That description is accurate, but it leaves out many of the decisions that determine whether the resulting data can be trusted. At Doximity, we already had a batch pipeline that periodically copied snapshots of application databases into our data warehouse. Those snapshots were reliable, but their freshness was measured in hours. We introduced CDC to make changes available in minutes so downstream transformations and operational analytics would not have to wait for the next batch snapshot. We continued using the batch pipeline for the consistency and recovery guarantees it already provided. Over a 12-day measurement window, 95% of events from a stratified sample of active tables reached the queryable intermediate layer within eight minutes of being published to Kafka. The most interesting parts of the project were not the connections from a source database to Kafka or from Kafka to Snowflake, but four questions we had to answer around them: How could we reuse our existing, transactionally consistent batch snapshots as an on-demand starting point for CDC, without reprocessing every existing row? How could we onboard new tables and absorb schema changes from many source databases across our products without growing operational overhead for each one? How could new changes become queryable without waiting for the warehouse to merge them into place? How could we handle cascading child deletes that MySQL performs but never emits as individual binary-log events? Our answers are the four design decisions in this article: a trusted batch snapshot, metadata-preserving routing, a base-plus-delta view, and synthetic cascade deletes. Together, they turned a stream of row changes into a system we could bootstrap, scale, validate, and recover. The sections that follow explain the tradeoffs and guardrails so rea