# Estuary

Estuary

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## How to Build Customer-Facing Analytics Pipelines with CDC

DevFeed: [How to Build Customer-Facing Analytics Pipelines with CDC](<https://devfeed.tech/articles/how-to-build-customer-facing-analytics-pipelines-with-cdc-129458.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/customer-facing-analytics-cdc>)

Author: Brian Sheehan

Published: 2026-11-11T06:36:13Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [build](<https://devfeed.tech/tags/build.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [customer](<https://devfeed.tech/tags/customer.md>), [data](<https://devfeed.tech/tags/data.md>), [data-freshness](<https://devfeed.tech/tags/data-freshness.md>), [examples](<https://devfeed.tech/tags/examples.md>), [guide](<https://devfeed.tech/tags/guide.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [isolation](<https://devfeed.tech/tags/isolation.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [practical](<https://devfeed.tech/tags/practical.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [saas](<https://devfeed.tech/tags/saas.md>), [serving](<https://devfeed.tech/tags/serving.md>), [streaming-analytics](<https://devfeed.tech/tags/streaming-analytics.md>), [tenant-isolation](<https://devfeed.tech/tags/tenant-isolation.md>), [with](<https://devfeed.tech/tags/with.md>), [world](<https://devfeed.tech/tags/world.md>)

### AI overview

Customer facing analytics needs a CDC pipeline that keeps dashboard scans off production while meeting freshness, concurrency, tenant isolation, and correctness requirements. The described flow captures database log changes, maintains history and current state, transforms reporting tables, and serves them through a layer that enforces tenant access. Freshness depends on capture lag, destination apply schedules, transformations, serving refresh, and caching; a 30 minute warehouse apply interval can leave data up to about 30 minutes old before transformations. Read replicas move scans off the primary but long queries can be canceled, increase replica lag, or contribute to primary database bloat, depending on configuration. Tenant identifiers must follow rows through the pipeline, with access policies or isolated databases enforced at the serving layer. Monitoring should track processing progress as well as row freshness, and validate current state against history and capture completeness against the source. The excerpt reports Deel reduced reporting latency from 2-6 hours to under 1 hour with Estuary and Snowflake; Connect&GO typically loaded MySQL data to Snowflake within 15 seconds

### Source excerpt

A practical architecture guide to customer-facing analytics with CDC, covering data freshness, serving layers, tenant isolation, and real-world SaaS examples.

## Oracle to S3 as Parquet: Export Tables and Replicate Changes with CDC

DevFeed: [Oracle to S3 as Parquet: Export Tables and Replicate Changes with CDC](<https://devfeed.tech/articles/oracle-to-s3-as-parquet-export-tables-and-replicate-changes-with-cdc-121706.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/oracle-to-s3>)

Author: Seán Whelan

Published: 2026-11-06T12:27:12Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [streaming data to S3 with Parquet](<https://devfeed.tech/topics/streaming-data-to-s3-with-parquet.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [Oracle Database](<https://devfeed.tech/topics/oracle-database.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Debezium Connector](<https://devfeed.tech/topics/debezium-connector.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [changes](<https://devfeed.tech/tags/changes.md>), [compare](<https://devfeed.tech/tags/compare.md>), [dms](<https://devfeed.tech/tags/dms.md>), [export](<https://devfeed.tech/tags/export.md>), [goldengate](<https://devfeed.tech/tags/goldengate.md>), [managed](<https://devfeed.tech/tags/managed.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [replicate](<https://devfeed.tech/tags/replicate.md>), [s3](<https://devfeed.tech/tags/s3.md>), [tables](<https://devfeed.tech/tags/tables.md>)

### AI overview

Oracle-to-S3 Parquet choices depend on whether you need change capture: DMS and log-based CDC tools read redo and capture deletes, while exports and scheduled query jobs see surviving rows only. All approaches must handle Oracle type mapping and immutable Parquet files, where updates and deletes become new rows; current-state queries therefore deduplicate by key and filter the latest delete. The excerpt details DMS setup, including archived redo and supplemental logging, Parquet and timestamp settings, bounded LOB sizing, and file batching. It also describes Estuary's LogMiner capture and S3 materialization: backfill runs alongside redo mining, changes append on a configurable upload interval, and deletes and key changes are represented as rows. Estuary's stated tradeoffs include skipped LOB and several other column types, string or numeric mappings that may need configuration, no hard-delete or truncate handling, and retained historical files. Alternatives include Autonomous Database exports for simple full-query exports, and Spark or Glue for custom transforms; those query-based approaches do not capture deletes.

### Source excerpt

Move Oracle to S3 as Parquet with AWS DMS, AWS Glue, GoldenGate or a managed CDC pipeline. Compare how each handles NUMBER, DATE, LOBs and deletes.

## Webhooks to Kafka: How to Receive Events and Publish Them to Topics

DevFeed: [Webhooks to Kafka: How to Receive Events and Publish Them to Topics](<https://devfeed.tech/articles/webhooks-to-kafka-how-to-receive-events-and-publish-them-to-topics-121710.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/webhooks-to-kafka>)

Author: Seán Whelan

Published: 2026-11-06T12:27:07Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [compare](<https://devfeed.tech/tags/compare.md>), [events](<https://devfeed.tech/tags/events.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [http](<https://devfeed.tech/tags/http.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [managed](<https://devfeed.tech/tags/managed.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [retries](<https://devfeed.tech/tags/retries.md>), [webhooks](<https://devfeed.tech/tags/webhooks.md>)

### AI overview

Webhook providers cannot post directly to Kafka, so an HTTPS receiver must verify each request and publish it. The excerpt compares a custom service or HTTP gateway, API Gateway with Lambda, Kafka Connect, and a managed pipeline; the choice depends on operational control, AWS or Connect infrastructure, and who hosts the endpoint. For a direct receiver, verify signatures over raw bytes, produce with the delivery ID as key, and return success only after Kafka acknowledges; provider retries make delivery at least once, so redeliveries can become duplicate messages. Providers have different deadlines and retry behavior--Shopify allows five seconds, while GitHub does not automatically redeliver. Estuary instead acknowledges after committing requests to a durable collection, then materializes them to Kafka independently, allowing capture to continue during a broker outage. Its excerpted limits include a 64 MiB request cap, message truncation thresholds, and signature-header formats that do not support GitHub's or Stripe's built-in formats through the custom setting. The text also covers Kafka size and authentication requirements, recovery, deduplication, and alternate routes.

### Source excerpt

Send webhooks to Kafka with a webhook source connector, an HTTP gateway, AWS Lambda or a managed pipeline. Compare signatures, retries and duplicates.

## S3 to Kafka: How to Turn New S3 Files into Kafka Messages

DevFeed: [S3 to Kafka: How to Turn New S3 Files into Kafka Messages](<https://devfeed.tech/articles/s3-to-kafka-how-to-turn-new-s3-files-into-kafka-messages-121707.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/s3-to-kafka>)

Author: Seán Whelan

Published: 2026-11-06T12:27:02Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [parquet](<https://devfeed.tech/topics/parquet.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [compare](<https://devfeed.tech/tags/compare.md>), [files](<https://devfeed.tech/tags/files.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [managed](<https://devfeed.tech/tags/managed.md>), [messages](<https://devfeed.tech/tags/messages.md>), [ordering](<https://devfeed.tech/tags/ordering.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [s3](<https://devfeed.tech/tags/s3.md>), [setup](<https://devfeed.tech/tags/setup.md>), [source](<https://devfeed.tech/tags/source.md>), [spark](<https://devfeed.tech/tags/spark.md>)

### AI overview

S3 files can become Kafka messages through Kafka Connect, S3 event notifications with Lambda, Spark or Flink, or a managed pipeline. The main differences are how each discovers and tracks files, supported parsing and transformation, and setup; most approaches deliver at least once, so consumers may need to handle duplicates. Listing tools can miss late-arriving keys when they track only the last key, while re-reading modified files can produce duplicates. The guide details S3 and Kafka permissions, format and message-size constraints, and operational tradeoffs. Estuary's described route lists a bucket prefix, parses new files into a durable collection, then materializes records to Kafka; it runs every five minutes by default, delivers at least once, and has specific Parquet size, field truncation, schema-subject, authentication, and deletion limitations. Event-triggered Lambda is positioned for a modest number of uploads needing quick reaction, and Spark or Flink for files requiring transformation.

### Source excerpt

Move S3 data into Kafka with an S3 source connector, AWS Lambda, Spark or a managed pipeline. Compare file ordering, formats, duplicates and setup.

## Oracle to ClickHouse: CDC, Batch & Type Mapping Guide

DevFeed: [Oracle to ClickHouse: CDC, Batch & Type Mapping Guide](<https://devfeed.tech/articles/oracle-to-clickhouse-cdc-batch-type-mapping-guide-121705.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/oracle-to-clickhouse>)

Author: Seán Whelan

Published: 2026-11-06T12:26:57Z

Content type: tutorial

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Oracle Database](<https://devfeed.tech/topics/oracle-database.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data type](<https://devfeed.tech/topics/data-type.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>)

Tags: [batch](<https://devfeed.tech/tags/batch.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [guide](<https://devfeed.tech/tags/guide.md>), [mapping](<https://devfeed.tech/tags/mapping.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [type](<https://devfeed.tech/tags/type.md>)

### AI overview

For Oracle tables that change continuously, the excerpt recommends log-based CDC through LogMiner: it reads redo logs in SCN order, captures hard deletes, and resumes from a checkpoint if the required archived logs remain available. Setup requires ARCHIVELOG mode, supplemental logging, adequate log retention, and primary keys on replicated tables. ClickHouse stores changes as row versions; ReplacingMergeTree collapses versions by sorting key during merges, while queries may need FINAL before merges complete. The excerpt also explains tombstones and delete markers, key-change handling, type-mapping pitfalls (especially Oracle DATE and NUMBER), insert batching, and recovery limits. Method choice depends on needs: file exports and scheduled batch copies suit snapshots or reporting; Debezium with Kafka fits teams already running that stack, though its ClickHouse sink needs pre-created tables and does not support deletes directly; ClickPipes has no Oracle CDC source. Estuary is presented as a managed LogMiner capture and ClickHouse materialization, with configurable sync frequency and documented type limitations, including skipped LOB and binary columns.

### Source excerpt

Move data from Oracle to ClickHouse: map types, handle updates and deletes with ReplacingMergeTree, and compare 4 CDC and batch methods.

## Salesforce to Iceberg: How to Load CRM Objects and Keep Tables Current

DevFeed: [Salesforce to Iceberg: How to Load CRM Objects and Keep Tables Current](<https://devfeed.tech/articles/salesforce-to-iceberg-how-to-load-crm-objects-and-keep-tables-current-121708.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/salesforce-to-iceberg>)

Author: Seán Whelan

Published: 2026-11-06T12:26:52Z

Content type: tutorial

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [etl](<https://devfeed.tech/topics/etl.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [compare](<https://devfeed.tech/tags/compare.md>), [copy](<https://devfeed.tech/tags/copy.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [load](<https://devfeed.tech/tags/load.md>), [salesforce](<https://devfeed.tech/tags/salesforce.md>), [zero-copy](<https://devfeed.tech/tags/zero-copy.md>)

### AI overview

Keeping Salesforce data current in Iceberg requires an initial Bulk API 2.0 load followed by REST `queryAll` sweeps over `SystemModstamp` and `MERGE INTO` operations keyed on `Id`. Sweeps consume the org's shared API allocation; `IsDeleted` records need explicit treatment, purged records are not returned, and formula fields that depend on parent records need scheduled rereads. The document compares a custom Spark/dlt pipeline, AWS Glue zero-ETL for hourly AWS syncs, Snowflake Openflow (Preview), and a managed pipeline; Data 360 zero copy reads existing Iceberg tables rather than copying Salesforce objects into a catalog you control. It describes an Estuary route that polls every five minutes by default and merges on a 30-minute schedule, with daily formula refreshes and EMR Serverless Spark. Its stated limits include unsupported external and big objects, double-typed currency and percent fields, and variant columns requiring Spark 4 unless disabled. Polling and merge frequency trade freshness against API use, compute, and file maintenance.

### Source excerpt

Salesforce to Iceberg: compare 4 ways to load CRM objects into Iceberg tables, see where Data 360 zero copy fits, and handle deletes and formulas.

## Salesforce to S3: Export Objects as Parquet and Capture Changes

DevFeed: [Salesforce to S3: Export Objects as Parquet and Capture Changes](<https://devfeed.tech/articles/salesforce-to-s3-export-objects-as-parquet-and-capture-changes-121709.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/salesforce-to-s3>)

Author: Seán Whelan

Published: 2026-11-06T12:26:46Z

Content type: tutorial

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>)

Tags: [17](<https://devfeed.tech/tags/17.md>), [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [about](<https://devfeed.tech/tags/about.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [capture](<https://devfeed.tech/tags/capture.md>), [changes](<https://devfeed.tech/tags/changes.md>), [compare](<https://devfeed.tech/tags/compare.md>), [export](<https://devfeed.tech/tags/export.md>), [fields](<https://devfeed.tech/tags/fields.md>), [managed](<https://devfeed.tech/tags/managed.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [python](<https://devfeed.tech/tags/python.md>), [s3](<https://devfeed.tech/tags/s3.md>), [salesforce](<https://devfeed.tech/tags/salesforce.md>), [types](<https://devfeed.tech/tags/types.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Salesforce-to-S3 Parquet exports rely on an initial object load followed by polling `SystemModstamp` or consuming change events; updates append new rows, and deletes appear only when the method includes deleted records. The choice depends on scale and destination: AppFlow suits a few objects in AWS but has a 15 GB per-run cap and needs `preserveSourceDataTyping` enabled to retain types; Glue zero-ETL writes Iceberg tables that can be compacted but omits some objects; Python allows custom logic; managed pipelines suit many objects or destinations. Plan for shared Salesforce API limits, per-object S3 prefixes, and incremental recovery. Formula values can change without `SystemModstamp`, so they need periodic rereads; append-only files then require a query that combines the latest full row with refreshed formula values. Purged records cannot be recovered, and reloading can duplicate rows. Validate types, edits, deletes, and new fields after loading.

### Source excerpt

Move Salesforce to S3 as Parquet with Amazon AppFlow, AWS Glue, Python or a managed pipeline. Compare how each handles types, formula fields and deletes.

## Kafka to ClickHouse: Streaming Ingestion, Duplicates & Tombstones Guide

DevFeed: [Kafka to ClickHouse: Streaming Ingestion, Duplicates & Tombstones Guide](<https://devfeed.tech/articles/kafka-to-clickhouse-streaming-ingestion-duplicates-tombstones-guide-121703.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/kafka-to-clickhouse>)

Author: Seán Whelan

Published: 2026-11-06T12:26:41Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [self-hosted databases vs managed services](<https://devfeed.tech/topics/self-hosted-databases-vs-managed-services.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>)

Tags: [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [compare](<https://devfeed.tech/tags/compare.md>), [connect](<https://devfeed.tech/tags/connect.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [engine](<https://devfeed.tech/tags/engine.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [guide](<https://devfeed.tech/tags/guide.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [stream](<https://devfeed.tech/tags/stream.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [table](<https://devfeed.tech/tags/table.md>)

### AI overview

Kafka ingestion into ClickHouse is generally at least once, so replayed messages can create duplicates; reliable designs need batching and tables suited to event or keyed data. The article compares the Kafka table engine, ClickPipes, the Kafka Connect sink, and managed pipelines: the engine runs consumers on ClickHouse servers, ClickPipes is recommended for ClickHouse Cloud, and the sink fits teams already running Kafka Connect. For event streams, store partition and offset in a ReplacingMergeTree sorting key; keyed topics with tombstones need ReplacingMergeTree keyed by message key and queries with FINAL. Tombstone support differs: ClickPipes can delete rows in exactly-once mode or mark them deleted, while the Connect sink lists deletes as unsupported and the table engine's documentation does not describe null-value messages. Retention limits recovery, and compaction can remove tombstones after the configured retention period. Large, infrequent inserts reduce part pressure; monitor lag and active parts. The article also describes Estuary capture and materialization, which keeps Kafka progress in its own checkpoints and writes to ClickHouse on a configurable schedule.

### Source excerpt

Stream Kafka to ClickHouse with the Kafka table engine, ClickPipes or Kafka Connect. Compare 4 methods, delivery guarantees and duplicate handling.

## NetSuite to ClickHouse: SuiteAnalytics Connect, Sync Methods & Ledger Modeling Guide

DevFeed: [NetSuite to ClickHouse: SuiteAnalytics Connect, Sync Methods & Ledger Modeling Guide](<https://devfeed.tech/articles/netsuite-to-clickhouse-suiteanalytics-connect-sync-methods-ledger-modeling-guide-121704.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/netsuite-to-clickhouse>)

Author: Seán Whelan

Published: 2026-11-06T12:26:35Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [clickhouse](<https://devfeed.tech/topics/clickhouse.md>), [elt](<https://devfeed.tech/topics/elt.md>), [build vs buy database solutions](<https://devfeed.tech/topics/build-vs-buy-database-solutions.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2026](<https://devfeed.tech/tags/2026.md>), [3](<https://devfeed.tech/tags/3.md>), [64-bit](<https://devfeed.tech/tags/64-bit.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [compare](<https://devfeed.tech/tags/compare.md>), [connect](<https://devfeed.tech/tags/connect.md>), [custom](<https://devfeed.tech/tags/custom.md>), [custom-fields](<https://devfeed.tech/tags/custom-fields.md>), [data](<https://devfeed.tech/tags/data.md>), [elt](<https://devfeed.tech/tags/elt.md>), [fields](<https://devfeed.tech/tags/fields.md>), [guide](<https://devfeed.tech/tags/guide.md>), [ledger](<https://devfeed.tech/tags/ledger.md>), [limits](<https://devfeed.tech/tags/limits.md>), [netsuite](<https://devfeed.tech/tags/netsuite.md>), [script](<https://devfeed.tech/tags/script.md>), [sync](<https://devfeed.tech/tags/sync.md>), [tool](<https://devfeed.tech/tags/tool.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

NetSuite-to-ClickHouse replication can use SuiteAnalytics Connect, REST/SuiteQL, exports, or an ELT platform; Connect is a paid, read-only add-on and is the route detailed here. Syncs backfill tables, then poll last-modified fields, re-read tables without usable cursors, and consult NetSuite's DeletedRecord log. Since deleted transaction lines are not logged, they require periodic re-reads and cleanup. In ClickHouse, use ReplacingMergeTree keyed by NetSuite IDs (transaction lines need `(transaction, id)`) and query with `FINAL` to resolve versions. For financial accuracy, model accounting lines by subsidiary and accounting book, and preserve amounts as Decimal; Estuary's Connect driver reads numeric values as Float64, so its guide recommends `castToString` followed by `toDecimal64` for exact sums. Estuary provides incremental capture and ClickHouse materialization, but has specific limits: accounting lines are refreshed through changed transactions, custom fields must be selected, and its SuiteQL connector without Connect is full-refresh only. The excerpt also outlines token-auth setup, role permissions, backfill, polling, and trial-balance validation.

### Source excerpt

Move NetSuite data to ClickHouse with SuiteAnalytics Connect, an ELT tool or a script. Compare 5 methods and plan for deletes, custom fields and limits.

## What your CDC pipeline does when a primary key changes

DevFeed: [What your CDC pipeline does when a primary key changes](<https://devfeed.tech/articles/what-your-cdc-pipeline-does-when-a-primary-key-changes-121702.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/cdc-primary-key-changes>)

Author: Dani Pálma

Published: 2026-11-06T05:20:53Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Debezium Connector](<https://devfeed.tech/topics/debezium-connector.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>)

Tags: [cdc](<https://devfeed.tech/tags/cdc.md>), [cdc-tools](<https://devfeed.tech/tags/cdc-tools.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [data-privacy](<https://devfeed.tech/tags/data-privacy.md>), [database](<https://devfeed.tech/tags/database.md>), [debezium](<https://devfeed.tech/tags/debezium.md>), [delete](<https://devfeed.tech/tags/delete.md>), [docs](<https://devfeed.tech/tags/docs.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [estuary](<https://devfeed.tech/tags/estuary.md>), [key](<https://devfeed.tech/tags/key.md>), [mysql](<https://devfeed.tech/tags/mysql.md>), [tests](<https://devfeed.tech/tags/tests.md>), [updates](<https://devfeed.tech/tags/updates.md>)

### AI overview

A CDC pipeline can silently retain a stale row when a primary key changes: the connector must emit a delete for the old key, and the destination must apply it. In tests with MySQL, Debezium 3.0.8's JDBC sink left orphans with its default delete.enabled=false; enabling deletes matched the source. Estuary's MySQL capture handled the tested key changes with both soft-delete and hard-delete Postgres materializations; soft-deleted rows remain stored and must be filtered from live results. FULL row images provide old and new values but add log data, and correctness still depends on connector and sink behavior. To check a pipeline, update a key in staging, compare destination rows with source IDs, then test key swaps and reuse. The text also cautions that incomplete or ambiguous old keys may prevent a safe delete, and before-images can expose sensitive data.

### Source excerpt

A hands-on look at how CDC pipelines handle primary key updates, deletes, and orphan rows, with tests using MySQL, Debezium, and Estuary.

## HubSpot to ClickHouse: Sync Methods, Setup & API Limits Guide

DevFeed: [HubSpot to ClickHouse: Sync Methods, Setup & API Limits Guide](<https://devfeed.tech/articles/hubspot-to-clickhouse-sync-methods-setup-api-limits-guide-118921.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/hubspot-to-clickhouse>)

Author: Seán Whelan

Published: 2026-11-04T22:42:36Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [hubspot](<https://devfeed.tech/topics/hubspot.md>), [clickhouse](<https://devfeed.tech/topics/clickhouse.md>), [elt](<https://devfeed.tech/topics/elt.md>), [API](<https://devfeed.tech/topics/api.md>), [REST API](<https://devfeed.tech/topics/rest-api.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [compare](<https://devfeed.tech/tags/compare.md>), [dedupe](<https://devfeed.tech/tags/dedupe.md>), [elt](<https://devfeed.tech/tags/elt.md>), [guide](<https://devfeed.tech/tags/guide.md>), [hubspot](<https://devfeed.tech/tags/hubspot.md>), [limits](<https://devfeed.tech/tags/limits.md>), [managed](<https://devfeed.tech/tags/managed.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [setup](<https://devfeed.tech/tags/setup.md>), [sync](<https://devfeed.tech/tags/sync.md>)

### AI overview

HubSpot can feed ClickHouse through CSV for one-off analysis, custom API code, open-source ELT, or a managed connector. Ongoing syncs must account for HubSpot API quotas, delayed search indexing, and incremental searches that omit archived or deleted records; calculated properties also need separate refreshes. In ClickHouse, use ReplacingMergeTree keyed by record ID and query with FINAL to deduplicate versions. The text describes Estuary polling recent-change and delayed search endpoints, retrying 429s, and refreshing calculated properties daily, but its HubSpot connector does not remove archived or merged records; those require reconciliation. Its associations omit labels and type IDs. Setup also depends on read scopes, a suitable ClickHouse version and grants, network access, sync scheduling, and batch sizes that avoid excessive parts. Rebuild history depends on collection storage retention, and property typing can trade convenience for precision.

### Source excerpt

Sync HubSpot to ClickHouse with a managed tool, open-source ELT or the HubSpot API. Compare 4 methods, plan for rate limits and dedupe with FINAL.

## MongoDB to S3: Parquet Files, Change Streams & Nested Documents Guide

DevFeed: [MongoDB to S3: Parquet Files, Change Streams & Nested Documents Guide](<https://devfeed.tech/articles/mongodb-to-s3-parquet-files-change-streams-nested-documents-guide-118922.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/mongodb-to-s3>)

Author: Seán Whelan

Published: 2026-11-04T22:41:24Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [streaming data to S3 with Parquet](<https://devfeed.tech/topics/streaming-data-to-s3-with-parquet.md>), [MongoDB](<https://devfeed.tech/topics/mongodb.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [AWS Database Migration Service](<https://devfeed.tech/topics/aws-database-migration-service.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>), [DLT](<https://devfeed.tech/topics/data-load-tool.md>), [Amazon DocumentDB](<https://devfeed.tech/topics/amazon-documentdb.md>)

Tags: [change](<https://devfeed.tech/tags/change.md>), [mongodb](<https://devfeed.tech/tags/mongodb.md>), [nested](<https://devfeed.tech/tags/nested.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [parquet-files](<https://devfeed.tech/tags/parquet-files.md>), [s3](<https://devfeed.tech/tags/s3.md>)

### AI overview

MongoDB change streams can feed append-only Parquet files in S3, but updates and deletes become new rows; readers must retain the latest row per `_id` and filter delete markers to represent current state. Change streams require a replica set or sharded cluster and an oplog window long enough to cover outages. A time-window copy or batch job misses deletes. Atlas Data Federation, AWS DMS, custom Spark/Glue/dlt jobs, and managed CDC services differ in setup and how they map nested BSON data; structs, flattened columns, and JSON text are the described options. Schema changes affect later files, and Decimal128 or digit-only strings can lose precision or leading zeros unless written as suitable decimal or string types. Estuary's described route captures changes durably and appends Parquet on a configurable interval, with nested-field depth settings; its Decimal128 and integer inference behavior may require `castToString`. The excerpt also gives S3 permissions, lifecycle, validation, and recovery requirements, including re-reading collections after an oplog history gap.

### Source excerpt

Move MongoDB to S3 as Parquet with Atlas Data Federation, AWS DMS, Spark or a managed pipeline. Compare how each handles nested documents and deletes.

## Multi-Tenant CDC: Database-per-Tenant SaaS Architecture Guide

DevFeed: [Multi-Tenant CDC: Database-per-Tenant SaaS Architecture Guide](<https://devfeed.tech/articles/multi-tenant-cdc-database-per-tenant-saas-architecture-guide-116277.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/multi-tenant-cdc>)

Author: Brian Sheehan

Published: 2026-11-03T08:35:50Z

Content type: tutorial

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Per-user Database](<https://devfeed.tech/topics/per-user-database.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [tenant data protection](<https://devfeed.tech/topics/tenant-data-protection.md>), [rdbms](<https://devfeed.tech/topics/rdbms.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [database](<https://devfeed.tech/tags/database.md>), [estuary](<https://devfeed.tech/tags/estuary.md>), [guide](<https://devfeed.tech/tags/guide.md>), [keys](<https://devfeed.tech/tags/keys.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [practical](<https://devfeed.tech/tags/practical.md>), [saas](<https://devfeed.tech/tags/saas.md>)

### AI overview

For database-per-tenant PostgreSQL SaaS, log-based CDC consolidates tenant data downstream by running one capture and replication slot per database, tagging each row with a stable tenant ID, and keying shared destination tables on tenant ID plus the source primary key. The guide's three-tenant test found that captures fail and recover independently, while shared materializations and PostgreSQL server resources can affect all tenants. Practical setup includes unique slot names, explicit table mappings, careful discovery and schema rollout, and monitoring each capture and slot; shared-server slot limits and stalled-slot WAL retention constrain scale. Schema drift can pause shared delivery or reject incompatible changes, and resets may rebuild data for every tenant. The test used sample data and does not establish production-scale behavior. Estuary supplies capture, tagging, grouping, and status features, with per-tenant connector-instance costs; self-managed connectors, ordinary CDC on already-shared tables, or scheduled ETL for workloads that tolerate it are noted alternatives.

### Source excerpt

A practical guide to multi-tenant CDC with PostgreSQL, tenant-aware keys, and shared analytics tables using Estuary.

## Bigger Warehouse, Slower Queries: Benchmarking The Top 7 Cloud Data Warehouses

DevFeed: [Bigger Warehouse, Slower Queries: Benchmarking The Top 7 Cloud Data Warehouses](<https://devfeed.tech/articles/bigger-warehouse-slower-queries-benchmarking-the-top-7-cloud-data-warehouses-102802.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/data-warehouse-benchmark>)

Author: Team Estuary

Published: 2026-10-27T14:21:02Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [database-optimization](<https://devfeed.tech/topics/database-optimization.md>), [cloud cost management](<https://devfeed.tech/topics/cloud-cost-management.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>)

Tags: [1-5](<https://devfeed.tech/tags/1-5.md>), [2026](<https://devfeed.tech/tags/2026.md>), [about](<https://devfeed.tech/tags/about.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-data](<https://devfeed.tech/tags/cloud-data.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [estuary](<https://devfeed.tech/tags/estuary.md>), [measuring](<https://devfeed.tech/tags/measuring.md>), [performance](<https://devfeed.tech/tags/performance.md>), [performance-scalability](<https://devfeed.tech/tags/performance-scalability.md>), [queries](<https://devfeed.tech/tags/queries.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [top](<https://devfeed.tech/tags/top.md>), [warehouse](<https://devfeed.tech/tags/warehouse.md>)

### AI overview

Estuary and CorrDyn benchmarked Snowflake, Databricks, BigQuery, Redshift, Microsoft Fabric, ClickHouse, and MotherDuck across 244 queries, data sizes from 1 GB to 1 TB, and up to 100 simultaneous sessions. They used the same generated rows in each warehouse; engines ran untuned with default layouts and automatic scaling off. At 1 GB and 10 GB, multiplying compute by 16 produced only 0.8 to 1.5 times the speed in tested configurations, and the largest size could be slower. The excerpt says rankings shifted more between join-heavy reporting and scan-heavy analytics than with compute size, while useful concurrency and idle-time billing also affect decisions. Its takeaway is to size for the workload, concurrency, and cost rather than assume a universally fastest warehouse. Detailed engine results and cost comparisons are not included in the excerpt.

### Source excerpt

Estuary and CorrDyn benchmarked 7 cloud data warehouses across 244 queries, measuring performance, scalability, and cost from 1 GB to 1 TB.

## How Much Does Real-Time Data Ingestion Cost? Pricing Models Compared

DevFeed: [How Much Does Real-Time Data Ingestion Cost? Pricing Models Compared](<https://devfeed.tech/articles/how-much-does-real-time-data-ingestion-cost-pricing-models-compared-95300.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/real-time-data-ingestion-cost>)

Author: Dani Pálma

Published: 2026-10-22T09:40:44Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>)

Tags: [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [pricing](<https://devfeed.tech/tags/pricing.md>), [real-time](<https://devfeed.tech/tags/real-time.md>)

### AI overview

Real-time ingestion has no universal per-GB price: vendors meter bytes, distinct monthly active rows (MAR), events, credits, connectors, or running capacity. Compare total costs for the same change volume, distinct keys, destinations, and freshness, including destination processing, storage, network, backfills, and engineering. In the excerpt's illustrative estimates, the cheapest qualifying monthly total varies by workload: per-GB pricing favors mostly new rows, while MAR can favor frequent updates to the same rows if its sync frequency meets freshness needs. Multiple destinations can multiply connection or delivery charges. Measure serialized change volume and monthly distinct keys, check retry and backfill rules, price each vendor's published meter or quote, and add cloud and operating costs. Estimates use stated assumptions and list rates, not measured customer benchmarks; actual costs depend on workload and configuration.

### Source excerpt

Compare real-time data ingestion pricing by GB, active rows, compute, and connectors. Learn what drives total cost and how to estimate your pipeline budget.

## How to Migrate a Production CDC Pipeline With Minimal Downtime and No Data Gaps

DevFeed: [How to Migrate a Production CDC Pipeline With Minimal Downtime and No Data Gaps](<https://devfeed.tech/articles/how-to-migrate-a-production-cdc-pipeline-with-minimal-downtime-and-no-data-gaps-87144.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/migrate-production-cdc-pipeline>)

Author: Emily Lucek

Published: 2026-10-12T07:42:13Z

Content type: tutorial

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [data migrations](<https://devfeed.tech/topics/data-migrations.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [Redpanda](<https://devfeed.tech/topics/redpanda-data-streaming.md>)

Tags: [cdc](<https://devfeed.tech/tags/cdc.md>), [data](<https://devfeed.tech/tags/data.md>), [downtime](<https://devfeed.tech/tags/downtime.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [migration](<https://devfeed.tech/tags/migration.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [risks](<https://devfeed.tech/tags/risks.md>)

### AI overview

This guide explains how to migrate a production change data capture (CDC) pipeline while limiting data gaps and consumer disruption. It recommends starting the new pipeline with its own backfill and replication position, running both pipelines in parallel into separate tables, validating data including deletes at a shared boundary, and switching consumers in groups. It also covers source log retention, rollback planning, monitoring, and differences among CDC tools and databases.

### Source excerpt

Migrating a production CDC pipeline requires careful planning to avoid missing changes, duplicate records, or disrupting downstream systems. This guide covers the key steps, risks, and strategies for a safe migration with minimal downtime.

## Monitoring a Pipeline with Estuary's Observability Tools

DevFeed: [Monitoring a Pipeline with Estuary's Observability Tools](<https://devfeed.tech/articles/monitoring-a-pipeline-with-estuary-s-observability-tools-82076.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/data-pipeline-monitoring-prometheus>)

Author: Ruhee Shrestha

Published: 2026-10-02T07:32:10Z

Content type: tutorial

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [observability](<https://devfeed.tech/topics/observability.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [Prometheus](<https://devfeed.tech/topics/prometheus.md>), [data observability](<https://devfeed.tech/topics/data-observability.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [dashboard](<https://devfeed.tech/tags/dashboard.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [observability](<https://devfeed.tech/tags/observability.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [prometheus](<https://devfeed.tech/tags/prometheus.md>)

### AI overview

This tutorial demonstrates how to monitor a real-time data pipeline built with Estuary. It uses Estuary's dashboard, alerts, and flowctl for operational visibility, then connects the OpenMetrics API to Prometheus for queries, dashboards, and custom alerts tracking freshness, throughput, backlog, and errors. It also explains how to use these signals together to locate issues in ingestion, pipeline processing, or downstream materialization.

### Source excerpt

Learn how to monitor a real-time data pipeline with Estuary's native dashboard, alerts, flowctl, and OpenMetrics API, then extend monitoring with Prometheus to track freshness, throughput, backlog, errors, and overall pipeline health.

## Why Transactional Consistency Matters

DevFeed: [Why Transactional Consistency Matters](<https://devfeed.tech/articles/why-transactional-consistency-matters-82430.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/transactional-consistency-matters>)

Author: Mark Van de Wiel

Published: 2026-09-21T16:37:21Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [consistency](<https://devfeed.tech/topics/consistency.md>), [strict-serializability](<https://devfeed.tech/topics/strict-serializability.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [building-software](<https://devfeed.tech/tags/building-software.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [data](<https://devfeed.tech/tags/data.md>), [data-management-best-practices](<https://devfeed.tech/tags/data-management-best-practices.md>), [database](<https://devfeed.tech/tags/database.md>), [databases](<https://devfeed.tech/tags/databases.md>), [real-time](<https://devfeed.tech/tags/real-time.md>)

### AI overview

The article explains transactional consistency and read consistency, then argues that they matter increasingly as analytics systems consume more frequent data and support operational decisions. It connects reliable, consistent data to autonomous agents, which may act on incomplete or unexpected inputs.

### Source excerpt

Transactional consistency is an important concept for applications and their underlying databases. Learn why this concept can no longer be neglected for analytical applications with near real-time data movement and autonomous agents.

## Does CDC Slow Down a Production Database? The Real Cost of Log-Based CDC

DevFeed: [Does CDC Slow Down a Production Database? The Real Cost of Log-Based CDC](<https://devfeed.tech/articles/does-cdc-slow-down-a-production-database-the-real-cost-of-log-based-cdc-82034.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/cdc-database-performance>)

Author: Emily Lucek

Published: 2026-09-21T12:50:17Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [self-hosted databases vs managed services](<https://devfeed.tech/topics/self-hosted-databases-vs-managed-services.md>), [read-replica](<https://devfeed.tech/topics/read-replica.md>), [database-consistency](<https://devfeed.tech/topics/database-consistency.md>)

Tags: [behavior](<https://devfeed.tech/tags/behavior.md>), [cases](<https://devfeed.tech/tags/cases.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [database-performance](<https://devfeed.tech/tags/database-performance.md>), [has](<https://devfeed.tech/tags/has.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [production](<https://devfeed.tech/tags/production.md>), [replication](<https://devfeed.tech/tags/replication.md>), [vacuum](<https://devfeed.tech/tags/vacuum.md>), [wal](<https://devfeed.tech/tags/wal.md>)

### AI overview

The article explains how PostgreSQL log-based change data capture (CDC) affects production database performance. It covers WAL generation, decoding resource use, configuration costs such as REPLICA IDENTITY FULL, and how stalled replication slots and old vacuum horizons can lead to retained WAL or table bloat. The impact varies by workload and available capacity.

### Source excerpt

See where PostgreSQL CDC adds overhead, from WAL and logical decoding to replication slots, vacuum pressure, and production query performance.

## How Gazette separates live streaming from object-store history

DevFeed: [How Gazette separates live streaming from object-store history](<https://devfeed.tech/articles/a-decade-of-gazette-a-streaming-broker-that-doesn-t-store-data-82139.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/gazette-streaming-broker-architecture>)

Author: Dani Pálma

Published: 2026-09-17T04:47:37Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [etcd](<https://devfeed.tech/topics/etcd.md>), [rocksdb](<https://devfeed.tech/topics/rocksdb.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Apache Pulsar](<https://devfeed.tech/topics/pulsar.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [estuary](<https://devfeed.tech/tags/estuary.md>), [exactly-once](<https://devfeed.tech/tags/exactly-once.md>)

### AI overview

Gazette combines synchronous replication of live writes with immutable historical fragments in object storage, using one journal offset space across both. Consumers can read historical data directly without routing backfills through live brokers. The article explains transactional appends, etcd coordination, recovery logs, ownership fencing, and how consumer transactions provide exactly-once committed effects above an at-least-once append API.

### Source excerpt

See how Gazette separates live streaming from object-store history to support fast backfills, disposable brokers, and exactly-once effects in Estuary.

## PostgreSQL CDC Backfills at Scale: Running a Multi-Day Backfill While Production Keeps Writing

DevFeed: [PostgreSQL CDC Backfills at Scale: Running a Multi-Day Backfill While Production Keeps Writing](<https://devfeed.tech/articles/postgresql-cdc-backfills-at-scale-running-a-multi-day-backfill-while-production-keeps-writing-82302.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/postgres-cdc-backfill>)

Author: Emily Lucek; Dani Pálma

Published: 2026-09-10T09:45:31Z

Content type: tutorial

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [PostgreSQL](<https://devfeed.tech/topics/postgresql.md>), [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [load](<https://devfeed.tech/tags/load.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [scale](<https://devfeed.tech/tags/scale.md>), [wal](<https://devfeed.tech/tags/wal.md>)

### AI overview

The guide explains how large PostgreSQL CDC backfills interact with ongoing production writes. It covers WAL retention and replication-slot risks, how chunk size affects source load, acknowledgment intervals and restart costs, and options for reducing or recovering historical reads. It also describes Estuary's watermark-and-fence approach for reconciling backfill rows with live changes.

### Source excerpt

Learn how PostgreSQL CDC backfills affect WAL retention, source load, chunk sizing, and recovery while production continues writing.

## What Is Estuary? A Platform for CDC, Streaming & Batch Data

DevFeed: [What Is Estuary? A Platform for CDC, Streaming & Batch Data](<https://devfeed.tech/articles/what-is-estuary-a-platform-for-cdc-streaming-batch-data-82106.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/estuary-platform-overview>)

Author: David Yaffe

Published: 2026-09-01T06:32:33Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Apache Pulsar](<https://devfeed.tech/topics/pulsar.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [estuary](<https://devfeed.tech/tags/estuary.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

### AI overview

Estuary is a data integration platform that captures data from databases, SaaS applications and event streams into durable collections, then delivers it to destinations on continuous or scheduled cadences. The article describes its CDC, batch and streaming capabilities, deployment and security options, cost model, and runtime architecture. It also outlines use cases in analytics, operational systems and AI workflows that depend on current data.

### Source excerpt

Estuary captures data once and lets teams reuse it across warehouses, operational systems, and AI workflows without maintaining separate batch and streaming stacks.

## Why we rebuilt the engine behind Estuary

DevFeed: [Why we rebuilt the engine behind Estuary](<https://devfeed.tech/articles/why-we-rebuilt-the-engine-behind-estuary-82454.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/why-we-rebuilt-estuary>)

Author: David Yaffe

Published: 2026-08-26T10:46:34Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [elt](<https://devfeed.tech/topics/elt.md>), [consistency](<https://devfeed.tech/topics/consistency.md>), [Redpanda](<https://devfeed.tech/topics/redpanda-data-streaming.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>)

Tags: [data-infrastructure](<https://devfeed.tech/tags/data-infrastructure.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [downtime](<https://devfeed.tech/tags/downtime.md>), [estuary](<https://devfeed.tech/tags/estuary.md>), [exactly-once](<https://devfeed.tech/tags/exactly-once.md>), [scaling](<https://devfeed.tech/tags/scaling.md>)

### AI overview

Estuary describes rebuilding its data integration runtime to improve throughput, transactional correctness, and autonomous scaling. The new engine uses Rust, disk-backed staging, coordinated checkpoints, and atomic transaction visibility across logs. New pipelines use it by default, while existing pipelines are moving over gradually.

### Source excerpt

Estuary rebuilt its runtime for faster, more scalable data movement with transactional correctness, autonomous scaling, and simpler operations.

## A Hands-On Guide to Building Streaming Transformation with Estuary Derivation Agent Skills

DevFeed: [A Hands-On Guide to Building Streaming Transformation with Estuary Derivation Agent Skills](<https://devfeed.tech/articles/a-hands-on-guide-to-building-streaming-transformation-with-estuary-derivation-agent-skills-82103.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/estuary-agent-skills-streaming-data-transformations>)

Author: Ruhee Shrestha

Published: 2026-08-20T13:51:25Z

Content type: tutorial

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Agent Skills](<https://devfeed.tech/topics/agent-skills.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Loop Engineering](<https://devfeed.tech/topics/loop-engineering.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>)

Tags: [agent-skills](<https://devfeed.tech/tags/agent-skills.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-assistant](<https://devfeed.tech/tags/ai-assistant.md>), [ai-assistants](<https://devfeed.tech/tags/ai-assistants.md>), [ai-coding-assistant](<https://devfeed.tech/tags/ai-coding-assistant.md>), [ai-coding-assistants](<https://devfeed.tech/tags/ai-coding-assistants.md>), [ai-development](<https://devfeed.tech/tags/ai-development.md>), [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [build](<https://devfeed.tech/tags/build.md>), [building](<https://devfeed.tech/tags/building.md>), [data](<https://devfeed.tech/tags/data.md>), [estuary](<https://devfeed.tech/tags/estuary.md>), [guide](<https://devfeed.tech/tags/guide.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [published](<https://devfeed.tech/tags/published.md>), [python](<https://devfeed.tech/tags/python.md>), [skills](<https://devfeed.tech/tags/skills.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [streaming-data](<https://devfeed.tech/tags/streaming-data.md>), [using](<https://devfeed.tech/tags/using.md>)

### AI overview

This tutorial demonstrates Estuary Derivation Agent Skills for building streaming data transformations from natural-language prompts. Using a PostgreSQL banking dataset, it walks through filtering, aggregation, stateful metrics, joins, array flattening, rolling windows, and Python-based merchant classification before materializing results in ClickHouse. It also highlights production concerns such as reviewing generated logic and schemas, handling updates in CDC streams, bounding state, and checking multi-shard behavior.

### Source excerpt

Build streaming data transformations with Estuary Agent Skills using natural-language prompts, from filtering and joins to stateful logic, windowing, and Python.

[Next page](<https://devfeed.tech/sources/estuary.md?cursor=WyIyMDI2LTA4LTIwVDEzOjUxOjI1KzAwOjAwIiwgIjQ4NTIzMjg2LWFjNzgtNDdjNy1hZGI2LTFhMDdhMDQ0ZTYzYyJd>)