# Snowflake streaming data architecture

A real-time data architecture in which producer-generated streams flow through streaming services into Snowflake for analysis, using technologies such as Kafka, Snowpipe Streaming, Streams, Tasks, Dynamic Tables, Snowpark, and Container Services.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Do You Still Need Airflow for Snowflake Pipelines? An Honest Answer From Someone Who's Run Both

DevFeed: [Do You Still Need Airflow for Snowflake Pipelines? An Honest Answer From Someone Who's Run Both](<https://devfeed.tech/articles/do-you-still-need-airflow-for-snowflake-pipelines-an-honest-answer-from-someone-who-s-run-both-145278.md>)

Original publisher: [Read original article](<https://medium.com/snowflake/do-you-still-need-airflow-for-snowflake-pipelines-an-honest-answer-from-someone-whos-run-both-0e8d4a5a9036?source=rss----34b6daafc07---4>)

Author: Sanket Prabhu

Published: 2026-11-20T01:56:59Z

Content type: opinion

Language: en

Sources: [Snowflake Builders](<https://devfeed.tech/sources/snowflake-builders.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [dag](<https://devfeed.tech/tags/dag.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [task-graph](<https://devfeed.tech/tags/task-graph.md>)

### AI overview

Snowflake task graphs may now cover many pipelines that previously needed Airflow, but the new graph object, atomic deployments, advanced branching, and DCM support are described as Preview features. The article says native graphs suit end-to-end Snowflake workloads owned by one team, with data arriving in Snowflake and a preference for centralized lineage, logs, cost, and access. Airflow remains useful for cross-system dependencies, routine date-range backfills, and a company-wide orchestration control plane; Preview risk may also argue against moving critical production pipelines. A suggested middle ground is to let Airflow handle external readiness and triggers, then execute one Snowflake task graph for retries, branching, notifications, and transformations. Regular tasks with AFTER dependencies are identified as generally available.

### Source excerpt

Snowflake task graphs just grew up. That doesn't mean your DAGs are obsolete. Apache Airflow and Snowflake Graph Every time Snowflake ships a new orchestration feature, I see the same post on my feed: "Airflow is dead." I don't buy it. But I also don't buy the opposite reflex, where every Snowflake pipeline gets an Airflow DAG by default because "that's how we do orchestration here." I've run Airflow in production and I've spent about a decade building data platforms on Snowflake. Over that time, my answer to "do we need Airflow for this?" has changed more than once. With the task graph changes Snowflake released in late October, it changed again. If every step of your pipeline already runs inside Snowflake, Airflow is often a second system you're paying to babysit. That's the short version. The longer version has some important exceptions. Status note up front: the new TASK GRAPH object, atomic deployments, advanced branching and DCM DEFINE TASK GRAPH are all in Preview as of this writing (released Oct 23 and Oct 27, 2026). Regular tasks with AFTER dependencies have been GA for a long time. 1. Why Airflow ended up in so many Snowflake stacks It's worth being fair to Airflow here. Most teams didn't choose it by accident. For years, native Snowflake tasks were fine for simple chains but awkward for real pipelines: Configuration was scattered across every task in the graph Changing a running graph meant suspending the root, editing tasks one by one, and resuming Branching was limited. A child ran when its predecessors succeeded, and handling a failure path cleanly took workarounds Deployment wasn't atomic. A half-applied change could leave the graph in a state nobody intended So teams reached for an orchestrator that already handled all of that. My team was using Airflow because of the operator versatility. Is that still the right call today? 2. What actually changed in October Two releases matter here. Task graphs as a first-class object (Preview, Oct 23). You can no

## Snowflake Is Moving the Database Boundary: Why Postgres, OpenFlow, and Agents Matter More Together

DevFeed: [Snowflake Is Moving the Database Boundary: Why Postgres, OpenFlow, and Agents Matter More Together](<https://devfeed.tech/articles/snowflake-is-moving-the-database-boundary-why-postgres-openflow-and-agents-matter-more-together-143912.md>)

Original publisher: [Read original article](<https://medium.com/snowflake/snowflake-is-moving-the-database-boundary-why-postgres-openflow-and-agents-matter-more-together-a2331da1a745?source=rss----34b6daafc07---4>)

Author: Akshay Sharma

Published: 2026-11-19T15:01:04Z

Content type: opinion

Language: en

Sources: [Snowflake Builders](<https://devfeed.tech/sources/snowflake-builders.md>)

Topics: [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [Database](<https://devfeed.tech/topics/database.md>), [Local AI](<https://devfeed.tech/topics/local-ai.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-pipeline](<https://devfeed.tech/tags/analytics-pipeline.md>), [application](<https://devfeed.tech/tags/application.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [batch](<https://devfeed.tech/tags/batch.md>), [boundary](<https://devfeed.tech/tags/boundary.md>), [capture](<https://devfeed.tech/tags/capture.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [context](<https://devfeed.tech/tags/context.md>), [customer](<https://devfeed.tech/tags/customer.md>), [customer-analytics](<https://devfeed.tech/tags/customer-analytics.md>), [dashboard](<https://devfeed.tech/tags/dashboard.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [decision](<https://devfeed.tech/tags/decision.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [snowflake-cortex](<https://devfeed.tech/tags/snowflake-cortex.md>), [snowflake-openflow](<https://devfeed.tech/tags/snowflake-openflow.md>)

### AI overview

A fast warehouse does not ensure timely operational decisions when data must cross multiple boundaries. The author recounts a financial-services analytics pipeline whose dashboard showed customer status several hours late despite fast queries, due to friction among operational systems, ingestion, warehouse, semantic logic, and the application. The excerpt identifies change data capture, schema drift, batch coordination, reconciliation, and duplicate customer records as burdens. It argues that AI agents need reliable context because, unlike human analysts, they cannot readily interpret stale data or diagnose pipeline failures. The excerpt ends before explaining how Snowflake Postgres, OpenFlow, and agents address these issues.

### Source excerpt

These views are my own, not those of my employer. Continue reading on Snowflake Builders Blog: Data Engineers, App Developers, AI, & Data Science "

## I benchmarked Snowflake Postgres mirroring for two days. Here are the numbers.

DevFeed: [I benchmarked Snowflake Postgres mirroring for two days. Here are the numbers.](<https://devfeed.tech/articles/i-benchmarked-snowflake-postgres-mirroring-for-two-days-here-are-the-numbers-136382.md>)

Original publisher: [Read original article](<https://medium.com/snowflake/i-benchmarked-snowflake-postgres-mirroring-for-two-days-here-are-the-numbers-b50eba22b8de?source=rss----34b6daafc07---4>)

Author: Ondřej Blaha

Published: 2026-11-17T01:00:34Z

Content type: article

Language: en

Sources: [Snowflake Builders](<https://devfeed.tech/sources/snowflake-builders.md>)

Topics: [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [elt](<https://devfeed.tech/topics/elt.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [data type](<https://devfeed.tech/topics/data-type.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [arrays](<https://devfeed.tech/tags/arrays.md>), [batch](<https://devfeed.tech/tags/batch.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [gotchas](<https://devfeed.tech/tags/gotchas.md>), [impact](<https://devfeed.tech/tags/impact.md>), [latency](<https://devfeed.tech/tags/latency.md>), [load](<https://devfeed.tech/tags/load.md>), [mirroring](<https://devfeed.tech/tags/mirroring.md>), [numbers](<https://devfeed.tech/tags/numbers.md>), [oltp](<https://devfeed.tech/tags/oltp.md>), [operations](<https://devfeed.tech/tags/operations.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [postgresql-replication](<https://devfeed.tech/tags/postgresql-replication.md>), [refresh](<https://devfeed.tech/tags/refresh.md>), [replication](<https://devfeed.tech/tags/replication.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [source](<https://devfeed.tech/tags/source.md>), [two-days](<https://devfeed.tech/tags/two-days.md>)

### AI overview

On a Snowflake Postgres STANDARD_M instance in one AWS region, mirrored rows reached the near-real-time `TABLE$live` view in about 26 seconds at the median and 37 seconds at p90 across refresh intervals; the regular table could lag by up to one interval. Under the tested write load, moving from a 30-second to a 2-minute interval cut apply-task cost 73%, while small OLTP transactions showed little source impact and batch inserts into mirrored tables were 20-30% slower. The author recommends starting at 2 minutes, using `$live` for operational freshness, adding primary keys before mirroring, and committing bulk loads in chunks of up to 100,000 rows. Important limits include lossy mappings for some PostgreSQL types, `$changes` history being cleared by column changes, and `RESTART_MIRROR` triggering a full table copy. Results are from one instance size and region with limited repetitions, and are not an SLA.

### Source excerpt

Latency distributions, cost per refresh interval, impact on the source database, bulk loads and operations. Measured on the smallest instance that supports mirroring. Snowflake Postgres, managed PostgreSQL running inside a Snowflake account, has been generally available since 24 February 2026, and data mirroring into Snowflake followed on 15 October 2026. I didn't just want to see that it works. I wanted real numbers under a simulated load, so that I can tell clients how it actually behaves and compare the cost with what we had expected. The marketing says "seconds" and "sub-minute", and nobody had published a latency distribution, a cost per interval or the impact on the source database, so I spent two days measuring them. TL;DR Freshness: new rows show up in TABLE$live after ~26 s (median), ~37 s (p90), no matter whether the refresh interval is 30 seconds or 10 minutes. The table itself lags behind $live by up to one refresh interval. Cost: the serverless apply task cost 0.063 credits/h idle at a 30-second interval and 0.25 credits/h under a steady write load. Switching to 2 minutes cut the loaded cost by 73 %, to 10 minutes by 88 %. Source impact: small OLTP transactions barely notice the mirror. Batch inserts into a mirrored table were 20-30 % slower than into a non-mirrored table on the same instance. Operations: SUSPEND_MIRROR took 2.5-4.7 s under load, RESTART_MIRROR re-copies every table despite saying "soft", adding an 8M-row table to a running mirror took under a minute, and the replication slot survived every suspend/resume of the instance. Gotchas: numeric without precision becomes FLOAT, json/jsonb become TEXT, 2-D arrays become NULL, PostgreSQL refuses ADD COLUMN ... DEFAULT <volatile> on mirrored tables, any ADD, DROP or RENAME COLUMN wipes that table's $changes history, and tables without a primary key turn insert-only. Why this matters Every team that runs an application on Postgres and analytics in Snowflake owns a pipeline between the two. It is usu

## Stream your customers' audit logs into their own Snowflake warehouse

DevFeed: [Stream your customers' audit logs into their own Snowflake warehouse](<https://devfeed.tech/articles/stream-your-customers-audit-logs-into-their-own-snowflake-warehouse-121079.md>)

Original publisher: [Read original article](<https://workos.com/blog/audit-log-streaming-snowflake>)

Author: WorkOS

Published: 2026-11-05T00:00:00Z

Content type: tutorial

Language: en

Sources: [WorkOS](<https://devfeed.tech/sources/workos-blog.md>)

Topics: [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [audit](<https://devfeed.tech/topics/audit.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2026](<https://devfeed.tech/tags/2026.md>), [3](<https://devfeed.tech/tags/3.md>), [audit](<https://devfeed.tech/tags/audit.md>), [audit-logs](<https://devfeed.tech/tags/audit-logs.md>), [auth](<https://devfeed.tech/tags/auth.md>), [customer](<https://devfeed.tech/tags/customer.md>), [customers](<https://devfeed.tech/tags/customers.md>), [event](<https://devfeed.tech/tags/event.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [key](<https://devfeed.tech/tags/key.md>), [logs](<https://devfeed.tech/tags/logs.md>), [pair](<https://devfeed.tech/tags/pair.md>), [schema](<https://devfeed.tech/tags/schema.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [stream](<https://devfeed.tech/tags/stream.md>), [table](<https://devfeed.tech/tags/table.md>), [up](<https://devfeed.tech/tags/up.md>), [warehouse](<https://devfeed.tech/tags/warehouse.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

WorkOS Log Streams can write each organization's audit events as rows in the customer's own Snowflake account, where the customer controls storage and retention. Setup requires a dedicated user authenticated by an RSA key pair, a table with the documented schema, and a default role granted access to the warehouse, database, schema, and table; accounts with network policies must allow WorkOS's published IP addresses. The dashboard and Admin Portal configure the same stream, but the portal lets the customer provide credentials directly so the private key need not pass through support. Monitor actual delivery with a customer-specific freshness query in Snowflake: an Active state alone does not establish that rows are arriving, and the article says not to assume stop-delivery notifications. Error or Invalid streams require fixing the cause and revalidating. Delivery resumes from the last successful event, subject to the organization's WorkOS retention window; previously delivered rows remain subject to the customer's own warehouse policy.

### Source excerpt

Each audit event lands as a row in your customer's own Snowflake table, with key-pair auth, a fixed schema, grants up front, and a freshness check.

## How a one-person data team at Keep built real-time banking infrastructure on dbt

DevFeed: [How a one-person data team at Keep built real-time banking infrastructure on dbt](<https://devfeed.tech/articles/how-a-one-person-data-team-at-keep-built-real-time-banking-infrastructure-on-dbt-102649.md>)

Original publisher: [Read original article](<https://www.getdbt.com/blog/keep-real-time-banking-dbt-data-platform>)

Author: Daniel Poppy

Published: 2026-10-27T18:03:00Z

Content type: news

Language: en

Sources: [dbt Labs](<https://devfeed.tech/sources/dbt-labs.md>)

Topics: [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [stream financial data to Snowflake](<https://devfeed.tech/topics/stream-financial-data-to-snowflake.md>), [Finance](<https://devfeed.tech/topics/finance.md>), [build vs buy database solutions](<https://devfeed.tech/topics/build-vs-buy-database-solutions.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [banking](<https://devfeed.tech/tags/banking.md>), [banking-infrastructure](<https://devfeed.tech/tags/banking-infrastructure.md>), [data](<https://devfeed.tech/tags/data.md>), [data-team](<https://devfeed.tech/tags/data-team.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [keep](<https://devfeed.tech/tags/keep.md>), [learn](<https://devfeed.tech/tags/learn.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>)

### AI overview

Keep's one-person data team replaced spreadsheet-based underwriting that took a business day with Snowflake-triggered processing that completes in under a minute after bank data arrives. Artie copies Postgres and MongoDB changes into Snowflake at sub-second latency; Snowflake streams and tasks flatten incoming records, and dbt governs models, including transaction deduplication and risk features. The team chose this stack to avoid maintaining custom streaming infrastructure, accepting reduced observability in the real-time components. Custom freshness tests help catch failures, though some can be silent. The under-one-minute figure excludes Flinks' bank connection, which can take up to 30 minutes (under five minutes 95% of the time). Each underwriting result is stored with its producing code's git hash, supporting traceability and portfolio-wide reprocessing.

### Source excerpt

Keep's one-person data team used dbt, Artie, and Snowflake to cut credit underwriting from one business day to under a minute.

## How Writer Keeps Its AI Infrastructure Running with Snowflake Hybrid Tables

DevFeed: [How Writer Keeps Its AI Infrastructure Running with Snowflake Hybrid Tables](<https://devfeed.tech/articles/how-writer-keeps-its-ai-infrastructure-running-with-snowflake-hybrid-tables-101920.md>)

Original publisher: [Read original article](<https://medium.com/snowflake/how-writer-keeps-its-ai-infrastructure-running-with-snowflake-hybrid-tables-80f55dd652fa?source=rss----34b6daafc07---4>)

Author: Aeril Xie

Published: 2026-10-26T14:01:05Z

Content type: article

Language: en

Sources: [Snowflake Builders](<https://devfeed.tech/sources/snowflake-builders.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [database-optimization](<https://devfeed.tech/topics/database-optimization.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-infrastructure](<https://devfeed.tech/tags/data-infrastructure.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [enterprise-ai](<https://devfeed.tech/tags/enterprise-ai.md>), [etl](<https://devfeed.tech/tags/etl.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [observability](<https://devfeed.tech/tags/observability.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [platform](<https://devfeed.tech/tags/platform.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>)

### AI overview

Writer moved its high-volume LLM trace writes from a standard Snowflake table to a Hybrid Table after lock contention caused P50 insert latency of 2,362 ms and failures in about 45% of insertions. The migration required creating a Hybrid Table with a schema and primary key, updating the API destination, and backfilling historical data. Hybrid Tables reduced P50 latency to 154 ms and failures to near zero (typically 3-10 per day); Writer also reduced its ingest warehouse from medium to extra small. As the table grew beyond 33 million rows, analytics and reconciliation queries slowed because Hybrid Tables use row-oriented storage optimized for primary-key access. Writer now keeps writes on the Hybrid Table and uses a Snowflake Task to incrementally copy rows into a timestamp-clustered standard table for analytics. This retains in-Snowflake access without external infrastructure, while using each table type for the workload it handles best.

### Source excerpt

Aeril Xie, Data Engineer at Writer At Writer, data is at the center of everything we do. As an enterprise AI platform, we route every customer request through LLM inference providers. Every one of those calls generates a trace record: the model called, the prompt, the response, latency, token counts, and outcome. Rather than running a separate observability database alongside our data warehouse, we made Snowflake the destination for raw trace writes too, so our engineers and analysts can query everything in one place. My job as a data engineer is to make sure that data lands reliably, because it powers our QA processes and gives us the visibility we need into how our models are behaving in production. At the scale we operate, "reliably" is harder than it sounds. The Problem: Lock Contention at Scale Every Writer customer interaction that touches an LLM generates a row. At enterprise scale, that's not a batch job you can schedule. It's a continuous stream that doesn't pause. Our LLM tracing pipeline writes to Snowflake at high velocity: up to 600 rows per minute at peak, and roughly 1.3 million rows per day for our primary traces table. For a while, we were directing this traffic to a standard Snowflake table. And for a while, it mostly worked. Then it didn't. We started seeing write failures from lock contention -- too many concurrent writes competing for the same table resource. Standard Snowflake tables aren't designed for that kind of continuous, high-concurrency write pattern. At our peak traffic levels, INSERT latency at P50 was reaching 2,362 milliseconds. Failure rates climbed to 45% of all insertions. Every failed write was a lost data point. Initially the data wasn't business-critical, so the failures were tolerable. But as this tracing data became the foundation of our observability pipeline and QA processes, the gap it left in our data got harder to ignore. Evaluating the Options When we opened a support ticket with Snowflake to investigate the failures, o

## Introducing PyDebeziumAI: Real-Time CDC for LangChain and LangGraph

DevFeed: [Introducing PyDebeziumAI: Real-Time CDC for LangChain and LangGraph](<https://devfeed.tech/articles/introducing-pydebeziumai-real-time-cdc-for-langchain-and-langgraph-89911.md>)

Original publisher: [Read original article](<https://debezium.io/blog/2026/10/19/introducing-pydebeziumai/>)

Author: Kodukulla Mohnish Mythreya

Published: 2026-10-19T00:00:00Z

Content type: release

Language: en

Sources: [Debezium](<https://devfeed.tech/sources/debezium-io.md>)

Topics: [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [Debezium Connector](<https://devfeed.tech/topics/debezium-connector.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [LangChain](<https://devfeed.tech/topics/langchain.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [Langgraph](<https://devfeed.tech/topics/langgraph.md>), [LanceDB vector database](<https://devfeed.tech/topics/lancedb-vector-database.md>), [software development triangle](<https://devfeed.tech/topics/software-development-triangle.md>)

Tags: [batch](<https://devfeed.tech/tags/batch.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [cron](<https://devfeed.tech/tags/cron.md>), [debezium](<https://devfeed.tech/tags/debezium.md>), [llm](<https://devfeed.tech/tags/llm.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [rag](<https://devfeed.tech/tags/rag.md>), [retrieval-augmented-generation-rag](<https://devfeed.tech/tags/retrieval-augmented-generation-rag.md>)

### AI overview

PyDebeziumAI is an open-source Python library that uses Debezium change data capture to keep vector stores synchronized with database changes for LangChain and LangGraph applications. It embeds Debezium in the Python process, compacts batched changes, and supports vector store adapters and retriever integrations. The article also describes a MySQL-to-Milvus example and reports benchmark latency in the low hundreds of milliseconds.

### Source excerpt

Retrieval-Augmented Generation (RAG) has become a central pattern in AI application development. The idea is simple: rather than relying solely on an LLM's pre-trained knowledge, you give it fresh context by retrieving relevant documents from a vector store at query time. But there is a catch that most tutorials skip over entirely. Where does that vector data come from, and how do you keep it current? Today, most teams answer that question with a cron job, a nightly batch reload, or a manual re-embedding script. That approach works until the database changes faster than the schedule runs, and your AI starts answering questions based on data that no longer exists or doesn't reflect a new record committed five minutes ago. This is the problem that PyDebeziumAI solves. It is an open-source Python library, developed as part of Google Summer of Code 2026 under the Debezium project, that wires Debezium's Change Data Capture engine directly into LangChain and LangGraph pipelines, ensuring your vector store updates the moment the database does. Read more...

## Passing Parameters Between Snowflake Notebooks: A Complete Guide

DevFeed: [Passing Parameters Between Snowflake Notebooks: A Complete Guide](<https://devfeed.tech/articles/passing-parameters-between-snowflake-notebooks-a-complete-guide-88060.md>)

Original publisher: [Read original article](<https://medium.com/snowflake/passing-parameters-between-snowflake-notebooks-a-complete-guide-b95cb5249338?source=rss----34b6daafc07---4>)

Author: Swati Goel

Published: 2026-10-15T19:01:01Z

Content type: tutorial

Language: en

Sources: [Snowflake Builders](<https://devfeed.tech/sources/snowflake-builders.md>)

Topics: [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [etl](<https://devfeed.tech/topics/etl.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [jupyter-notebook](<https://devfeed.tech/tags/jupyter-notebook.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [python](<https://devfeed.tech/tags/python.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>)

### AI overview

This Snowflake tutorial compares ways to pass parameters between notebooks, including shared in-memory variables with %run and return values passed through Task DAGs. It also covers static task configuration, multiple predecessors, and operational pitfalls such as malformed JSON arguments and deploying notebook changes to update a Code Bundle.

### Source excerpt

How to build production notebook pipelines with parameter passing across every workspace type and execution mode. Full working code for all scenarios is available in the GitHub repo. This guide covers every scenario with working examples you can copy into your own account. Understanding the Landscape:Workspace TypesExecution Modes This distinction matters for every pattern below. Terminology: This guide uses Code Bundle (formerly Notebook Project Object/NPO). Snowflake renamed the object; the legacy NOTEBOOK PROJECT SQL grammar is still supported. See Snowflake Code Bundles.Scenario 1: %run -- Shared Variables in Memory Best for: Sequential notebooks in the same workspace. The simplest approach. How it works: The %run executes another notebook in the same Python process. All variables, functions, and imports are shared. Architecture The sender sets variables in its last cell: OUTPUT_TABLE = f'{TARGET_DB}.{TARGET_SCHEMA}.ORDER_SUMMARY' ROW_COUNT = session.sql(f'SELECT COUNT(*) AS C FROM {OUTPUT_TABLE}').collect()[0]['C'] STATUS = 'SUCCESS' The orchestrator calls both and can gate between them: %run data_prep.ipynb assert STATUS == 'SUCCESS', f"Data prep failed: {STATUS}" %run report.ipynb The receiver reads the variables directly -- no imports, no parsing: print(f"Reading from: {OUTPUT_TABLE}") # Already in memory report_df = session.sql(f"SELECT * FROM {OUTPUT_TABLE}") Full notebook code: GitHub repo / scenario-1-run Scenario 2: Task DAG with SYSTEM$SET_RETURN_VALUE Best for: Independent notebooks running as separate production units. Each notebook is its own Code Bundle. A Task DAG chains them. The first task sets a return value, the second reads it and passes it as ARGUMENTS. The sender sets a pipe-delimited return value (not JSON -- see Gotcha #4): return_payload = f"{OUTPUT_TABLE}|{ROW_COUNT}|{RUN_DATE}|{STATUS}" try: session.sql(f"SELECT SYSTEM$SET_RETURN_VALUE('{return_payload}')").collect() except Exception as e: if 'must be called from within a task' in str(e):

## The Easiest Way to Replicate Postgres Data to Snowflake

DevFeed: [The Easiest Way to Replicate Postgres Data to Snowflake](<https://devfeed.tech/articles/the-easiest-way-to-replicate-postgres-data-to-snowflake-87879.md>)

Original publisher: [Read original article](<https://hackernoon.com/the-easiest-way-to-replicate-postgres-data-to-snowflake?source=rss>)

Author: Pipelineguy

Published: 2026-10-15T09:01:11Z

Content type: tutorial

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [Heroku Postgres](<https://devfeed.tech/topics/heroku-postgres.md>), [CRUD](<https://devfeed.tech/topics/crud.md>)

Tags: [cdc](<https://devfeed.tech/tags/cdc.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-replication](<https://devfeed.tech/tags/data-replication.md>), [estuary](<https://devfeed.tech/tags/estuary.md>), [good-company](<https://devfeed.tech/tags/good-company.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>)

### AI overview

This tutorial shows how to build a PostgreSQL-to-Snowflake change data capture pipeline with Estuary. It covers enabling PostgreSQL logical replication, capturing changes into Estuary collections, configuring Snowflake materialization, and verifying that inserts, updates, and deletes reach the destination. It also discusses production considerations such as replication lag, WAL retention, backfills, schema changes, monitoring, and synchronization frequency.

### Source excerpt

Learn how to replicate PostgreSQL to Snowflake using Estuary and CDC. Build a step-by-step pipeline that captures inserts, updates, and deletes.

## "Why Did We Miss Target?" -- Ask Snowflake CoWork, Get the Root-Cause PDF

DevFeed: ["Why Did We Miss Target?" -- Ask Snowflake CoWork, Get the Root-Cause PDF](<https://devfeed.tech/articles/why-did-we-miss-target-ask-snowflake-cowork-get-the-root-cause-pdf-88062.md>)

Original publisher: [Read original article](<https://medium.com/snowflake/why-did-we-miss-target-ask-snowflake-cowork-get-the-root-cause-pdf-1b937dbbc2f7?source=rss----34b6daafc07---4>)

Author: Christopher Vimal Selvadurai

Published: 2026-10-14T19:01:02Z

Content type: tutorial

Language: en

Sources: [Snowflake Builders](<https://devfeed.tech/sources/snowflake-builders.md>)

Topics: [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [atlassian automated rca](<https://devfeed.tech/topics/atlassian-automated-rca.md>)

Tags: [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [cowork](<https://devfeed.tech/tags/cowork.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [ddl](<https://devfeed.tech/tags/ddl.md>), [docker](<https://devfeed.tech/tags/docker.md>), [powerpoint](<https://devfeed.tech/tags/powerpoint.md>), [python](<https://devfeed.tech/tags/python.md>), [rca](<https://devfeed.tech/tags/rca.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [snowflake-cortex-agent](<https://devfeed.tech/tags/snowflake-cortex-agent.md>)

### AI overview

This tutorial describes a Snowflake stored procedure that analyzes missed manufacturing production targets, renders a self-contained HTML root-cause report, and converts it to PDF using WeasyPrint from Snowflake's PyPI Artifact Repository. It explains the report calculations, the limitations encountered with other PDF libraries, and how a Snowflake Cortex Agent can return a download link for the report.

### Source excerpt

Analysts used to pull data from several systems, build a PowerPoint and email versions of it. Now one question returns a production-grade RCA PDF, all inside Snowflake, with no SPCS and no Docker. Every shift change in a manufacturing plant starts with the same questions: why did we miss target, what actually caused it, and can we recover it next shift? A 24/7 line short of its daily production target needs more than an "equipment issues" shrug -- it needs a quantified variance bridge: top causes, impact in units, what's recoverable, what to action first. Today, that takes an analyst and one or two reviewers 3-6 hours, pulling from four source systems and assembling a PowerPoint that usually goes through a revision or two before it ships around noon. By then, the next shift has already started blind. We built something different: one stored procedure call that queries the data, computes the root-cause analysis, renders a styled HTML report, converts it to a polished PDF, and hands back a download link -- entirely inside Snowflake. Now, the shift supervisor or plant manager simply asks in CoWork (Snowflake's personal agent for knowledge workers, formerly Snowflake Intelligence) and get a finished PDF back in 5 minutes, in time to brief the incoming shift before it starts, not after. That frees the analyst from manual report-building to focus on higher-value work: strengthening the underlying data and driving adoption of the system across the plant. Here's how we built it, the dead ends we hit along the way, and the surprisingly simple fix that made it work. GitHub repo: The complete code -- engine, sample data generator, and DDL -- is available at repo. Clone it, point it at your own views, and adapt the reason-bucketing to your domain. Why We Started With a Spec, Not Code Before writing a line of code, we defined what the system must do using structured requirements. This isn't ceremony for its own sake -- it's about making technology choices testable instead of hoping t

## How UTS established a modern curriculum data services foundation on AWS

DevFeed: [How UTS established a modern curriculum data services foundation on AWS](<https://devfeed.tech/articles/how-uts-established-a-modern-curriculum-data-services-foundation-on-aws-87252.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/how-uts-established-a-modern-curriculum-data-services-foundation-on-aws/>)

Author: Oleg Polianski

Published: 2026-10-14T14:29:10Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [AWS Amplify](<https://devfeed.tech/topics/aws-amplify.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [Amazon OpenSearch Service](<https://devfeed.tech/topics/amazon-opensearch-service.md>), [Redis](<https://devfeed.tech/topics/redis.md>), [Authorization](<https://devfeed.tech/topics/authorization.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>)

Tags: [amazon-api-gateway](<https://devfeed.tech/tags/amazon-api-gateway.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-eventbridge](<https://devfeed.tech/tags/amazon-eventbridge.md>), [amazon-msk](<https://devfeed.tech/tags/amazon-msk.md>), [amazon-neptune](<https://devfeed.tech/tags/amazon-neptune.md>), [amazon-opensearch-service](<https://devfeed.tech/tags/amazon-opensearch-service.md>), [amazon-route-53](<https://devfeed.tech/tags/amazon-route-53.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-appsync](<https://devfeed.tech/tags/aws-appsync.md>), [aws-certificate-manager](<https://devfeed.tech/tags/aws-certificate-manager.md>), [aws-identity-and-access-management-iam](<https://devfeed.tech/tags/aws-identity-and-access-management-iam.md>), [aws-key-management-service](<https://devfeed.tech/tags/aws-key-management-service.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [aws-secrets-manager](<https://devfeed.tech/tags/aws-secrets-manager.md>), [aws-waf](<https://devfeed.tech/tags/aws-waf.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data](<https://devfeed.tech/tags/data.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [semantic-search](<https://devfeed.tech/tags/semantic-search.md>)

### AI overview

The University of Technology Sydney built an AWS-based curriculum data service that keeps its Curriculum Management System as the system of record while exposing governed APIs, graph traversal, semantic search, and event delivery. Since production launch, the service has recorded zero incidents, averages 1.6-second delivery of changes, and is operated by one engineer.

### Source excerpt

The University of Technology Sydney built a modern curriculum data services foundation on AWS that treats curriculum data as a governed, reusable product, available through APIs, graph traversal, semantic search, and event delivery, while leaving the source Curriculum Management System authoritative. One engineer now operates the domain.

## End-to-end Snowflake Observability is now generally available in Dynatrace

DevFeed: [End-to-end Snowflake Observability is now generally available in Dynatrace](<https://devfeed.tech/articles/end-to-end-snowflake-observability-is-now-generally-available-in-dynatrace-85469.md>)

Original publisher: [Read original article](<https://www.dynatrace.com/news/blog/end-to-end-snowflake-observability-is-now-generally-available-in-dynatrace/>)

Author: Michael Schachner

Published: 2026-10-12T13:30:36Z

Content type: release

Language: en

Sources: [Dynatrace news](<https://devfeed.tech/sources/dynatrace-news.md>)

Topics: [observability](<https://devfeed.tech/topics/observability.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [dynatrace](<https://devfeed.tech/topics/dynatrace.md>)

Tags: [dashboards](<https://devfeed.tech/tags/dashboards.md>), [data-observability](<https://devfeed.tech/tags/data-observability.md>), [data-quality-issues](<https://devfeed.tech/tags/data-quality-issues.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [dynatrace](<https://devfeed.tech/tags/dynatrace.md>), [observability](<https://devfeed.tech/tags/observability.md>), [performance](<https://devfeed.tech/tags/performance.md>), [product-news](<https://devfeed.tech/tags/product-news.md>), [security](<https://devfeed.tech/tags/security.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [snowflake-observability](<https://devfeed.tech/tags/snowflake-observability.md>)

### AI overview

Dynatrace has made Snowflake Observability generally available through its Data Observability app. It provides dashboards and guided onboarding for monitoring costs, query performance, security telemetry, operations, and data quality. The release includes automated telemetry collection and a choice of automated or manual deployment.

### Source excerpt

Dynatrace Snowflake Observability is now generally available, giving teams unified visibility into cost, performance, security, operations, and data quality through the Dynatrace Data Observability app. Guided onboarding, automated telemetry collection, and 14 ready-made dashboards help teams understand the health of their Snowflake environments from day one, with no blank-canvas setup required. As enterprises expand their [...] The post End-to-end Snowflake Observability is now generally available in Dynatrace appeared first on Dynatrace news.

## How to Migrate a Production CDC Pipeline With Minimal Downtime and No Data Gaps

DevFeed: [How to Migrate a Production CDC Pipeline With Minimal Downtime and No Data Gaps](<https://devfeed.tech/articles/how-to-migrate-a-production-cdc-pipeline-with-minimal-downtime-and-no-data-gaps-87144.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/migrate-production-cdc-pipeline>)

Author: Emily Lucek

Published: 2026-10-12T07:42:13Z

Content type: tutorial

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [data migrations](<https://devfeed.tech/topics/data-migrations.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [Redpanda](<https://devfeed.tech/topics/redpanda-data-streaming.md>)

Tags: [cdc](<https://devfeed.tech/tags/cdc.md>), [data](<https://devfeed.tech/tags/data.md>), [downtime](<https://devfeed.tech/tags/downtime.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [migration](<https://devfeed.tech/tags/migration.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [risks](<https://devfeed.tech/tags/risks.md>)

### AI overview

This guide explains how to migrate a production change data capture (CDC) pipeline while limiting data gaps and consumer disruption. It recommends starting the new pipeline with its own backfill and replication position, running both pipelines in parallel into separate tables, validating data including deletes at a shared boundary, and switching consumers in groups. It also covers source log retention, rollback planning, monitoring, and differences among CDC tools and databases.

### Source excerpt

Migrating a production CDC pipeline requires careful planning to avoid missing changes, duplicate records, or disrupting downstream systems. This guide covers the key steps, risks, and strategies for a safe migration with minimal downtime.

## Mainframe Data Meets Snowflake Datastream: TDT Goes Snowflake Native

DevFeed: [Mainframe Data Meets Snowflake Datastream: TDT Goes Snowflake Native](<https://devfeed.tech/articles/mainframe-data-meets-snowflake-datastream-tdt-goes-snowflake-native-78488.md>)

Original publisher: [Read original article](<https://medium.com/snowflake/mainframe-data-meets-snowflake-datastream-tdt-goes-snowflake-native-d68259fa00ad?source=rss----34b6daafc07---4>)

Author: Joseph Brady

Published: 2026-10-07T14:01:05Z

Content type: article

Language: en

Sources: [Snowflake Builders](<https://devfeed.tech/sources/snowflake-builders.md>)

Topics: [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Mainframe](<https://devfeed.tech/topics/mainframe.md>), [rbac](<https://devfeed.tech/topics/rbac.md>)

Tags: [access-control](<https://devfeed.tech/tags/access-control.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [app](<https://devfeed.tech/tags/app.md>), [architect](<https://devfeed.tech/tags/architect.md>), [authentication](<https://devfeed.tech/tags/authentication.md>), [based](<https://devfeed.tech/tags/based.md>), [business](<https://devfeed.tech/tags/business.md>), [can](<https://devfeed.tech/tags/can.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [complete](<https://devfeed.tech/tags/complete.md>), [control](<https://devfeed.tech/tags/control.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-migration](<https://devfeed.tech/tags/data-migration.md>), [mainframe](<https://devfeed.tech/tags/mainframe.md>), [mainframe-data](<https://devfeed.tech/tags/mainframe-data.md>), [mainframe-modernization](<https://devfeed.tech/tags/mainframe-modernization.md>), [native](<https://devfeed.tech/tags/native.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>)

### AI overview

Treehouse's TDT Snowflake Native App moves mainframe data into Snowflake using Snowflake Datastream as its Kafka-compatible streaming layer. Treehouse says its mainframe component, Rocket's RDRS, captures source changes and publishes them to Datastream; TDT then inspects the incoming structure and provisions Snowflake schemas, delta tables, views, and monitoring resources. It supports bulk loads and ongoing change data capture for sources including Db2, VSAM, Adabas, IMS, IDMS, Datacom, and flat files. TDT stores hierarchical data as JSON and retains change history in delta tables; generated JSON views preserve hierarchy, while relational views provide a current SQL-style snapshot. The article says Datastream is in private preview and describes its streams landing in Snowflake or Iceberg tables, with Snowflake governance controls applied there. TDT is presented as a Snowflake Native App available wherever Snowflake runs, with its underlying technology reported in production for four years at a large auto manufacturer.

### Source excerpt

by Joseph Brady, Director of Business Development at Treehouse Software, Dan Vimont Director of Innovation at Treehouse Software, and Eleonora Savova, Cloud Solutions Architect at Treehouse Software Snowflake is redefining how data streams into its platform with the introduction of Snowflake Datastream. For Treehouse Software, this has opened the door for something bigger: a mainframe-to-Snowflake data delivery solution built to run natively inside the Snowflake framework itself! Leveraging Datastream, Treehouse refactored its existing offering, Treehouse Dataflow Toolkit (TDT), to run inside Snowflake, using Datastream as its Kafka layer. Welcome the TDT Snowflake Native App! The technology behind TDT itself is not new. TDT has been proven in production for years, delivering mainframe data into Snowflake and other modern analytics platforms for large enterprise customers. What's changed is where it can run: the TDT Snowflake Native App now operates entirely inside Snowflake. Introducing Snowflake Datastream While many organizations rely on Kafka for their streaming backbone, running it apart from your analytics platform creates friction: duplicate costs, split governance, and delayed data delivery. Snowflake Datastream (private preview) solves this by bringing an Apache Kafka-compatible streaming service natively into Snowflake. This eliminates the overhead of managing two separate systems. Data streams continuously into native Snowflake or open Apache Iceberg™ tables and becomes queryable in seconds. Because it lives within Snowflake, data is instantly secured by role-based access control (RBAC) and the Horizon Catalog, inheriting classification, lineage, and masking policies the moment it lands. Setup is just as streamlined: simply define your pipeline, and CoCo manages Datastream authentication and team onboarding in minutes, requiring zero specialized Kafka expertise. Beyond Datastream: A Complete Data Delivery Solution With the TDT Snowflake Native App, enterp

## Converting Snowflake Native Tables to Iceberg -- Part 1

DevFeed: [Converting Snowflake Native Tables to Iceberg -- Part 1](<https://devfeed.tech/articles/converting-snowflake-native-tables-to-iceberg-part-1-78486.md>)

Original publisher: [Read original article](<https://medium.com/snowflake/converting-snowflake-native-tables-to-iceberg-part-1-603fdb0fdcd1?source=rss----34b6daafc07---4>)

Author: Leo Pasta

Published: 2026-10-02T14:01:08Z

Content type: article

Language: en

Sources: [Snowflake Builders](<https://devfeed.tech/sources/snowflake-builders.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>)

Tags: [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [article](<https://devfeed.tech/tags/article.md>), [conversion](<https://devfeed.tech/tags/conversion.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-migration](<https://devfeed.tech/tags/data-migration.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [ecosystem](<https://devfeed.tech/tags/ecosystem.md>), [etl](<https://devfeed.tech/tags/etl.md>), [feature](<https://devfeed.tech/tags/feature.md>), [format](<https://devfeed.tech/tags/format.md>), [governance](<https://devfeed.tech/tags/governance.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [latency](<https://devfeed.tech/tags/latency.md>), [native](<https://devfeed.tech/tags/native.md>), [performance](<https://devfeed.tech/tags/performance.md>), [platform](<https://devfeed.tech/tags/platform.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [tables](<https://devfeed.tech/tags/tables.md>)

### AI overview

Iceberg conversion is most useful when specific datasets need access from multiple engines; converting every table can add complexity and cost without equivalent benefit. Start with a targeted, phased migration and inventory each table's consumers, engine versions, data types, write patterns, streams, and swap-based pipelines before choosing settings. Consumer capabilities constrain Iceberg version, types, partition pruning, codecs, and delete behavior. Resolve unsupported or lossy types such as TIMESTAMP_TZ, VECTOR, and some V2 conversions, and account for limitations involving defaults and collation for external engines. Choose CTAS for tables around 1-2 TB or with types that make Parquet staging risky; for larger compatible tables, Parquet export with COPY INTO ... LOAD_MODE=ADD_FILES_COPY may be faster and use fewer credits. Select partitioning or clustering based on consumer pruning and update workload, and choose storage and serialization settings according to cloud, access, key-management, and operational needs. Validate a representative sample before conversion; execution, validation, and cutover are reserved for Part 2.

### Source excerpt

Part 1 of 2 -- this article covers why you might convert to Iceberg, the decisions you need to make and how to approach them. Part 2 covers the conversion execution, validation and cutover. Why convert to Iceberg? Snowflake native tables are tightly integrated into the Snowflake platform: They offer the most complete support in areas such as performance optimisation, disaster recovery, replication and the full range of governance features. For workloads that live entirely within Snowflake, native tables are usually the right choice. But enterprise data environments are growing more and more sophisticated, so adopting an open table format like Iceberg can solve real challenges, like having multiple engines -- Apache Spark, Trino, Amazon Athena, Databricks, DuckDB and many others -- able to read and, in some configurations, write the same Iceberg tables without relying on Snowflake to process the query or on ETL jobs duplicating the data with the complexities and latency challenges it brings. Snowflake responded to this trend by offering what is widely regarded as the most complete support for the Apache Iceberg ecosystem, both as a manager and consumer of Iceberg catalogs. Conversion is not free of trade-offs though. The Iceberg ecosystem is also more complex to manage, with more options and configurations to consider and often more vendors and technologies involved. It is also less mature and well known than "traditional" Snowflake tables, which usually means a longer learning and adoption curve for engineers. The Apache Iceberg project is managed by a committee which, while fundamental for the extensive vendor support it has attracted, also means the adoption of new features and capabilities might happen later than with native tables. Over time, as the Iceberg format evolves and Snowflake features are released, the feature gap between native tables and Iceberg tables continues to narrow, but it is not zero yet. A few of those limitations will be discussed in the rest

## Cloudflare moves into analytics workloads with a serverless alternative to dedicated data infrastructure

DevFeed: [Cloudflare moves into analytics workloads with a serverless alternative to dedicated data infrastructure](<https://devfeed.tech/articles/cloudflare-moves-into-analytics-workloads-with-a-serverless-alternative-to-dedicated-data-infrastructure-63241.md>)

Original publisher: [Read original article](<https://siliconangle.com/2026/10/01/cloudflare-moves-into-analytics-workloads-with-a-serverless-alternative-that-doesnt-require-dedicated-servers-or-data-movement/>)

Author: Mike Wheatley

Published: 2026-10-01T13:00:08Z

Content type: news

Language: en

Sources: [SiliconANGLE](<https://devfeed.tech/sources/siliconangle.md>)

Topics: [Data Infrastructure](<https://devfeed.tech/topics/data-infrastructure.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [databricks](<https://devfeed.tech/topics/databricks.md>)

Tags: [analytics-infrastructure](<https://devfeed.tech/tags/analytics-infrastructure.md>), [analytics-workloads](<https://devfeed.tech/tags/analytics-workloads.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-table-format](<https://devfeed.tech/tags/apache-iceberg-table-format.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [cloudflare](<https://devfeed.tech/tags/cloudflare.md>), [cloudflare-basin](<https://devfeed.tech/tags/cloudflare-basin.md>), [dane-knecht](<https://devfeed.tech/tags/dane-knecht.md>), [data-analytics](<https://devfeed.tech/tags/data-analytics.md>), [data-teams](<https://devfeed.tech/tags/data-teams.md>), [dedicated-servers](<https://devfeed.tech/tags/dedicated-servers.md>), [global-network](<https://devfeed.tech/tags/global-network.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [news](<https://devfeed.tech/tags/news.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [serverless-analytics](<https://devfeed.tech/tags/serverless-analytics.md>), [the-latest](<https://devfeed.tech/tags/the-latest.md>)

### AI overview

Cloudflare announced Basin, a serverless data analytics platform built on Apache Iceberg and Cloudflare R2. The article says it aims to help small businesses and developer teams ingest, store, catalog and query data without managing dedicated servers or moving data between systems. Basin runs data work on Cloudflare's global network and is positioned for smaller or edge analytics workloads, rather than as a replacement for every data warehouse.

### Source excerpt

Cloudflare Inc. is moving beyond its bread-and-butter content delivery networks and cybersecurity offerings with the launch of a new serverless data platform called Cloudflare Basin, which aims to make advanced data analytics cheaper and easier for small businesses and developer teams. Announced today, Cloudflare Basin is designed to eliminate the need for companies to set [...] The post Cloudflare moves into analytics workloads with a serverless alternative to dedicated data infrastructure appeared first on SiliconANGLE.

## Everpure Brings Carrier-Grade Storage to Red Hat's Telco Cloud Reference Architecture

DevFeed: [Everpure Brings Carrier-Grade Storage to Red Hat's Telco Cloud Reference Architecture](<https://devfeed.tech/articles/everpure-brings-carrier-grade-storage-to-red-hat-s-telco-cloud-reference-architecture-78493.md>)

Original publisher: [Read original article](<https://blog.everpuredata.com/news-events/everpure-brings-carrier-grade-storage-to-red-hat/>)

Author: Sean Schneyer

Published: 2026-09-24T13:00:00Z

Content type: news

Language: en

Sources: [Everpure](<https://devfeed.tech/sources/everpure.md>)

Topics: [Cloud Architecture](<https://devfeed.tech/topics/cloud-architecture.md>), [redhat](<https://devfeed.tech/topics/redhat.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [rhel](<https://devfeed.tech/topics/rhel.md>), [scalable-architecture](<https://devfeed.tech/topics/scalable-architecture.md>)

Tags: [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-native](<https://devfeed.tech/tags/cloud-native.md>), [cloud-native-operations](<https://devfeed.tech/tags/cloud-native-operations.md>), [everpure](<https://devfeed.tech/tags/everpure.md>), [news-events](<https://devfeed.tech/tags/news-events.md>), [red-hat](<https://devfeed.tech/tags/red-hat.md>), [storage](<https://devfeed.tech/tags/storage.md>), [telco](<https://devfeed.tech/tags/telco.md>)

### AI overview

Everpure announces that its Portworx CSI and FlashArray storage provide the data layer in Red Hat's telco network cloud reference architecture. The article describes how this setup supports cloud-native network functions across core, edge, and hybrid environments, with independently scalable storage and compute and non-disruptive upgrades.

### Source excerpt

Everpure Brings Carrier-Grade Storage to Red Hat's Telco Cloud Reference Architecture by Everpure Blog Learn how Everpure provides a validated data layer for the Red Hat telco network cloud, helping service providers streamline cloud-native operations. The post Everpure Brings Carrier-Grade Storage to Red Hat's Telco Cloud Reference Architecture appeared first on Everpure Blog.

## The post says data preparation can consume up to 80% of enterprise AI project timelines

DevFeed: [The post says data preparation can consume up to 80% of enterprise AI project timelines](<https://devfeed.tech/articles/the-80-tax-why-your-ai-projects-spend-most-of-their-time-not-doing-ai-78506.md>)

Original publisher: [Read original article](<https://blog.everpuredata.com/products/the-80-percent-tax-ai-projects/>)

Author: Austin Boone

Published: 2026-09-22T14:45:20Z

Content type: article

Language: en

Sources: [Everpure](<https://devfeed.tech/sources/everpure.md>)

Topics: [Data Management](<https://devfeed.tech/topics/data-management.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [Managed Services](<https://devfeed.tech/topics/managed-services.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-and-machine-learning](<https://devfeed.tech/tags/ai-and-machine-learning.md>), [ai-data-platform](<https://devfeed.tech/tags/ai-data-platform.md>), [apis](<https://devfeed.tech/tags/apis.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [business](<https://devfeed.tech/tags/business.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [solutions](<https://devfeed.tech/tags/solutions.md>), [the-everpure-platform](<https://devfeed.tech/tags/the-everpure-platform.md>)

### AI overview

The post argues that enterprise AI projects spend much of their time preparing data: ingesting, cleaning, curating, tagging, and converting it before models can use it. It describes fragmented pipelines as sources of delays, ownership gaps, and access-control risks, then presents Everpure Data Stream as a continuous product for those pipeline tasks. It also says storage throughput and concurrency affect whether GPUs wait on data.

### Source excerpt

The 80% Tax: Why Your AI Projects Spend Most of Their Time Not Doing AI by Everpure Blog Up to 80% of an AI project timeline can be spent on data preparation. Learn how a continuous, governed pipeline can help move AI into production faster. The post The 80% Tax: Why Your AI Projects Spend Most of Their Time Not Doing AI appeared first on Everpure Blog.

## Celebrating the 2026 dbt partner of the year winners

DevFeed: [Celebrating the 2026 dbt partner of the year winners](<https://devfeed.tech/articles/celebrating-the-2026-dbt-partner-of-the-year-winners-80128.md>)

Original publisher: [Read original article](<https://www.getdbt.com/blog/2026-partner-of-the-year-winners>)

Author: Yahsmene Butler

Published: 2026-09-15T21:00:00Z

Content type: news

Language: en

Sources: [dbt Labs](<https://devfeed.tech/sources/dbt-labs.md>)

Topics: [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [elementary-data compromise](<https://devfeed.tech/topics/elementary-data-compromise.md>), [Shopify](<https://devfeed.tech/topics/shopify.md>)

Tags: [dbt](<https://devfeed.tech/tags/dbt.md>), [future](<https://devfeed.tech/tags/future.md>), [has](<https://devfeed.tech/tags/has.md>), [labs](<https://devfeed.tech/tags/labs.md>), [partnerships](<https://devfeed.tech/tags/partnerships.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>)

### AI overview

dbt Labs announced its 2026 Partner of the Year winners: phData, Snowflake, Cívica, Datum Studio, and 66degrees. The article describes the partners' data engineering and analytics delivery work, including dbt implementations, data pipelines, and foundations for AI use cases.

### Source excerpt

Meet the 2026 dbt Labs Partner of the Year winners: phData, Snowflake, Cívica, Datum Studio, and 66degrees.

## How me&u adopted Omni for self-service analytics and Claude-assisted data queries

DevFeed: [How me&u adopted Omni for self-service analytics and Claude-assisted data queries](<https://devfeed.tech/articles/how-me-u-built-an-ai-foundation-with-snowflake-omni-and-claude-82502.md>)

Original publisher: [Read original article](<https://omni.co/blog/case-study-meandu>)

Published: 2026-09-08T16:00:00Z

Content type: article

Language: en

Sources: [Omni Blog](<https://devfeed.tech/sources/omni-blog.md>)

Topics: [MSP MCP](<https://devfeed.tech/topics/msp-mcp.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [anthropic](<https://devfeed.tech/topics/anthropic.md>), [Claude](<https://devfeed.tech/topics/claude.md>), [Continuous integration](<https://devfeed.tech/topics/continuous-integration.md>), [Beta channel](<https://devfeed.tech/topics/beta-channel.md>)

Tags: [access-controls](<https://devfeed.tech/tags/access-controls.md>), [ai](<https://devfeed.tech/tags/ai.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [bi-tools](<https://devfeed.tech/tags/bi-tools.md>), [claude](<https://devfeed.tech/tags/claude.md>), [data](<https://devfeed.tech/tags/data.md>), [data-infrastructure](<https://devfeed.tech/tags/data-infrastructure.md>), [data-modeling](<https://devfeed.tech/tags/data-modeling.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [migration](<https://devfeed.tech/tags/migration.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>)

### AI overview

After its 2023 merger with Mr Yum, me&u consolidated analytics on Omni, using a shared governed model connected to Snowflake and dbt. Within a year, analytics use roughly doubled, product managers increased self-service report publishing, and AI-driven queries grew more than 30 times as employees used Claude via MCP. The data team shifted time from routine reporting toward models, context, integrations, and governance.

### Source excerpt

Doubling adoption after a two-month BI migration

## A Hands-On Guide to Building Streaming Transformation with Estuary Derivation Agent Skills

DevFeed: [A Hands-On Guide to Building Streaming Transformation with Estuary Derivation Agent Skills](<https://devfeed.tech/articles/a-hands-on-guide-to-building-streaming-transformation-with-estuary-derivation-agent-skills-82103.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/estuary-agent-skills-streaming-data-transformations>)

Author: Ruhee Shrestha

Published: 2026-08-20T13:51:25Z

Content type: tutorial

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Agent Skills](<https://devfeed.tech/topics/agent-skills.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Loop Engineering](<https://devfeed.tech/topics/loop-engineering.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>)

Tags: [agent-skills](<https://devfeed.tech/tags/agent-skills.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-assistant](<https://devfeed.tech/tags/ai-assistant.md>), [ai-assistants](<https://devfeed.tech/tags/ai-assistants.md>), [ai-coding-assistant](<https://devfeed.tech/tags/ai-coding-assistant.md>), [ai-coding-assistants](<https://devfeed.tech/tags/ai-coding-assistants.md>), [ai-development](<https://devfeed.tech/tags/ai-development.md>), [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [build](<https://devfeed.tech/tags/build.md>), [building](<https://devfeed.tech/tags/building.md>), [data](<https://devfeed.tech/tags/data.md>), [estuary](<https://devfeed.tech/tags/estuary.md>), [guide](<https://devfeed.tech/tags/guide.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [published](<https://devfeed.tech/tags/published.md>), [python](<https://devfeed.tech/tags/python.md>), [skills](<https://devfeed.tech/tags/skills.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [streaming-data](<https://devfeed.tech/tags/streaming-data.md>), [using](<https://devfeed.tech/tags/using.md>)

### AI overview

This tutorial demonstrates Estuary Derivation Agent Skills for building streaming data transformations from natural-language prompts. Using a PostgreSQL banking dataset, it walks through filtering, aggregation, stateful metrics, joins, array flattening, rolling windows, and Python-based merchant classification before materializing results in ClickHouse. It also highlights production concerns such as reviewing generated logic and schemas, handling updates in CDC streams, bounding state, and checking multi-shard behavior.

### Source excerpt

Build streaming data transformations with Estuary Agent Skills using natural-language prompts, from filtering and joins to stateful logic, windowing, and Python.

## From Batch Snapshots to Near-Real-Time Data

DevFeed: [From Batch Snapshots to Near-Real-Time Data](<https://devfeed.tech/articles/from-batch-snapshots-to-near-real-time-data-20029.md>)

Original publisher: [Read original article](<https://technology.doximity.com/articles/from-batch-snapshots-to-near-real-time-data>)

Author: Doximity

Published: 2026-08-04T16:29:00Z

Content type: article

Language: en

Sources: [Doximity](<https://devfeed.tech/sources/doximity.md>)

Topics: [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [transaction-log](<https://devfeed.tech/topics/transaction-log.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [application](<https://devfeed.tech/tags/application.md>), [apply](<https://devfeed.tech/tags/apply.md>), [article](<https://devfeed.tech/tags/article.md>), [base](<https://devfeed.tech/tags/base.md>), [batch](<https://devfeed.tech/tags/batch.md>), [bootstrap](<https://devfeed.tech/tags/bootstrap.md>), [but](<https://devfeed.tech/tags/but.md>), [can](<https://devfeed.tech/tags/can.md>), [capture](<https://devfeed.tech/tags/capture.md>), [cascade](<https://devfeed.tech/tags/cascade.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [change](<https://devfeed.tech/tags/change.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [changes](<https://devfeed.tech/tags/changes.md>), [data](<https://devfeed.tech/tags/data.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [real-time-data](<https://devfeed.tech/tags/real-time-data.md>), [snapshots](<https://devfeed.tech/tags/snapshots.md>), [time](<https://devfeed.tech/tags/time.md>)

### AI overview

Doximity added CDC to make application database changes available in minutes while retaining its batch pipeline for transactionally consistent snapshots and recovery. In a 12-day measurement window, 95% of sampled events from active tables reached the queryable intermediate layer within eight minutes of publication to Kafka. The excerpt identifies four design decisions: use trusted batch snapshots to bootstrap CDC, preserve metadata while routing changes across source databases, expose new changes through a base-plus-delta view before warehouse merging, and synthesize cascade deletes that MySQL does not emit as individual binary-log events. The excerpt begins describing point-in-time snapshots using Amazon Aurora, but does not provide the full implementation details or tradeoffs for the four decisions.

### Source excerpt

Change Data Capture (CDC) is often presented as a straightforward pipeline: read a database transaction log, publish each change, and apply those changes to another system. That description is accurate, but it leaves out many of the decisions that determine whether the resulting data can be trusted. At Doximity, we already had a batch pipeline that periodically copied snapshots of application databases into our data warehouse. Those snapshots were reliable, but their freshness was measured in hours. We introduced CDC to make changes available in minutes so downstream transformations and operational analytics would not have to wait for the next batch snapshot. We continued using the batch pipeline for the consistency and recovery guarantees it already provided. Over a 12-day measurement window, 95% of events from a stratified sample of active tables reached the queryable intermediate layer within eight minutes of being published to Kafka. The most interesting parts of the project were not the connections from a source database to Kafka or from Kafka to Snowflake, but four questions we had to answer around them: How could we reuse our existing, transactionally consistent batch snapshots as an on-demand starting point for CDC, without reprocessing every existing row? How could we onboard new tables and absorb schema changes from many source databases across our products without growing operational overhead for each one? How could new changes become queryable without waiting for the warehouse to merge them into place? How could we handle cascading child deletes that MySQL performs but never emits as individual binary-log events? Our answers are the four design decisions in this article: a trusted batch snapshot, metadata-preserving routing, a base-plus-delta view, and synthetic cascade deletes. Together, they turned a stream of row changes into a system we could bootstrap, scale, validate, and recover. The sections that follow explain the tradeoffs and guardrails so rea

## Validating Trust in Data: How dltHub Delivers GxP-Ready, Auditable Ingestion into Snowflake

DevFeed: [Validating Trust in Data: How dltHub Delivers GxP-Ready, Auditable Ingestion into Snowflake](<https://devfeed.tech/articles/validating-trust-in-data-how-dlthub-delivers-gxp-ready-auditable-ingestion-into-snowflake-80428.md>)

Original publisher: [Read original article](<https://dlthub.com/blog/validating-trust-in-data-how-dlthub-delivers-gxp-ready-auditable-ingestion-into-snowflake>)

Author: Matthaus Krzykowski

Published: 2026-08-04T00:00:00Z

Content type: article

Language: en

Sources: [DLT Hub](<https://devfeed.tech/sources/dlt-hub.md>)

Topics: [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [ai security](<https://devfeed.tech/topics/ai-security.md>), [etl](<https://devfeed.tech/topics/etl.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ci](<https://devfeed.tech/tags/ci.md>), [coding](<https://devfeed.tech/tags/coding.md>), [product](<https://devfeed.tech/tags/product.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [tests](<https://devfeed.tech/tags/tests.md>)

### AI overview

The article describes dltHub's code-native data ingestion approach for regulated life sciences. Pipelines are versioned and tested, while the context catalog records lineage, schema, and run evidence to support audits and governed changes. Snowflake is the featured destination, and coding agents can help build and operate pipelines under review.

### Source excerpt

In regulated life sciences, "can you explain your data?" has moved from a QA question to a board-level one. dltHub turns ingestion into validated, testable code - built on dlt, run on dltHub - so your team and their agents produce data that lands in Snowflake with the lineage, tests, and evidence regulators expect.

## Retiring the dbt Snowflake Native App

DevFeed: [Retiring the dbt Snowflake Native App](<https://devfeed.tech/articles/retiring-the-dbt-snowflake-native-app-80147.md>)

Original publisher: [Read original article](<https://www.getdbt.com/blog/retiring-the-dbt-snowflake-native-app>)

Author: Kyle Dempsey

Published: 2026-07-24T15:10:00Z

Content type: news

Language: en

Sources: [dbt Labs](<https://devfeed.tech/sources/dbt-labs.md>)

Topics: [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [stream financial data to Snowflake](<https://devfeed.tech/topics/stream-financial-data-to-snowflake.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [product](<https://devfeed.tech/tags/product.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>)

### AI overview

dbt Labs will retire the dbt Snowflake Native App from the Snowflake Marketplace. It enters maintenance mode in July 2026, with only critical bug fixes and security patches, and will be removed in November 2026. The change does not affect dbt platform, dbt Core, the Semantic Layer, or other integrations. dbt says it is shifting investment toward integrations customers are adopting, including Semantic Views and native dbt execution on Snowflake.

### Source excerpt

The dbt Snowflake Native App retires in November 2026. Here's what it means for you.

[Next page](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md?cursor=WyIyMDI2LTA3LTI0VDE1OjEwOjAwKzAwOjAwIiwgIjAzMzI5MGY1LWJmZmMtNDNkNi04ODhhLTg5MDM1YzM5OTVhMiJd>)