# etl

Published articles for etl.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## NetBox Analytics: Get Insight from Your System of Record without Manual Effort

DevFeed: [NetBox Analytics: Get Insight from Your System of Record without Manual Effort](<https://devfeed.tech/articles/netbox-analytics-get-insight-from-your-system-of-record-without-manual-effort-146905.md>)

Original publisher: [Read original article](<https://netboxlabs.com/blog/netbox-analytics-ga/>)

Author: Josh Chase

Published: 2026-11-19T21:50:51Z

Content type: release

Language: en

Sources: [NetBox Labs](<https://devfeed.tech/sources/netbox-labs.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [NetBox](<https://devfeed.tech/topics/netbox.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>), [etl](<https://devfeed.tech/topics/etl.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-pipeline](<https://devfeed.tech/tags/analytics-pipeline.md>), [api](<https://devfeed.tech/tags/api.md>), [blog](<https://devfeed.tech/tags/blog.md>), [budget](<https://devfeed.tech/tags/budget.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [cios](<https://devfeed.tech/tags/cios.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [etl](<https://devfeed.tech/tags/etl.md>), [governance](<https://devfeed.tech/tags/governance.md>), [grafana](<https://devfeed.tech/tags/grafana.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [history](<https://devfeed.tech/tags/history.md>), [ipam](<https://devfeed.tech/tags/ipam.md>), [kentik](<https://devfeed.tech/tags/kentik.md>), [netbox](<https://devfeed.tech/tags/netbox.md>), [system-of-record](<https://devfeed.tech/tags/system-of-record.md>)

### AI overview

NetBox Analytics provides 16 pre-built dashboards for IP, rack, power, port, and hardware lifecycle capacity questions, with changes reflected within minutes. NetBox Labs operates the data pipeline and warehouse, so teams do not need to build or maintain ETL; engineers and managers can use the same NetBox record for detailed and rolled-up views. The Essentials bundle has eight single-domain dashboards, while Complete adds eight cross-domain views. Four dashboards were added at general availability, including two that require specific plugins; two preview dashboards were renamed or enhanced, with Audit Evidence adding field-level change details. It is a NetBox Cloud add-on requiring NetBox 4.5 or later, and the curated catalog has no self-service dashboard builder. Teams can use the free NetBox Data Source for Grafana for custom Grafana views. Analytics focuses on inventory and trends, complementing observability tools that report current network activity.

### Source excerpt

The post NetBox Analytics: Get Insight from Your System of Record without Manual Effort appeared first on NetBox Labs.

## Hitachi Vantara bets smarter storage can close AI's pilot-to-production gap

DevFeed: [Hitachi Vantara bets smarter storage can close AI's pilot-to-production gap](<https://devfeed.tech/articles/hitachi-vantara-bets-smarter-storage-can-close-ai-s-pilot-to-production-gap-144137.md>)

Original publisher: [Read original article](<https://siliconangle.com/2026/11/19/apache-iceberg-intelligent-storage-help-ai-reach-production-sc26/>)

Author: Kelly Knight

Published: 2026-11-19T18:39:55Z

Content type: news

Language: en

Sources: [SiliconANGLE](<https://devfeed.tech/sources/siliconangle.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [big-data](<https://devfeed.tech/topics/big-data.md>)

Tags: [accelerated-computing](<https://devfeed.tech/tags/accelerated-computing.md>), [access-control](<https://devfeed.tech/tags/access-control.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-data-centers](<https://devfeed.tech/tags/ai-data-centers.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [ai-supercomputers](<https://devfeed.tech/tags/ai-supercomputers.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [arun-prasad-raman](<https://devfeed.tech/tags/arun-prasad-raman.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [compute-express-link](<https://devfeed.tech/tags/compute-express-link.md>), [cube-event-coverage](<https://devfeed.tech/tags/cube-event-coverage.md>), [data](<https://devfeed.tech/tags/data.md>), [data-governance](<https://devfeed.tech/tags/data-governance.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [data-scientists](<https://devfeed.tech/tags/data-scientists.md>), [database](<https://devfeed.tech/tags/database.md>), [energy](<https://devfeed.tech/tags/energy.md>), [energy-efficiency](<https://devfeed.tech/tags/energy-efficiency.md>), [etl](<https://devfeed.tech/tags/etl.md>), [gap](<https://devfeed.tech/tags/gap.md>), [gemma-allen](<https://devfeed.tech/tags/gemma-allen.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [high-performance-computing](<https://devfeed.tech/tags/high-performance-computing.md>), [high-performance-storage](<https://devfeed.tech/tags/high-performance-storage.md>), [high-speed-networking](<https://devfeed.tech/tags/high-speed-networking.md>), [hitachi-iq](<https://devfeed.tech/tags/hitachi-iq.md>), [hitachi-vantara](<https://devfeed.tech/tags/hitachi-vantara.md>), [hpc](<https://devfeed.tech/tags/hpc.md>), [hpc-infrastructure](<https://devfeed.tech/tags/hpc-infrastructure.md>), [hpc-unites](<https://devfeed.tech/tags/hpc-unites.md>), [jeff-lundberg](<https://devfeed.tech/tags/jeff-lundberg.md>), [john-furrier](<https://devfeed.tech/tags/john-furrier.md>), [memory-pooling](<https://devfeed.tech/tags/memory-pooling.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [parallel-file-systems](<https://devfeed.tech/tags/parallel-file-systems.md>), [pilot](<https://devfeed.tech/tags/pilot.md>), [production](<https://devfeed.tech/tags/production.md>), [sc26](<https://devfeed.tech/tags/sc26.md>), [siliconangle](<https://devfeed.tech/tags/siliconangle.md>), [storage](<https://devfeed.tech/tags/storage.md>), [thecube](<https://devfeed.tech/tags/thecube.md>)

### AI overview

The excerpt identifies fragmented, ungoverned data and access controls as obstacles to moving AI projects into production, and describes Hitachi Vantara's approach of adding classification and governance directly to storage. Its VSP One object store can flag files containing personal information so organizations can exclude them from models or restrict cross-border replication. For analytics, the company describes storing CT scan metadata in Apache Iceberg and querying it directly--for example, to find abdominal scans from the past year--without a separate ETL process or database. These are company representatives' examples; the excerpt does not provide independent performance results.

### Source excerpt

Enterprises pushing AI from pilot to production are learning that data, not GPUs, is the hardest part, and open table formats such as Apache Iceberg are part of the answer. Demand for AI infrastructure shows few signs of slowing, but much of the information companies need is still scattered, ungoverned and out of reach. That [...] The post Hitachi Vantara bets smarter storage can close AI's pilot-to-production gap appeared first on SiliconANGLE.

## A modern dilemma: Consume data from a stream or a lake

DevFeed: [A modern dilemma: Consume data from a stream or a lake](<https://devfeed.tech/articles/a-modern-dilemma-consume-data-from-a-stream-or-a-lake-141161.md>)

Original publisher: [Read original article](<https://medium.com/capital-one-tech/a-modern-dilemma-consume-data-from-a-stream-or-a-lake-296bbb354094?source=rss----3db3a67cb648---4>)

Author: Capital One Tech

Published: 2026-11-18T17:35:14Z

Content type: article

Language: en

Sources: [Capital One Tech](<https://devfeed.tech/sources/capital-one-tech.md>)

Topics: [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [etl](<https://devfeed.tech/topics/etl.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [apis](<https://devfeed.tech/tags/apis.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [batch](<https://devfeed.tech/tags/batch.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [business-processes](<https://devfeed.tech/tags/business-processes.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-computing](<https://devfeed.tech/tags/cloud-computing.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-lakes](<https://devfeed.tech/tags/data-lakes.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>), [enrichment](<https://devfeed.tech/tags/enrichment.md>), [etl](<https://devfeed.tech/tags/etl.md>), [event](<https://devfeed.tech/tags/event.md>), [flink](<https://devfeed.tech/tags/flink.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [latency](<https://devfeed.tech/tags/latency.md>), [legacy](<https://devfeed.tech/tags/legacy.md>), [log](<https://devfeed.tech/tags/log.md>), [mapping](<https://devfeed.tech/tags/mapping.md>), [model-context-protocol](<https://devfeed.tech/tags/model-context-protocol.md>), [software-architecture](<https://devfeed.tech/tags/software-architecture.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [stream](<https://devfeed.tech/tags/stream.md>)

### AI overview

Choosing a data consumption pattern early can avoid costly architectural changes later. APIs support synchronous, record-by-record requests and paginated multirecord results, making them suitable for low-latency needs; Model Context Protocol can provide similar access for agents and is often an API wrapper. Event streams, commonly built with Kafka, Flink or Spark Streaming, suit sub-second latency and enable per-event calculations, transformations and enrichment. The excerpt also identifies Delta Change Data Feed (CDF) as a record-by-record pattern and traditional batch ETL as a multirecord pattern, but ends before explaining CDF's mechanics or comparing these approaches further.

### Source excerpt

Data is a critical component of software architecture, providing the foundation for business processes, including those powered by artificial intelligence (AI). Quality, volume, latency, consistency, ordering and duplicates are essential considerations in software design. Modern cloud architectures consume data in various ways, such as through application programming interfaces (APIs), streams and data lakes. You can consume data at the individual event or record level as it arrives or in larger batches. Legacy architectures often process data in scheduled batches, forcing surrounding architecture and business processes to depend on that approach. Getting the data consumption approach right up front is vital, because rearchitecting later is costly. General patterns to consume data There are several general data consumption patterns. These include APIs (record-by-record or multirecord results), event stream consumption (record-by-record), Delta Change Data Feed (CDF) consumption (record-by-record) or traditional batch ETL-extract, transform, load (multirecord results). APIs are ideal for low-latency, record-by-record requirements. They also support multirecord results, typically through pagination. APIs usually follow a synchronous approach, where the client makes a call and waits for a response before moving to the next step. Model Context Protocol is an industry specification for agents to consume data and can support record-by-record or batch results via pagination, similar to APIs. They are often a wrapper on top of APIs. The event stream pattern works well for sub-second latency requirements and typically runs on a tech stack like Kafka, Flink or Spark Streaming. These technologies excel at record-level calculations (like on-the-fly calculation), transformations (like column mapping) or enrichment (like adding data to an existing event). Delta Lake is often used in data lakes and serves as a transaction log to standard Apache Parquet data files (source). CDF is

## The Data Integration Revolution: A Solution Architect's Perspective | Blog | Fivetran

DevFeed: [The Data Integration Revolution: A Solution Architect's Perspective | Blog | Fivetran](<https://devfeed.tech/articles/the-data-integration-revolution-a-solution-architect-s-perspective-blog-fivetran-141341.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/the-data-integration-revolution-a-solution-architects-perspective>)

Author: Dom Orsini

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [elt](<https://devfeed.tech/topics/elt.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>), [data migrations](<https://devfeed.tech/topics/data-migrations.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>), [Tool](<https://devfeed.tech/topics/tool.md>)

Tags: [architect](<https://devfeed.tech/tags/architect.md>), [batch](<https://devfeed.tech/tags/batch.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloudera](<https://devfeed.tech/tags/cloudera.md>), [coding](<https://devfeed.tech/tags/coding.md>), [collaboration](<https://devfeed.tech/tags/collaboration.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [data](<https://devfeed.tech/tags/data.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [data-migration](<https://devfeed.tech/tags/data-migration.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-teams](<https://devfeed.tech/tags/data-teams.md>), [databases](<https://devfeed.tech/tags/databases.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [elt](<https://devfeed.tech/tags/elt.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [etl](<https://devfeed.tech/tags/etl.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [solution-architect](<https://devfeed.tech/tags/solution-architect.md>)

### AI overview

The article's central finding is that managed SaaS data integration with ELT and cloud data warehouses has made data delivery faster and less engineering-intensive, freeing data teams to focus on business use. It contrasts this with on-premise ETL, which required manual coding, hardware tuning, and close operational support, and big data/IaaS systems, which sped delivery but added interdependent services, maintenance burdens, sync bottlenecks, and technology churn. In the modern approach described, fully managed pipelines need minimal configuration, while analysts build SQL transformations; dbt adds version control and collaboration practices. The author says this reduces coordination and schema-change work, though the piece presents a solution architect's perspective and specifically promotes Fivetran.

### Source excerpt

Life for data professionals keeps getting better -- and more interesting.

## ETL Platform Alternatives: Fivetran vs. Informatica and More | Blog | Fivetran

DevFeed: [ETL Platform Alternatives: Fivetran vs. Informatica and More | Blog | Fivetran](<https://devfeed.tech/articles/etl-platform-alternatives-fivetran-vs-informatica-and-more-blog-fivetran-141305.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/etl-platform-alternatives-fivetran-vs-informatica-and-more>)

Author: Charles Wang

Published: 2026-11-18T06:13:15Z

Content type: comparison

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [etl](<https://devfeed.tech/topics/etl.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>)

Tags: [alternatives](<https://devfeed.tech/tags/alternatives.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [business-intelligence](<https://devfeed.tech/tags/business-intelligence.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [cloud-native](<https://devfeed.tech/tags/cloud-native.md>), [compare](<https://devfeed.tech/tags/compare.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [etl](<https://devfeed.tech/tags/etl.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [informatica](<https://devfeed.tech/tags/informatica.md>), [integration](<https://devfeed.tech/tags/integration.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [security](<https://devfeed.tech/tags/security.md>), [sources](<https://devfeed.tech/tags/sources.md>), [tools](<https://devfeed.tech/tags/tools.md>)

### AI overview

The excerpt compares ETL platforms on destinations, connectors, pricing, trials, compliance and replication behavior. It says Informatica has opaque, sales-led pricing and a 30-day trial; Matillion has just over 100 connectors across five destinations, usage-based tiers and a 14-day trial; Xplenty offers almost 100 integrations and broad destination types but charges per source-destination connection and lacks change data capture; Stitch offers 130+ connectors, consumption-based tiers and a 14-day trial. The article also recommends testing platforms with live data for volume, speed, types, record structures and documentation, and asking how they handle source schema changes at the destination. It claims Fivetran supports 700+ sources and 15+ destinations, uses monthly-active-row pricing, and automatically migrates warehouse schemas when source schemas change.

### Source excerpt

Compare popular data integration tools on cost, security, reliability and number of supported sources.

## Fivetran value guide: What you're really getting

DevFeed: [Fivetran value guide: What you're really getting](<https://devfeed.tech/articles/fivetran-value-guide-what-you-re-really-getting-140229.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/fivetran-value-guide-what-youre-really-getting>)

Author: Shira Krishnan

Published: 2026-11-18T04:00:44Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [etl](<https://devfeed.tech/topics/etl.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-copilots](<https://devfeed.tech/tags/ai-copilots.md>), [airbyte](<https://devfeed.tech/tags/airbyte.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [automate](<https://devfeed.tech/tags/automate.md>), [automated](<https://devfeed.tech/tags/automated.md>), [automation](<https://devfeed.tech/tags/automation.md>), [availability](<https://devfeed.tech/tags/availability.md>), [business](<https://devfeed.tech/tags/business.md>), [business-value](<https://devfeed.tech/tags/business-value.md>), [can](<https://devfeed.tech/tags/can.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [clinical-trials](<https://devfeed.tech/tags/clinical-trials.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [etl](<https://devfeed.tech/tags/etl.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [roi](<https://devfeed.tech/tags/roi.md>), [tco](<https://devfeed.tech/tags/tco.md>)

### AI overview

Fivetran's value proposition is to reduce the labor and maintenance involved in moving data through a fully managed platform. The source cites 700+ prebuilt connectors, change data capture, automated schema drift handling, recovery from failed syncs, and destinations including warehouses, lakes, and vector databases. Its usage-based pricing charges for Monthly Active Rows per connector; initial and historical syncs are free, and volume and annual-contract discounts may apply. The guide contrasts this with DIY pipelines, which require ongoing engineering, and open source or budget tools, which it says can leave support, compliance, monitoring, or advanced features to the customer. Examples include National Australia Bank's reported 50% reduction in ingestion costs and New Relic's estimated $500,000 savings. These are vendor-presented claims; the excerpt provides no independent comparison methodology.

### Source excerpt

Fivetran is more than another ETL tool: It radically lowers the TCO of data integration and delivers ROI across your enterprise.

## Why a Monolithic Data Warehouse Is the Right Choice for Most Organizations | Blog | Fivetran

DevFeed: [Why a Monolithic Data Warehouse Is the Right Choice for Most Organizations | Blog | Fivetran](<https://devfeed.tech/articles/why-a-monolithic-data-warehouse-is-the-right-choice-for-most-organizations-blog-fivetran-140272.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/why-a-monolithic-data-warehouse-is-the-right-choice-for-most-organizations>)

Author: George Fraser

Published: 2026-11-18T02:46:08Z

Content type: opinion

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [etl](<https://devfeed.tech/topics/etl.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [schema design](<https://devfeed.tech/topics/schema-design.md>), [elt](<https://devfeed.tech/topics/elt.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [api](<https://devfeed.tech/tags/api.md>), [apis](<https://devfeed.tech/tags/apis.md>), [changelog](<https://devfeed.tech/tags/changelog.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-native](<https://devfeed.tech/tags/cloud-native.md>), [complex](<https://devfeed.tech/tags/complex.md>), [connect](<https://devfeed.tech/tags/connect.md>), [data](<https://devfeed.tech/tags/data.md>), [data-gravity](<https://devfeed.tech/tags/data-gravity.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [data-virtualization](<https://devfeed.tech/tags/data-virtualization.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [database](<https://devfeed.tech/tags/database.md>), [databases](<https://devfeed.tech/tags/databases.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [enterprise-data](<https://devfeed.tech/tags/enterprise-data.md>), [environments](<https://devfeed.tech/tags/environments.md>), [etl](<https://devfeed.tech/tags/etl.md>), [etl-pipeline](<https://devfeed.tech/tags/etl-pipeline.md>)

### AI overview

The article argues that common enterprise data warehouse problems stem from snapshot-based ETL, not from a centralized warehouse itself. It proposes separating replication from transformation: replicate source changes using native changelogs, then transform the replicated data with SQL inside the warehouse. The article says this reduces latency and data movement, makes schema development faster and incremental, and can include non-relational sources. It contrasts this approach with query federation, which it says is too slow for most real-world sources; data virtualization, which it says works in practice only with high-bandwidth sources such as HDFS or S3; and HTAP, which it says is rarely used because organizations cannot realistically run all tools on one database. Accurate changelog replication is described as the hardest part, sometimes requiring additional API calls or queries for context.

### Source excerpt

Most problems associated with the EDW are actually problems with ETL. The foundation of a successful data warehouse is proper separation of concerns.

## The Evolution of Data Engineering | Blog | Fivetran

DevFeed: [The Evolution of Data Engineering | Blog | Fivetran](<https://devfeed.tech/articles/the-evolution-of-data-engineering-blog-fivetran-140260.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/the-evolution-of-data-engineering>)

Author: Fivetran

Published: 2026-11-18T02:45:59Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [analysts](<https://devfeed.tech/tags/analysts.md>), [business-intelligence](<https://devfeed.tech/tags/business-intelligence.md>), [career](<https://devfeed.tech/tags/career.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-based](<https://devfeed.tech/tags/cloud-based.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-engineers](<https://devfeed.tech/tags/data-engineers.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [etl](<https://devfeed.tech/tags/etl.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [tools](<https://devfeed.tech/tags/tools.md>)

### AI overview

At Rent the Runway, cloud-based automation has freed data engineers to focus beyond ETL on reusable frameworks, self-service data access and strategic guidance. Danielle Dalton's team partners with five specialized data engineering teams to understand project goals, surface resource and cost tradeoffs, and communicate regularly so work can adapt as requirements emerge. The team is also supporting analysts' rollout of Looker. The excerpt stresses that self-service depends on reliable underlying data: teams should invest in that foundation and build trust by rolling out models gradually, since a new tool alone cannot fix poor data.

### Source excerpt

As cloud-based tools increasingly handle the grunt work of ETL, data engineers can start thinking bigger.

## Neo4j Virtual Graph is now generally available

DevFeed: [Neo4j Virtual Graph is now generally available](<https://devfeed.tech/articles/neo4j-virtual-graph-is-now-generally-available-135999.md>)

Original publisher: [Read original article](<https://neo4j.com/blog/news/neo4j-virtual-graph-is-now-generally-available/>)

Author: Stu Moore

Published: 2026-11-16T17:25:06Z

Content type: release

Language: en

Sources: [Neo4j](<https://devfeed.tech/sources/graph-database-technology-neo4j-blog.md>)

Topics: [Neo4j](<https://devfeed.tech/topics/neo4j.md>), [graph-database](<https://devfeed.tech/topics/graph-database.md>), [Graph Mining](<https://devfeed.tech/topics/graph-mining.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-workflows](<https://devfeed.tech/tags/agent-workflows.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [airport](<https://devfeed.tech/tags/airport.md>), [algorithms](<https://devfeed.tech/tags/algorithms.md>), [analyst](<https://devfeed.tech/tags/analyst.md>), [analysts](<https://devfeed.tech/tags/analysts.md>), [aura](<https://devfeed.tech/tags/aura.md>), [aura-graph-analytics](<https://devfeed.tech/tags/aura-graph-analytics.md>), [aws](<https://devfeed.tech/tags/aws.md>), [azure](<https://devfeed.tech/tags/azure.md>), [connected-data](<https://devfeed.tech/tags/connected-data.md>), [cypher-gql](<https://devfeed.tech/tags/cypher-gql.md>), [etl](<https://devfeed.tech/tags/etl.md>), [fraud-detection](<https://devfeed.tech/tags/fraud-detection.md>), [gcp](<https://devfeed.tech/tags/gcp.md>), [graph](<https://devfeed.tech/tags/graph.md>), [graph-database](<https://devfeed.tech/tags/graph-database.md>), [graph-technology](<https://devfeed.tech/tags/graph-technology.md>), [graph-visualization](<https://devfeed.tech/tags/graph-visualization.md>), [graphrag](<https://devfeed.tech/tags/graphrag.md>), [knowledge-graph](<https://devfeed.tech/tags/knowledge-graph.md>), [knowledge-graphs](<https://devfeed.tech/tags/knowledge-graphs.md>), [neo4j](<https://devfeed.tech/tags/neo4j.md>), [neo4j-aura](<https://devfeed.tech/tags/neo4j-aura.md>), [news](<https://devfeed.tech/tags/news.md>), [security](<https://devfeed.tech/tags/security.md>), [virtual-graph](<https://devfeed.tech/tags/virtual-graph.md>), [zero](<https://devfeed.tech/tags/zero.md>)

### AI overview

Neo4j Virtual Graph is generally available in Aura on GCP, with AWS and Azure support described as coming soon. It maps warehouse tables into a graph and translates Cypher queries into deterministic SQL executed in Snowflake, Databricks, Iceberg tables on S3, or BigQuery; data stays in the warehouse, and its access follows the connecting role's permissions and masking. Users can model graphs manually, from schema, or with AI, then query them through Neo4j tools and drivers. The excerpt highlights bounded quantified path patterns for traversals such as fraud and supply chain analysis, plus monitoring, pause/resume, and Private Link for dedicated environments. Virtual Graph is read-only and supports only part of Cypher; unsupported queries return error 42NG1. Warehouse execution means latency depends on compute and query complexity, with responses expected in seconds or minutes. The excerpt recommends AuraDB when writes, full Cypher, or millisecond response times are required. External IdPs, Aura MCP integration, and multi-source graphs are listed as forthcoming.

### Source excerpt

First on GCP, coming soon to AWS and Azure Today, Neo4j Virtual Graph is generally available in Neo4j Aura on GCP, with support for AWS and Azure to follow soon. Virtual Graph lets you build zero-ETL knowledge graphs, GraphRAG systems,... Read more ->

## Apache Spark vs Apache Flink: which should you use?

DevFeed: [Apache Spark vs Apache Flink: which should you use?](<https://devfeed.tech/articles/apache-spark-vs-apache-flink-which-should-you-use-133225.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/spark-vs-flink>)

Author: Databricks Staff

Published: 2026-11-13T20:44:47.516244Z

Content type: comparison

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [batch](<https://devfeed.tech/tags/batch.md>), [better](<https://devfeed.tech/tags/better.md>), [choice](<https://devfeed.tech/tags/choice.md>), [data-plus-ai-foundations](<https://devfeed.tech/tags/data-plus-ai-foundations.md>), [etl](<https://devfeed.tech/tags/etl.md>), [flink](<https://devfeed.tech/tags/flink.md>), [spark](<https://devfeed.tech/tags/spark.md>), [sql](<https://devfeed.tech/tags/sql.md>), [teams](<https://devfeed.tech/tags/teams.md>), [vs](<https://devfeed.tech/tags/vs.md>)

### AI overview

Spark and Flink both support exactly-once processing, stateful operators and SQL APIs, but Spark spans batch, streaming, SQL and AI while Flink is stream-native. Spark Real-Time Mode uses a trigger on an existing Structured Streaming query to process continuously at millisecond latency, so teams can move between latency tiers without rewriting query code. The excerpt says Databricks' benchmark found Spark matched Flink on stateless work and had lower p99 latency for stateful aggregation and stream-static joins; results came from different providers and the comparison used Spark's worst run versus Flink's best, so teams should benchmark their own workloads. Flink may fit specialized cases such as very large keyed state, complex event processing, or an existing Flink platform. Real-Time Mode requires update output mode and Kafka or a row-level foreach sink; stateful operators are available on Databricks and are slated for open-source Spark 4.3, according to the excerpt.

### Source excerpt

For most teams, Apache Spark™ is the better choice. Spark runs batch ETL, SQL analytics,...

## Build a self-healing ETL pipeline with Amazon Bedrock AgentCore

DevFeed: [Build a self-healing ETL pipeline with Amazon Bedrock AgentCore](<https://devfeed.tech/articles/build-a-self-healing-etl-pipeline-with-amazon-bedrock-agentcore-133386.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/build-a-self-healing-etl-pipeline-with-amazon-bedrock-agentcore/>)

Author: Kevin Tinn

Published: 2026-11-13T19:08:10Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon Bedrock AgentCore](<https://devfeed.tech/topics/amazon-bedrock-agentcore.md>), [etl](<https://devfeed.tech/topics/etl.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon Bedrock Knowledge Bases](<https://devfeed.tech/topics/amazon-bedrock-knowledge-bases.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [knowledge-engineering](<https://devfeed.tech/topics/knowledge-engineering.md>), [aws-sns](<https://devfeed.tech/topics/aws-sns.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [agentcore](<https://devfeed.tech/tags/agentcore.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-bedrock-agentcore](<https://devfeed.tech/tags/amazon-bedrock-agentcore.md>), [automatically](<https://devfeed.tech/tags/automatically.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [bedrock](<https://devfeed.tech/tags/bedrock.md>), [bedrock-agentcore](<https://devfeed.tech/tags/bedrock-agentcore.md>), [build](<https://devfeed.tech/tags/build.md>), [can](<https://devfeed.tech/tags/can.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [domain](<https://devfeed.tech/tags/domain.md>), [errors](<https://devfeed.tech/tags/errors.md>), [etl](<https://devfeed.tech/tags/etl.md>), [etl-pipeline](<https://devfeed.tech/tags/etl-pipeline.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [self-healing](<https://devfeed.tech/tags/self-healing.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

The described AWS Glue pipeline routes records that fail schema validation to a four-agent system on Amazon Bedrock AgentCore. A correction agent queries a Bedrock Knowledge Base for domain guidance and looks up precomputed strategies in S3 Vectors in parallel, selecting the higher-confidence result. At the default 0.7 threshold, it applies qualifying corrections; otherwise, the orchestrator sends an SNS alert for human review. A persistence agent writes corrected records to both corrected and transformed data lake zones. The sample annotates records with the chosen strategy rather than changing field values; direct transformations require extending its correction function. With repository credentials configured, a remediation agent can analyze the source error and open a GitHub pull request; without them, that step is skipped. The post outlines AWS CDK deployment and sample-data workflow, and notes required AWS services, model access, Python 3.10+, and a container runtime for local testing.

### Source excerpt

Build a self-healing ETL pipeline on AWS Glue and Amazon Bedrock AgentCore that automatically corrects bad records using domain knowledge, writes the fixes back to the data lake, and can open pull requests to prevent the same errors from recurring.

## Automate ETL pipeline creation with Amazon Bedrock AgentCore and AWS Glue

DevFeed: [Automate ETL pipeline creation with Amazon Bedrock AgentCore and AWS Glue](<https://devfeed.tech/articles/automate-etl-pipeline-creation-with-amazon-bedrock-agentcore-and-aws-glue-126345.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/automate-etl-pipeline-creation-with-amazon-bedrock-agentcore-and-aws-glue/>)

Author: Mike Brouwers

Published: 2026-11-09T23:15:31Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AI for data automation](<https://devfeed.tech/topics/ai-for-data-automation.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Amazon Bedrock AgentCore](<https://devfeed.tech/topics/amazon-bedrock-agentcore.md>), [Amazon Bedrock](<https://devfeed.tech/topics/amazon-bedrock.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [Amazon Cognito](<https://devfeed.tech/topics/amazon-cognito.md>), [AWS CloudShell](<https://devfeed.tech/topics/aws-cloudshell.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-bedrock-agentcore](<https://devfeed.tech/tags/amazon-bedrock-agentcore.md>), [amazon-eventbridge](<https://devfeed.tech/tags/amazon-eventbridge.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [anthropic-claude](<https://devfeed.tech/tags/anthropic-claude.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [approval-gates](<https://devfeed.tech/tags/approval-gates.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [etl](<https://devfeed.tech/tags/etl.md>), [etl-pipeline](<https://devfeed.tech/tags/etl-pipeline.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Agentic Data Ingestion automates file or database-table ETL setup by proposing schema mappings, generating PySpark transformations, testing them in AWS Glue Spark, and creating and running AWS Glue jobs through a Step Functions workflow. Human approval gates review mappings and tested code; low-confidence mappings receive extra context, and a retry agent may rerun selected failed jobs while skipping cases where retries could duplicate writes or cannot fix the error. Generated code is kept reviewable with target-specific Jinja2 templates, an allowlist for the model-written transformation block, and checks that required job arguments resolve. Tests use a 100-row file sample in a matching Glue interactive session; database tests use schema-generated rows because the session cannot access databases. In the measured US West (Oregon) sample deployment, runs cost about $0.10-0.25 and idle operation about $15-20 monthly; wider sources and lower-cost models were not measured. The solution requires Claude Sonnet 4.6 access and, outside US regions, a model configuration change; production identity federation and role-based access are listed as future hardening steps.

### Source excerpt

Agentic Data Ingestion is an open-source solution that uses AI agents on Amazon Bedrock to automate data pipeline creation, from file upload to queryable tables. Point it at a file or database table, and foundation models propose the field mapping, generate and test the AWS Glue transform, and run the job behind human approval gates.

## Autonomous ETL Pipelines for Salesforce M&A Consolidation: A Reference Framework, Jitterbit Mapping, and Reproducible Simulation Study

DevFeed: [Autonomous ETL Pipelines for Salesforce M&A Consolidation: A Reference Framework, Jitterbit Mapping, and Reproducible Simulation Study](<https://devfeed.tech/articles/autonomous-etl-pipelines-for-salesforce-m-a-consolidation-a-reference-framework-jitterbit-mapping-and-reproducible-simulation-study-118118.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05918>)

Author: Imran Basha Shaik, Sahla Sultana, Khader Moinuddin Shaik

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [etl](<https://devfeed.tech/topics/etl.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [synchronization](<https://devfeed.tech/topics/synchronization.md>)

Tags: [autonomous](<https://devfeed.tech/tags/autonomous.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cs-et](<https://devfeed.tech/tags/cs-et.md>), [customer-data](<https://devfeed.tech/tags/customer-data.md>), [data-management](<https://devfeed.tech/tags/data-management.md>), [declarative](<https://devfeed.tech/tags/declarative.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [etl](<https://devfeed.tech/tags/etl.md>), [etl-pipelines](<https://devfeed.tech/tags/etl-pipelines.md>), [events](<https://devfeed.tech/tags/events.md>), [framework](<https://devfeed.tech/tags/framework.md>), [load](<https://devfeed.tech/tags/load.md>), [mapping](<https://devfeed.tech/tags/mapping.md>), [master-data-management](<https://devfeed.tech/tags/master-data-management.md>), [matching](<https://devfeed.tech/tags/matching.md>), [open](<https://devfeed.tech/tags/open.md>), [prototype](<https://devfeed.tech/tags/prototype.md>), [quality](<https://devfeed.tech/tags/quality.md>), [salesforce](<https://devfeed.tech/tags/salesforce.md>), [simulation](<https://devfeed.tech/tags/simulation.md>), [study](<https://devfeed.tech/tags/study.md>)

### AI overview

The proposed Salesforce consolidation framework combines partitioned bulk extraction, declarative mappings, quality gates, deterministic and blocked fuzzy matching, constrained loads with per-record error capture, and incremental synchronization protected by a data-health circuit breaker; its stages are mapped to Jitterbit Integration Studio. In synthetic two-organization simulations, hybrid matching achieved pairwise F1 scores of 0.918 for accounts and 0.885 for contacts at a preset threshold; object-specific calibration raised these to 0.938 and 0.963. Quality gates prevented load failures and silent loss of unmapped picklist values without flagging clean records; without them, runs had about 1,000 load failures. In five simulated days of hourly sync, replayed events were ignored, an upstream fault was contained to its batch, and the target matched a full rebuild on over 99.3% of field values. Production validation remains future work.

### Source excerpt

arXiv:2611.05918v1 Announce Type: new Abstract: Merging companies that each run Salesforce must consolidate their customer data into a single organization while both source systems remain in use. This requires reconciling different schemas and picklists, resolving duplicate customers across and within organizations, preserving relationships between records, and keeping the target current until cutover. This article proposes a reference framework for an autonomous extract-transform-load pipeline for this task. It combines partitioned bulk extraction, declarative mapping templates, rule-based and statistical quality gates, master data management with deterministic and blocked fuzzy matching, constrained loading with per-record error capture, and change data capture guarded by a data-health circuit breaker. Each stage is mapped to Jitterbit Integration Studio constructs. The framework was implemented as an open reference prototype and evaluated on synthetic two-organization scenarios with known ground truth at three scales, from 32,040 to 383,973 source records, over five random seeds. At a preset threshold, hybrid matching reached pairwise F1 scores of 0.918 for accounts and 0.885 for contacts; calibrating the threshold per object would raise these to 0.938 and 0.963. The quality gates prevented all load failures and all silent loss of unmapped picklist values without wrongly flagging any clean record, whereas removing them produced about 1,000 load failures per run. Over five simulated days of hourly incremental synchronization, replayed events were ignored, an injected upstream fault was contained in the batch in which it occurred, and the target agreed with a full rebuild on more than 99.3 percent of field values. Validation on a production deployment remains future work.

## NovaFilter - A Rule-Based ETL Staging Framework for Fault-Tolerant Relational Databases

DevFeed: [NovaFilter - A Rule-Based ETL Staging Framework for Fault-Tolerant Relational Databases](<https://devfeed.tech/articles/novafilter-a-rule-based-etl-staging-framework-for-fault-tolerant-relational-databases-113643.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.00959>)

Author: Meike Klettke

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [etl](<https://devfeed.tech/topics/etl.md>), [rule-engine](<https://devfeed.tech/topics/rule-engine.md>)

Tags: [combines](<https://devfeed.tech/tags/combines.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [data](<https://devfeed.tech/tags/data.md>), [data-transformation](<https://devfeed.tech/tags/data-transformation.md>), [databases](<https://devfeed.tech/tags/databases.md>), [discussion](<https://devfeed.tech/tags/discussion.md>), [domain](<https://devfeed.tech/tags/domain.md>), [etl](<https://devfeed.tech/tags/etl.md>), [example](<https://devfeed.tech/tags/example.md>), [experts](<https://devfeed.tech/tags/experts.md>), [formats](<https://devfeed.tech/tags/formats.md>), [framework](<https://devfeed.tech/tags/framework.md>), [human-in-the-loop](<https://devfeed.tech/tags/human-in-the-loop.md>), [input](<https://devfeed.tech/tags/input.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [models](<https://devfeed.tech/tags/models.md>), [process](<https://devfeed.tech/tags/process.md>), [relational-databases](<https://devfeed.tech/tags/relational-databases.md>), [schemas](<https://devfeed.tech/tags/schemas.md>), [script](<https://devfeed.tech/tags/script.md>)

### AI overview

NovaFilter derives structured relational data from heterogeneous formats by combining consistency checks, rule-based transformations, and a human-in-the-loop example discussion. User responses expand an explicit rule base, which the source says makes transformations deterministic, transparent, and reusable. The authors present it as a potential replacement for conventional ETL scripts and say domain experts without programming skills can contribute through the example discussion. The supplied text is an abstract and provides no implementation details or evaluation results.

### Source excerpt

arXiv:2611.00959v1 Announce Type: new Abstract: NovaFilter is a completely new approach to deriving structured relational data from heterogeneous input data in different data formats. It combines consistency checks, rule-based data transformation, and a human-in-the-loop example discussion. Using these three components, NovaFilter stores all transformation knowledge in an explicit rule base, which is continuously expanded through the example discussion with the user. The resulting data transformation process is deterministic, transparent, and reusable. As a result, the method can replace conventional ETL scripts and offers a completely new strategy for transforming data between different data models and schemas. Its main advantage is that users without programming skills (e.g. domain experts) can answer the example discussion and thus it is a building block for democratization of ETL script developement.

## Snowpark Python Supports Native to\_polars()

DevFeed: [Snowpark Python Supports Native to\_polars()](<https://devfeed.tech/articles/snowpark-python-supports-native-to-polars-106578.md>)

Original publisher: [Read original article](<https://medium.com/snowflake/snowpark-python-supports-native-to-polars-b84e03721b5d?source=rss----34b6daafc07---4>)

Author: Qinyi Ding

Published: 2026-10-29T17:22:34Z

Content type: release

Language: en

Sources: [Snowflake Builders](<https://devfeed.tech/sources/snowflake-builders.md>)

Topics: [Python](<https://devfeed.tech/topics/python.md>), [groupby](<https://devfeed.tech/topics/groupby.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [apache-arrow](<https://devfeed.tech/topics/apache-arrow.md>), [pandas](<https://devfeed.tech/topics/pandas.md>)

Tags: [data-processing](<https://devfeed.tech/tags/data-processing.md>), [etl](<https://devfeed.tech/tags/etl.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [polars](<https://devfeed.tech/tags/polars.md>), [polars-dataframe](<https://devfeed.tech/tags/polars-dataframe.md>), [python](<https://devfeed.tech/tags/python.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [snowpark](<https://devfeed.tech/tags/snowpark.md>)

### AI overview

Snowpark Python's `DataFrame.to_polars()` is generally available in Snowpark Python 1.55.3 with Polars 1.0 or later. It executes a Snowpark query and returns a fully materialized Polars DataFrame; filter and select data in Snowpark before conversion because later Polars work runs in Python and the result must fit in memory. Parquet is the default transport: Snowpark stages query results as Parquet files and reads them in parallel, which can speed large transfers, especially in stored procedures. For timezone-aware timestamps or full timestamp and floating-point precision, use `to_polars(transport="arrow")`: Parquet lacks support for `TIMESTAMP_LTZ` and `TIMESTAMP_TZ`, rounds `TIMESTAMP_NTZ` to milliseconds, and reduces `FLOAT` to `float32`. The example filters out South in Snowflake, then groups the remaining rows in Polars to yield North revenue of 320.

### Source excerpt

Photo by Annie Spratt on Unsplash Snowpark Python's DataFrame.to_polars() is now generally available, bringing Snowflake query results into a Polars DataFrame with a single call. Its default Parquet transport reads files in parallel to accelerate large data transfers, connecting data preparation in Snowflake with analytics and data processing in Polars. Why it matters Snowpark already supports pandas through to_pandas(). With to_polars(), we are extending that interoperability to developers who use Polars in their Python workflows. You can use Snowpark to filter, join, and aggregate large datasets in Snowflake, then return the results as a Polars DataFrame for further processing. This lets you combine Snowflake's data processing with your existing Polars application, using the same Snowpark session. How to use it With a configured Snowpark session and Polars installed, call to_polars() on a Snowpark DataFrame. Here is a small example: import polars as pl from snowflake.snowpark.functions import col snowpark_df = session.create_dataframe( [["North", 120], ["South", 80], ["North", 200]], schema=["REGION", "REVENUE"], ).filter(col("REVENUE") >= 100) polars_df = snowpark_df.to_polars() # Continue processing with Polars. summary = polars_df.group_by("REGION").agg( pl.col("REVENUE").sum() ) print(summary) In this example, calling to_polars() executes the Snowpark query, filtering out rows with revenue above 100 in Snowflake and returning the remaining rows as an eager Polars DataFrame. Polars then groups those rows by region and sums their revenue, producing one result: North with a total of 320. South is excluded by the filter. Note DataFrame.to_polars() uses Parquet transport by default. Snowpark unloads query results to the session stage as Parquet files and reads them in parallel into Polars. This can speed up large, transfer-heavy workloads, especially in Snowpark Python stored procedures. Use to_polars(transport="arrow") when you need to preserve timezone-aware time

## NetSuite-Snowflake integration: 3 methods and setup guide

DevFeed: [NetSuite-Snowflake integration: 3 methods and setup guide](<https://devfeed.tech/articles/netsuite-snowflake-integration-3-methods-and-setup-guide-102832.md>)

Original publisher: [Read original article](<https://www.fivetran.com/blog/netsuite-snowflake-integration>)

Author: Fivetran

Published: 2026-10-27T17:37:57Z

Content type: tutorial

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [elt](<https://devfeed.tech/topics/elt.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Tool](<https://devfeed.tech/topics/tool.md>), [build vs buy database solutions](<https://devfeed.tech/topics/build-vs-buy-database-solutions.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [ai](<https://devfeed.tech/tags/ai.md>), [airbyte](<https://devfeed.tech/tags/airbyte.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [api](<https://devfeed.tech/tags/api.md>), [application-integration](<https://devfeed.tech/tags/application-integration.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [audit](<https://devfeed.tech/tags/audit.md>), [celigo](<https://devfeed.tech/tags/celigo.md>), [etl](<https://devfeed.tech/tags/etl.md>), [etl-tools](<https://devfeed.tech/tags/etl-tools.md>), [guide](<https://devfeed.tech/tags/guide.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [integration](<https://devfeed.tech/tags/integration.md>), [netsuite](<https://devfeed.tech/tags/netsuite.md>), [platform](<https://devfeed.tech/tags/platform.md>), [setup](<https://devfeed.tech/tags/setup.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [snowflake-integration](<https://devfeed.tech/tags/snowflake-integration.md>), [tools](<https://devfeed.tech/tags/tools.md>)

### AI overview

NetSuite-to-Snowflake integration usually sends ERP data to Snowflake for analysis; sending data back requires approved fields, explicit permissions, and safeguards because NetSuite remains the system of record. The excerpt compares three approaches: managed ELT/ETL tools suit warehouse-centered analytics but require modeling and validation of reverse-write support; Oracle's NetSuite Integration Platform supports NetSuite-centered workflows and monitoring but adds licensing cost and may need a separate ELT layer for large analytics workloads; iPaaS tools coordinate events across applications but need technical ownership for complex mappings and are a poor fit for large historical extracts. Setup across the approaches involves choosing records and sync direction, connecting systems, mapping identifiers and fields, validating data, and configuring schedules, retries, conflict handling, and alerts. The article describes Fivetran as managed ingestion with incremental sync for eligible tables and scheduled re-imports for others, plus Activations for governed updates to supported NetSuite objects.

### Source excerpt

Learn how a NetSuite-Snowflake integration works. Explore methods like ETL tools, iPaaS solutions, and NetSuite Integration Platform, and how to set them up.

## NetSuite-Snowflake integration: 3 methods and setup guide

DevFeed: [NetSuite-Snowflake integration: 3 methods and setup guide](<https://devfeed.tech/articles/netsuite-snowflake-integration-3-methods-and-setup-guide-136810.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/netsuite-snowflake-integration>)

Author: Fivetran

Published: 2026-10-27T17:37:57Z

Content type: tutorial

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [elt](<https://devfeed.tech/topics/elt.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Cloud Architecture](<https://devfeed.tech/topics/cloud-architecture.md>)

Tags: [3](<https://devfeed.tech/tags/3.md>), [etl](<https://devfeed.tech/tags/etl.md>), [guide](<https://devfeed.tech/tags/guide.md>), [integration](<https://devfeed.tech/tags/integration.md>), [netsuite](<https://devfeed.tech/tags/netsuite.md>), [setup](<https://devfeed.tech/tags/setup.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [snowflake-integration](<https://devfeed.tech/tags/snowflake-integration.md>)

### AI overview

NetSuite-to-Snowflake integrations most often copy ERP data into Snowflake for analysis; sending modeled results back requires approved fields, supported objects, and safeguards because NetSuite remains the operational system of record. Three approaches fit different needs: managed ELT/ETL tools suit analytics-led replication but require warehouse modeling and validation of write-back support; Oracle's NetSuite Integration Platform coordinates NetSuite-centered workflows but adds licensing costs and may need a separate ELT layer for large analytical workloads; iPaaS tools such as Celigo, Boomi, and Workato orchestrate events across applications but are a poor fit for large historical analytics extracts. Setup generally involves selecting records and flows, authenticating connections, mapping IDs and fields, configuring schedules and error handling, then validating loads or testing conflicts and replays. The excerpt describes Fivetran's managed connector as handling initial loads, incremental syncs for eligible tables, and scheduled re-imports otherwise; Activations can create or update approved NetSuite records.

### Source excerpt

Learn how a NetSuite-Snowflake integration works. Explore methods like ETL tools, iPaaS solutions, and NetSuite Integration Platform, and how to set them up.

## Nine Months of Daily Calls: How I Took a Customer From 20% to 100% Customer Satisfaction Index

DevFeed: [Nine Months of Daily Calls: How I Took a Customer From 20% to 100% Customer Satisfaction Index](<https://devfeed.tech/articles/nine-months-of-daily-calls-how-i-took-a-customer-from-20-to-100-customer-satisfaction-index-100881.md>)

Original publisher: [Read original article](<https://hackernoon.com/nine-months-of-daily-calls-how-i-took-a-customer-from-20percent-to-100percent-customer-satisfaction-index?source=rss>)

Author: SnehaPatil

Published: 2026-10-26T19:31:14Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Support](<https://devfeed.tech/topics/support.md>), [etl](<https://devfeed.tech/topics/etl.md>)

Tags: [admin](<https://devfeed.tech/tags/admin.md>), [automation](<https://devfeed.tech/tags/automation.md>), [backlog](<https://devfeed.tech/tags/backlog.md>), [backup](<https://devfeed.tech/tags/backup.md>), [breaches](<https://devfeed.tech/tags/breaches.md>), [call](<https://devfeed.tech/tags/call.md>), [cron](<https://devfeed.tech/tags/cron.md>), [csi](<https://devfeed.tech/tags/csi.md>), [customer](<https://devfeed.tech/tags/customer.md>), [customer-experience](<https://devfeed.tech/tags/customer-experience.md>), [customer-satisfaction](<https://devfeed.tech/tags/customer-satisfaction.md>), [customer-satisfaction-index](<https://devfeed.tech/tags/customer-satisfaction-index.md>), [daily](<https://devfeed.tech/tags/daily.md>), [dashboard](<https://devfeed.tech/tags/dashboard.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [delivery-lead](<https://devfeed.tech/tags/delivery-lead.md>), [email](<https://devfeed.tech/tags/email.md>), [etl](<https://devfeed.tech/tags/etl.md>), [informatica](<https://devfeed.tech/tags/informatica.md>), [leadership](<https://devfeed.tech/tags/leadership.md>), [service-level-agreement](<https://devfeed.tech/tags/service-level-agreement.md>), [support](<https://devfeed.tech/tags/support.md>), [team](<https://devfeed.tech/tags/team.md>), [team-management](<https://devfeed.tech/tags/team-management.md>), [team-productivity](<https://devfeed.tech/tags/team-productivity.md>)

### AI overview

A 13-person Informatica administration team cleared a 300+ request backlog and brought reported SLA breaches from 90% to zero over nine months, while customer satisfaction rose from 20% to 100%. The lead paired one of four experienced admins with a junior on each shift, added buddy reviews and shift handover emails, held weekly technical training, assigned a dedicated request dispatcher, and tracked work on a dashboard shared live with the customer. After difficulties obtaining transition materials from the outgoing vendor, the lead escalated to secure key documents. The team later automated password resets, removing a reported 10-15 manual requests per month at about 20 minutes each. CSI was measured quarterly through a customer survey.

### Source excerpt

How I turned a failing Informatica Admin project around - Fixing SLA, clearing 300+ backlog, and rebuilding high-performing support team

## How Writer Keeps Its AI Infrastructure Running with Snowflake Hybrid Tables

DevFeed: [How Writer Keeps Its AI Infrastructure Running with Snowflake Hybrid Tables](<https://devfeed.tech/articles/how-writer-keeps-its-ai-infrastructure-running-with-snowflake-hybrid-tables-101920.md>)

Original publisher: [Read original article](<https://medium.com/snowflake/how-writer-keeps-its-ai-infrastructure-running-with-snowflake-hybrid-tables-80f55dd652fa?source=rss----34b6daafc07---4>)

Author: Aeril Xie

Published: 2026-10-26T14:01:05Z

Content type: article

Language: en

Sources: [Snowflake Builders](<https://devfeed.tech/sources/snowflake-builders.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [database-optimization](<https://devfeed.tech/topics/database-optimization.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-infrastructure](<https://devfeed.tech/tags/data-infrastructure.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [enterprise-ai](<https://devfeed.tech/tags/enterprise-ai.md>), [etl](<https://devfeed.tech/tags/etl.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [observability](<https://devfeed.tech/tags/observability.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [platform](<https://devfeed.tech/tags/platform.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>)

### AI overview

Writer moved its high-volume LLM trace writes from a standard Snowflake table to a Hybrid Table after lock contention caused P50 insert latency of 2,362 ms and failures in about 45% of insertions. The migration required creating a Hybrid Table with a schema and primary key, updating the API destination, and backfilling historical data. Hybrid Tables reduced P50 latency to 154 ms and failures to near zero (typically 3-10 per day); Writer also reduced its ingest warehouse from medium to extra small. As the table grew beyond 33 million rows, analytics and reconciliation queries slowed because Hybrid Tables use row-oriented storage optimized for primary-key access. Writer now keeps writes on the Hybrid Table and uses a Snowflake Task to incrementally copy rows into a timestamp-clustered standard table for analytics. This retains in-Snowflake access without external infrastructure, while using each table type for the workload it handles best.

### Source excerpt

Aeril Xie, Data Engineer at Writer At Writer, data is at the center of everything we do. As an enterprise AI platform, we route every customer request through LLM inference providers. Every one of those calls generates a trace record: the model called, the prompt, the response, latency, token counts, and outcome. Rather than running a separate observability database alongside our data warehouse, we made Snowflake the destination for raw trace writes too, so our engineers and analysts can query everything in one place. My job as a data engineer is to make sure that data lands reliably, because it powers our QA processes and gives us the visibility we need into how our models are behaving in production. At the scale we operate, "reliably" is harder than it sounds. The Problem: Lock Contention at Scale Every Writer customer interaction that touches an LLM generates a row. At enterprise scale, that's not a batch job you can schedule. It's a continuous stream that doesn't pause. Our LLM tracing pipeline writes to Snowflake at high velocity: up to 600 rows per minute at peak, and roughly 1.3 million rows per day for our primary traces table. For a while, we were directing this traffic to a standard Snowflake table. And for a while, it mostly worked. Then it didn't. We started seeing write failures from lock contention -- too many concurrent writes competing for the same table resource. Standard Snowflake tables aren't designed for that kind of continuous, high-concurrency write pattern. At our peak traffic levels, INSERT latency at P50 was reaching 2,362 milliseconds. Failure rates climbed to 45% of all insertions. Every failed write was a lost data point. Initially the data wasn't business-critical, so the failures were tolerable. But as this tracing data became the foundation of our observability pipeline and QA processes, the gap it left in our data got harder to ignore. Evaluating the Options When we opened a support ticket with Snowflake to investigate the failures, o

## A Successful ETL Job Can Hide Thousands of Data Discrepancies

DevFeed: [A Successful ETL Job Can Hide Thousands of Data Discrepancies](<https://devfeed.tech/articles/a-successful-etl-job-can-hide-thousands-of-data-discrepancies-97404.md>)

Original publisher: [Read original article](<https://hackernoon.com/a-successful-etl-job-can-hide-thousands-of-data-discrepancies?source=rss>)

Author: RISHI SHRIVASTAVA

Published: 2026-10-23T08:49:40Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [etl](<https://devfeed.tech/topics/etl.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>), [SQL](<https://devfeed.tech/topics/sql.md>), [data type](<https://devfeed.tech/topics/data-type.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [about](<https://devfeed.tech/tags/about.md>), [acceptance](<https://devfeed.tech/tags/acceptance.md>), [added](<https://devfeed.tech/tags/added.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [approach](<https://devfeed.tech/tags/approach.md>), [arithmetic](<https://devfeed.tech/tags/arithmetic.md>), [attention](<https://devfeed.tech/tags/attention.md>), [automated](<https://devfeed.tech/tags/automated.md>), [can](<https://devfeed.tech/tags/can.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [data-validation](<https://devfeed.tech/tags/data-validation.md>), [errors](<https://devfeed.tech/tags/errors.md>), [etl](<https://devfeed.tech/tags/etl.md>), [etl-data-reconciliation](<https://devfeed.tech/tags/etl-data-reconciliation.md>), [etl-pipeline-reliability](<https://devfeed.tech/tags/etl-pipeline-reliability.md>), [help](<https://devfeed.tech/tags/help.md>), [python](<https://devfeed.tech/tags/python.md>), [python-pandas](<https://devfeed.tech/tags/python-pandas.md>), [reconciliation](<https://devfeed.tech/tags/reconciliation.md>), [records](<https://devfeed.tech/tags/records.md>), [sql](<https://devfeed.tech/tags/sql.md>), [sql-data-validation](<https://devfeed.tech/tags/sql-data-validation.md>)

### AI overview

A successful ETL run does not prove its output is complete or correct: matching total row counts can hide different missing and newly added records. In a fictional example, a net count difference of 12,330 masks 13,360 distinct identifier exceptions. The article recommends bidirectional reconciliation by business key, followed by duplicate and null-key checks, attribute-level validation, and exception classification with source or master data to guide investigation. SQL set comparisons work for database data; Python with pandas supports repeatable comparisons across files, while preserving key types such as leading-zero identifiers. Both approaches depend on sound normalization and uniqueness assumptions, and pandas sets discard duplicates; large datasets may require database-side or distributed processing. Business rules and context are still needed to determine whether discrepancies are defects. The excerpt proposes layered checks from dataset structure and counts through identifier and attribute validation to documented investigation and resolution.

### Source excerpt

Learn why successful ETL jobs can produce incorrect data and how SQL, Python, and bidirectional reconciliation help detect missing records and data errors.

## Custom ETL or AI? Choosing the Right Strategy for Legacy Data Migration

DevFeed: [Custom ETL or AI? Choosing the Right Strategy for Legacy Data Migration](<https://devfeed.tech/articles/custom-etl-or-ai-choosing-the-right-strategy-for-legacy-data-migration-86498.md>)

Original publisher: [Read original article](<https://hackernoon.com/custom-etl-or-ai-choosing-the-right-strategy-for-legacy-data-migration?source=rss>)

Author: aparnamitra

Published: 2026-10-13T20:05:03Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [data migrations](<https://devfeed.tech/topics/data-migrations.md>), [etl](<https://devfeed.tech/topics/etl.md>), [AI for data automation](<https://devfeed.tech/topics/ai-for-data-automation.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-assisted-data-migration](<https://devfeed.tech/tags/ai-assisted-data-migration.md>), [ai-generated-sql](<https://devfeed.tech/tags/ai-generated-sql.md>), [custom-etl-pipelines](<https://devfeed.tech/tags/custom-etl-pipelines.md>), [data-migration](<https://devfeed.tech/tags/data-migration.md>), [database-reconciliation](<https://devfeed.tech/tags/database-reconciliation.md>), [etl](<https://devfeed.tech/tags/etl.md>), [etl-transformation-testing](<https://devfeed.tech/tags/etl-transformation-testing.md>), [human-in-the-loop-engineering](<https://devfeed.tech/tags/human-in-the-loop-engineering.md>), [legacy-database-modernization](<https://devfeed.tech/tags/legacy-database-modernization.md>), [migration](<https://devfeed.tech/tags/migration.md>), [mysql-to-postgresql-migration](<https://devfeed.tech/tags/mysql-to-postgresql-migration.md>)

### AI overview

The article compares custom ETL with AI-assisted workflows for migrating legacy databases. It describes AI drafting target schemas and transformation logic, while engineers verify business rules, correct date and status mappings, and reconcile results before production. Its recommendation is to use AI for repetitive, lower-risk work and keep people responsible for sensitive logic and validation.

### Source excerpt

Compare Custom ETL vs. AI data pipeline migration. Learn why the best strategy lets AI automate the grunt work while engineers manage the high-risk tasks.

## CDC Tools With the Most Predictable Pricing for High-Volume Data Pipelines

DevFeed: [CDC Tools With the Most Predictable Pricing for High-Volume Data Pipelines](<https://devfeed.tech/articles/cdc-tools-with-the-most-predictable-pricing-for-high-volume-data-pipelines-84153.md>)

Original publisher: [Read original article](<https://hackernoon.com/cdc-tools-with-the-most-predictable-pricing-for-high-volume-data-pipelines?source=rss>)

Author: Pipelineguy

Published: 2026-10-09T09:04:00Z

Content type: comparison

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [self-hosted databases vs managed services](<https://devfeed.tech/topics/self-hosted-databases-vs-managed-services.md>), [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [AWS Database Migration Service](<https://devfeed.tech/topics/aws-database-migration-service.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>)

Tags: [cdc](<https://devfeed.tech/tags/cdc.md>), [cdc-tools](<https://devfeed.tech/tags/cdc-tools.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [confluent](<https://devfeed.tech/tags/confluent.md>), [costs](<https://devfeed.tech/tags/costs.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [etl](<https://devfeed.tech/tags/etl.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [pricing](<https://devfeed.tech/tags/pricing.md>), [real-time-data-pipelines](<https://devfeed.tech/tags/real-time-data-pipelines.md>), [streaming-data](<https://devfeed.tech/tags/streaming-data.md>), [tools](<https://devfeed.tech/tags/tools.md>)

### AI overview

The article compares pricing models for change data capture tools including Estuary, Airbyte, Google Cloud Datastream, AWS DMS, Hevo, Fivetran, Confluent Cloud, and Debezium. It explains how workload measures such as data volume, changed rows or events, processing capacity, backfills, destinations, and infrastructure affect cost predictability. It recommends estimating costs with representative workloads and production-like tests, while noting that infrastructure and engineering effort add to total costs.

### Source excerpt

Compare CDC pricing models from Estuary, Airbyte, Datastream, AWS DMS, Hevo, Fivetran, Confluent, and Debezium, and learn how to forecast CDC costs at scale.

## How to Repoint dbt ETL Pipelines to Databricks

DevFeed: [How to Repoint dbt ETL Pipelines to Databricks](<https://devfeed.tech/articles/how-to-repoint-dbt-etl-pipelines-to-databricks-66821.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/how-repoint-dbt-etl-pipelines-databricks>)

Author: Khagay Nagdimov; Ismail Makhlouf; Shweta Verma

Published: 2026-10-06T18:53:48Z

Content type: tutorial

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [data-warehousing](<https://devfeed.tech/tags/data-warehousing.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [etl](<https://devfeed.tech/tags/etl.md>), [etl-pipelines](<https://devfeed.tech/tags/etl-pipelines.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>)

### AI overview

This tutorial explains how to move a dbt transformation project to Databricks by configuring the adapter and connection profile, mapping namespaces, and addressing SQL dialect differences. It recommends validating models against the existing warehouse, comparing outputs, then migrating in batches and cutting over after validation.

### Source excerpt

More teams are running their dbt transformations on Databricks Lakehouse: an open...

## Converting Snowflake Native Tables to Iceberg -- Part 1

DevFeed: [Converting Snowflake Native Tables to Iceberg -- Part 1](<https://devfeed.tech/articles/converting-snowflake-native-tables-to-iceberg-part-1-78486.md>)

Original publisher: [Read original article](<https://medium.com/snowflake/converting-snowflake-native-tables-to-iceberg-part-1-603fdb0fdcd1?source=rss----34b6daafc07---4>)

Author: Leo Pasta

Published: 2026-10-02T14:01:08Z

Content type: article

Language: en

Sources: [Snowflake Builders](<https://devfeed.tech/sources/snowflake-builders.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>)

Tags: [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [article](<https://devfeed.tech/tags/article.md>), [conversion](<https://devfeed.tech/tags/conversion.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-migration](<https://devfeed.tech/tags/data-migration.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [ecosystem](<https://devfeed.tech/tags/ecosystem.md>), [etl](<https://devfeed.tech/tags/etl.md>), [feature](<https://devfeed.tech/tags/feature.md>), [format](<https://devfeed.tech/tags/format.md>), [governance](<https://devfeed.tech/tags/governance.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [latency](<https://devfeed.tech/tags/latency.md>), [native](<https://devfeed.tech/tags/native.md>), [performance](<https://devfeed.tech/tags/performance.md>), [platform](<https://devfeed.tech/tags/platform.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [tables](<https://devfeed.tech/tags/tables.md>)

### AI overview

Iceberg conversion is most useful when specific datasets need access from multiple engines; converting every table can add complexity and cost without equivalent benefit. Start with a targeted, phased migration and inventory each table's consumers, engine versions, data types, write patterns, streams, and swap-based pipelines before choosing settings. Consumer capabilities constrain Iceberg version, types, partition pruning, codecs, and delete behavior. Resolve unsupported or lossy types such as TIMESTAMP_TZ, VECTOR, and some V2 conversions, and account for limitations involving defaults and collation for external engines. Choose CTAS for tables around 1-2 TB or with types that make Parquet staging risky; for larger compatible tables, Parquet export with COPY INTO ... LOAD_MODE=ADD_FILES_COPY may be faster and use fewer credits. Select partitioning or clustering based on consumer pruning and update workload, and choose storage and serialization settings according to cloud, access, key-management, and operational needs. Validate a representative sample before conversion; execution, validation, and cutover are reserved for Part 2.

### Source excerpt

Part 1 of 2 -- this article covers why you might convert to Iceberg, the decisions you need to make and how to approach them. Part 2 covers the conversion execution, validation and cutover. Why convert to Iceberg? Snowflake native tables are tightly integrated into the Snowflake platform: They offer the most complete support in areas such as performance optimisation, disaster recovery, replication and the full range of governance features. For workloads that live entirely within Snowflake, native tables are usually the right choice. But enterprise data environments are growing more and more sophisticated, so adopting an open table format like Iceberg can solve real challenges, like having multiple engines -- Apache Spark, Trino, Amazon Athena, Databricks, DuckDB and many others -- able to read and, in some configurations, write the same Iceberg tables without relying on Snowflake to process the query or on ETL jobs duplicating the data with the complexities and latency challenges it brings. Snowflake responded to this trend by offering what is widely regarded as the most complete support for the Apache Iceberg ecosystem, both as a manager and consumer of Iceberg catalogs. Conversion is not free of trade-offs though. The Iceberg ecosystem is also more complex to manage, with more options and configurations to consider and often more vendors and technologies involved. It is also less mature and well known than "traditional" Snowflake tables, which usually means a longer learning and adoption curve for engineers. The Apache Iceberg project is managed by a committee which, while fundamental for the extensive vendor support it has attracted, also means the adoption of new features and capabilities might happen later than with native tables. Over time, as the Iceberg format evolves and Snowflake features are released, the feature gap between native tables and Iceberg tables continues to narrow, but it is not zero yet. A few of those limitations will be discussed in the rest

[Next page](<https://devfeed.tech/tags/etl.md?cursor=WyIyMDI2LTEwLTAyVDE0OjAxOjA4KzAwOjAwIiwgImE0ODk2MjQ2LWM5ZjQtNGIwNS04MzcwLTM5OGU3OGFiYjk4NCJd>)