# Data pipelines

Data pipelines are series of processing steps that systematically collect, transform, and store data from various sources, including for analysis.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## ArrivalBench: Agent-Generated Data Pipelines Are Correct Once and Wrong Under Time

DevFeed: [ArrivalBench: Agent-Generated Data Pipelines Are Correct Once and Wrong Under Time](<https://devfeed.tech/articles/arrivalbench-agent-generated-data-pipelines-are-correct-once-and-wrong-under-time-144954.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.02363>)

Author: Pranay Kothari

Published: 2026-11-20T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [data observability](<https://devfeed.tech/topics/data-observability.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>)

Tags: [adversarial](<https://devfeed.tech/tags/adversarial.md>), [agent](<https://devfeed.tech/tags/agent.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [batch](<https://devfeed.tech/tags/batch.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [crash](<https://devfeed.tech/tags/crash.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [failure](<https://devfeed.tech/tags/failure.md>), [idempotency](<https://devfeed.tech/tags/idempotency.md>), [log](<https://devfeed.tech/tags/log.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [ordering](<https://devfeed.tech/tags/ordering.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [time](<https://devfeed.tech/tags/time.md>)

### AI overview

ArrivalBench tests agent-built data pipelines by replaying late, duplicated, out-of-order, and retried records, then comparing the final state with a batch recomputation of the full log. Unlike one-shot snapshot grading, this exposes silent wrong results separately from crashes. Across 40 tasks and eleven models, one-shot grading certified 86-100% of pipelines, while replay found 7.0-79.2% of those certified pipelines silently wrong. Idempotency hazards failed more often than ordering hazards in every model. In one model, a hazard warning reduced silent failures from 48.2% to 10.5%, but raised crashes from 9.0% to 37.0%; total failure fell only from 51.0% to 44.0%. Independently rerunning all eleven arms shifted rates by at most 5.9 percentage points.

### Source excerpt

arXiv:2610.02363v2 Announce Type: replace Abstract: Benchmarks for agent-generated data work grade a pipeline by running it once against a fixed snapshot. ArrivalBench instead re-executes the pipeline an agent leaves behind under adversarial but replayable delivery schedules (late, duplicated, out-of-order and retried records) and requires its final state to equal a batch recomputation of the complete log. Because the oracle recomputes rather than classifies, a wrong table and a crash are distinct verdicts: a crash is visible to monitoring a team already runs, and a wrong table is not. On 40 tasks we built, our reimplementation of single-execution grading certifies 86-100% of the pipelines eleven models produce; re-executing the same artifacts finds 7.0-79.2% of the certified ones silently wrong. The gap is not produced by the repair loop: within the same model and task, pipelines repaired against the snapshot test fail replay about as often as those that passed it first time. In every model, idempotency hazards fail more often than ordering hazards. Separating a wrong answer from a crash also changes how interventions read: a hazard warning cuts one model's silent failure from 48.2% to 10.5% while raising its crash rate from 9.0% to 37.0%, so all-in failure moves only from 51.0% to 44.0%. All eleven arms were independently re-run, and rates moved by at most 5.9 points.

## Continuous Prompts: LLM-Augmented Pipeline Processing over Unstructured Streams

DevFeed: [Continuous Prompts: LLM-Augmented Pipeline Processing over Unstructured Streams](<https://devfeed.tech/articles/continuous-prompts-llm-augmented-pipeline-processing-over-unstructured-streams-145730.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2512.03389>)

Author: Shu Chen, Deepti Raghavan, U\\u{g}ur \\c{C}etintemel

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [batching](<https://devfeed.tech/tags/batching.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-reasoning](<https://devfeed.tech/tags/llm-reasoning.md>), [long-running](<https://devfeed.tech/tags/long-running.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [operator](<https://devfeed.tech/tags/operator.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [probing](<https://devfeed.tech/tags/probing.md>), [queries](<https://devfeed.tech/tags/queries.md>), [rag](<https://devfeed.tech/tags/rag.md>), [stream](<https://devfeed.tech/tags/stream.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>), [streams](<https://devfeed.tech/tags/streams.md>)

### AI overview

Continuous Prompts (CPs) brings persistent LLM reasoning to stream processing by extending retrieval-augmented generation to streaming data and defining continuous semantic operators. Implemented in VectraFlow, it offers multiple implementations, mainly LLM-based plus an embedding-based variant. Tuple batching and operator fusion improve efficiency at the cost of accuracy. To manage that trade-off, CPs uses lightweight shadow executions and cost-aware multi-objective Bayesian optimization to learn throughput-accuracy frontiers and adapt plans within probing budgets. The abstract reports evaluations with operator microbenchmarks and real-dataset pipelines, finding that VectraFlow adapts to workload changes and sustains semantic queries over evolving unstructured streams.

### Source excerpt

arXiv:2512.03389v2 Announce Type: replace Abstract: Monitoring unstructured streams increasingly requires persistent, semantics-aware computation, yet today's LLM frameworks remain stateless and one-shot, limiting their usefulness for long-running analytics. We introduce Continuous Prompts (CPs), the first framework that brings LLM reasoning into continuous stream processing. CPs extend RAG to streaming settings, define continuous semantic operators, and provide multiple implementations, primarily focusing on LLM-based approaches but also reporting one embedding-based variants. Furthermore, we study two LLM-centric optimizations, tuple batching and operator fusion, to significantly improve efficiency while managing accuracy loss. Because these optimizations inherently trade accuracy for speed, we present a dynamic optimization framework that uses lightweight shadow executions and cost-aware multi-objective Bayesian optimization (MOBO) to learn throughput-accuracy frontiers and adapt plans under probing budgets. We implement CPs in the VectraFlow stream processing system. Using operator-level microbenchmarks and streaming pipelines on real datasets, we show that VectraFlow can adapt to workload dynamics, navigate accuracy-efficiency trade-offs, and sustain persistent semantic queries over evolving unstructured streams.

## Do You Still Need Airflow for Snowflake Pipelines? An Honest Answer From Someone Who's Run Both

DevFeed: [Do You Still Need Airflow for Snowflake Pipelines? An Honest Answer From Someone Who's Run Both](<https://devfeed.tech/articles/do-you-still-need-airflow-for-snowflake-pipelines-an-honest-answer-from-someone-who-s-run-both-145278.md>)

Original publisher: [Read original article](<https://medium.com/snowflake/do-you-still-need-airflow-for-snowflake-pipelines-an-honest-answer-from-someone-whos-run-both-0e8d4a5a9036?source=rss----34b6daafc07---4>)

Author: Sanket Prabhu

Published: 2026-11-20T01:56:59Z

Content type: opinion

Language: en

Sources: [Snowflake Builders](<https://devfeed.tech/sources/snowflake-builders.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [dag](<https://devfeed.tech/tags/dag.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [task-graph](<https://devfeed.tech/tags/task-graph.md>)

### AI overview

Snowflake task graphs may now cover many pipelines that previously needed Airflow, but the new graph object, atomic deployments, advanced branching, and DCM support are described as Preview features. The article says native graphs suit end-to-end Snowflake workloads owned by one team, with data arriving in Snowflake and a preference for centralized lineage, logs, cost, and access. Airflow remains useful for cross-system dependencies, routine date-range backfills, and a company-wide orchestration control plane; Preview risk may also argue against moving critical production pipelines. A suggested middle ground is to let Airflow handle external readiness and triggers, then execute one Snowflake task graph for retries, branching, notifications, and transformations. Regular tasks with AFTER dependencies are identified as generally available.

### Source excerpt

Snowflake task graphs just grew up. That doesn't mean your DAGs are obsolete. Apache Airflow and Snowflake Graph Every time Snowflake ships a new orchestration feature, I see the same post on my feed: "Airflow is dead." I don't buy it. But I also don't buy the opposite reflex, where every Snowflake pipeline gets an Airflow DAG by default because "that's how we do orchestration here." I've run Airflow in production and I've spent about a decade building data platforms on Snowflake. Over that time, my answer to "do we need Airflow for this?" has changed more than once. With the task graph changes Snowflake released in late October, it changed again. If every step of your pipeline already runs inside Snowflake, Airflow is often a second system you're paying to babysit. That's the short version. The longer version has some important exceptions. Status note up front: the new TASK GRAPH object, atomic deployments, advanced branching and DCM DEFINE TASK GRAPH are all in Preview as of this writing (released Oct 23 and Oct 27, 2026). Regular tasks with AFTER dependencies have been GA for a long time. 1. Why Airflow ended up in so many Snowflake stacks It's worth being fair to Airflow here. Most teams didn't choose it by accident. For years, native Snowflake tasks were fine for simple chains but awkward for real pipelines: Configuration was scattered across every task in the graph Changing a running graph meant suspending the root, editing tasks one by one, and resuming Branching was limited. A child ran when its predecessors succeeded, and handling a failure path cleanly took workarounds Deployment wasn't atomic. A half-applied change could leave the graph in a state nobody intended So teams reached for an orchestrator that already handled all of that. My team was using Airflow because of the operator versatility. Is that still the right call today? 2. What actually changed in October Two releases matter here. Task graphs as a first-class object (Preview, Oct 23). You can no

## How to Generate Synthetic Healthcare Data Using Synthea: A Hands On Tutorial

DevFeed: [How to Generate Synthetic Healthcare Data Using Synthea: A Hands On Tutorial](<https://devfeed.tech/articles/how-to-generate-synthetic-healthcare-data-using-synthea-a-hands-on-tutorial-144721.md>)

Original publisher: [Read original article](<https://www.freecodecamp.org/news/how-to-generate-synthetic-healthcare-data-using-synthea-a-hands-on-tutorial/>)

Author: Lakshmi Mahabaleshwara

Published: 2026-11-19T22:00:42Z

Content type: tutorial

Language: en

Sources: [Free Code Camp](<https://devfeed.tech/sources/freecodecamp-programming-tutorials-python-javascript-git-more.md>)

Topics: [Tutorial](<https://devfeed.tech/topics/tutorial.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Structured-data](<https://devfeed.tech/topics/structured-data.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-pipeline](<https://devfeed.tech/tags/analytics-pipeline.md>), [applications](<https://devfeed.tech/tags/applications.md>), [csv](<https://devfeed.tech/tags/csv.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [dataengineering](<https://devfeed.tech/tags/dataengineering.md>), [generate](<https://devfeed.tech/tags/generate.md>), [healthcare](<https://devfeed.tech/tags/healthcare.md>), [healthcareai](<https://devfeed.tech/tags/healthcareai.md>), [healthcaredata](<https://devfeed.tech/tags/healthcaredata.md>), [healthtech](<https://devfeed.tech/tags/healthtech.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [synthetichealthcaredata](<https://devfeed.tech/tags/synthetichealthcaredata.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>), [using](<https://devfeed.tech/tags/using.md>)

### AI overview

Synthea simulates longitudinal patient records, making it useful for prototyping and testing healthcare data workflows without using real patient histories. The tutorial shows how to build it with Java 17 or newer, generate about 100 patients, enable CSV export explicitly, and explore linked patient, encounter, condition, observation, medication, and procedure files with Python and Pandas. A fixed seed helps reproduce runs; version, configuration, and reference date also affect results. Default population overflow can produce more records than requested (115 in the described run); disabling it targets the requested count but may change how closely survivor demographics match population targets. The tutorial demonstrates basic relationship and missing-value checks and reconstructing a patient's timeline. Synthetic data is not necessarily representative, clinically accurate, unbiased, or suitable evidence for real-world model performance, so validate concepts, data quality, and representativeness before use.

### Source excerpt

Healthcare generates huge amounts of data every day, from patient visits and lab results to diagnoses, medications, and medical imaging. For developers and data engineers building healthcare applicati

## Locality-Aware Scheduling for Stream Processing Applications in Serverless Environments

DevFeed: [Locality-Aware Scheduling for Stream Processing Applications in Serverless Environments](<https://devfeed.tech/articles/locality-aware-scheduling-for-stream-processing-applications-in-serverless-environments-142973.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23879>)

Author: Tianyu Qi, Maria A. Rodriguez, Rajkumar Buyya

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [scheduling](<https://devfeed.tech/topics/scheduling.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>)

Tags: [balance](<https://devfeed.tech/tags/balance.md>), [chaining](<https://devfeed.tech/tags/chaining.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [function](<https://devfeed.tech/tags/function.md>), [function-as-a-service](<https://devfeed.tech/tags/function-as-a-service.md>), [i-o](<https://devfeed.tech/tags/i-o.md>), [latency](<https://devfeed.tech/tags/latency.md>), [local](<https://devfeed.tech/tags/local.md>), [locality](<https://devfeed.tech/tags/locality.md>), [monitors](<https://devfeed.tech/tags/monitors.md>), [network](<https://devfeed.tech/tags/network.md>), [operator](<https://devfeed.tech/tags/operator.md>), [operators](<https://devfeed.tech/tags/operators.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [reduce](<https://devfeed.tech/tags/reduce.md>), [routing](<https://devfeed.tech/tags/routing.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>)

### AI overview

LcSched targets a locality tradeoff in stateful serverless stream pipelines: keeping each operator's state with its execution can force chained invocations to cross workers, raising latency and I/O overhead. The distributed strategy aims to run chained invocations locally while ensuring they access state on the same worker, and monitors runtime workload to balance workers. The abstract reports up to 3.6x higher throughput and up to 79% lower end-to-end latency than the state-of-the-art distributed scheduling strategy for stateful serverless environments.

### Source excerpt

arXiv:2611.23879v1 Announce Type: new Abstract: Stateful Serverless (Function-as-a-Service) environment are increasingly used for stream processing applications. These applications are often structured as pipelines where data flows through a sequence of distinct processing operators. Many operators are stateful and require access to the processing state during execution. To address this, stateful serverless platforms introduce state servers on worker nodes, co-locating state with computation to dramatically reduce access latency. In serverless, these data pipelines are realized by chaining function invocations, passing the processed data from one function to the next. To reduce network overhead, state-of-the-art scheduling approaches employ a distributed worker-level scheduling model, routing locally generated chained requests directly to destination workers, bypassing centralized schedulers. However, these approaches always co-locate state and execution for the same operator in the same worker, which forces most invocations to the next operator to be transferred across workers, thereby increasing both latency and I/O overhead. As a result, this paper proposes LcSched, a novel distributed scheduling strategy tailored for stream processing applications in stateful serverless environments. LcSched seeks to maximise local execution of chained function invocations while ensuring that each execution accesses its state from the local worker. It also monitors workload at runtime to maintain workload balance between workers. Experimental results show that LcSched improves throughput by up to 3.6x and reduces end-to-end latency by up to 79% compared to the state-of-the-art distributed scheduling strategy for stateful serverless environments.

## ProvCards: Artifact-Oriented Provenance for Heterogeneous Scientific Workflows

DevFeed: [ProvCards: Artifact-Oriented Provenance for Heterogeneous Scientific Workflows](<https://devfeed.tech/articles/provcards-artifact-oriented-provenance-for-heterogeneous-scientific-workflows-142291.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24894>)

Author: Tanja Auge, Marinus Holitschke, Meike Klettke

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [RDF (Resource Description Framework)](<https://devfeed.tech/topics/rdf.md>)

Tags: [cs-db](<https://devfeed.tech/tags/cs-db.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [provenance](<https://devfeed.tech/tags/provenance.md>), [querying](<https://devfeed.tech/tags/querying.md>), [reproducibility](<https://devfeed.tech/tags/reproducibility.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

### AI overview

ProvCards captures workflow provenance as portable artifacts grounded in the W3C PROV Data Model. Components can independently create, exchange, reuse, and compose cards without relying on a particular workflow system or centralized execution context. ProvDecks group related cards while preserving their identities; derived ProvGraphs support reconstruction, querying, visualization, and analysis. A graph-based implementation evaluated in a scientific data-processing workflow demonstrated component-level capture, cross-component composition, and reconstruction. The evidence is limited to the abstract and does not specify comparative performance or broader evaluation results.

### Source excerpt

arXiv:2611.24894v1 Announce Type: new Abstract: Reproducibility in scientific workflows is hindered by heterogeneous components and execution environments, as well as fragmented provenance capture. Although existing approaches offer established semantics and workflow-specific solutions, provenance remains difficult to capture, exchange, and compose across workflow boundaries. We present ProvCards, an artifact-oriented approach that treats provenance as a first-class artifact grounded in the W3C PROV Data Model. Individual workflow components can generate cards that are stored, exchanged, reused, and composed independently of any workflow system or centralized execution context. ProvDecks group related cards into portable scopes while preserving their identities. ProvGraphs provide derived views for provenance reconstruction, querying, visualization, and analysis. Together, these representations support coherent, task-specific provenance views across heterogeneous components, facilitating traceability, reuse, and reproducibility. We implement ProvCards using a graph-based infrastructure and evaluate them in a scientific data-processing workflow. The results demonstrate independent provenance capture at the component level, composition across component boundaries, and reconstruction for querying and analysis. ProvCards provide a portable, composable, and incrementally extensible foundation for provenance management in heterogeneous scientific workflows.

## Architectural Design Decisions for Data Ingestion in Big Data Architectures

DevFeed: [Architectural Design Decisions for Data Ingestion in Big Data Architectures](<https://devfeed.tech/articles/architectural-design-decisions-for-data-ingestion-in-big-data-architectures-142161.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23297>)

Author: Yikang Huang, Stefano Fossati, Indika Kumara, Damian A. Tamburri, Willem-Jan van den Heuvel

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [data-architecture](<https://devfeed.tech/tags/data-architecture.md>), [data-engineers](<https://devfeed.tech/tags/data-engineers.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [data-ingestion-pipeline](<https://devfeed.tech/tags/data-ingestion-pipeline.md>), [data-ingestion-pipelines](<https://devfeed.tech/tags/data-ingestion-pipelines.md>), [data-lakes](<https://devfeed.tech/tags/data-lakes.md>), [decision](<https://devfeed.tech/tags/decision.md>), [design](<https://devfeed.tech/tags/design.md>), [drivers](<https://devfeed.tech/tags/drivers.md>), [experts](<https://devfeed.tech/tags/experts.md>), [framework](<https://devfeed.tech/tags/framework.md>), [functional](<https://devfeed.tech/tags/functional.md>), [options](<https://devfeed.tech/tags/options.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>)

### AI overview

The paper presents a framework to help data engineers and architects choose designs for data ingestion pipelines based on functional and non-functional requirements. A systematic review of 42 gray literature articles and qualitative analysis produced a catalog of 8 architectural decisions, 59 options, and 36 selection drivers. Semi-structured expert interviews assessed the framework's completeness, simplicity, understandability, operational feasibility, and usefulness. The supplied text reports that these qualities were evaluated, but does not give the interview findings or name the decisions and options.

### Source excerpt

arXiv:2611.23297v1 Announce Type: new Abstract: Introduction: Data ingestion is the first critical step in a data architecture, in which data from various sources is moved to storage systems, such as data lakes and data warehouses, for analysis and insight generation. When developing a data ingestion pipeline, practitioners (e.g., data engineers and architects) must understand the data ingestion requirements (functional and non-functional) and select appropriate design options to support them. Objective: To help practitioners systematically develop data ingestion processes, this paper aims to compile a catalog of data ingestion design decisions, their options, and the selection drivers. Method: We conducted a systematic gray literature review on data ingestion, analyzed the selected literature using qualitative data analysis methods, and conducted semi-structured interviews with experts to assess the framework's practicality and utility. Results: From 42 gray literature articles, we identified 8 architectural design decisions, 59 decision options, and 36 decision drivers to guide practitioners in the design and implementation of data ingestion pipelines. The framework was evaluated through expert interviews, assessing its completeness, simplicity, understandability, operational feasibility, and usefulness.

## Debezium 3.8.0.Alpha2 Released

DevFeed: [Debezium 3.8.0.Alpha2 Released](<https://devfeed.tech/articles/debezium-3-8-0-alpha2-released-143957.md>)

Original publisher: [Read original article](<https://debezium.io/blog/2026/11/19/debezium-3-8-alpha2-released/>)

Author: Chris Cranford

Published: 2026-11-19T00:00:00Z

Content type: release

Language: en

Sources: [Debezium](<https://devfeed.tech/sources/debezium-io.md>)

Topics: [Debezium Connector](<https://devfeed.tech/topics/debezium-connector.md>), [Release notes](<https://devfeed.tech/topics/release-notes.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Prometheus](<https://devfeed.tech/topics/prometheus.md>)

Tags: [3](<https://devfeed.tech/tags/3.md>), [announce](<https://devfeed.tech/tags/announce.md>), [bug](<https://devfeed.tech/tags/bug.md>), [bug-fixes](<https://devfeed.tech/tags/bug-fixes.md>), [capture](<https://devfeed.tech/tags/capture.md>), [change](<https://devfeed.tech/tags/change.md>), [changes](<https://devfeed.tech/tags/changes.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [debezium](<https://devfeed.tech/tags/debezium.md>), [deployments](<https://devfeed.tech/tags/deployments.md>), [details](<https://devfeed.tech/tags/details.md>), [faster](<https://devfeed.tech/tags/faster.md>), [formats](<https://devfeed.tech/tags/formats.md>), [full](<https://devfeed.tech/tags/full.md>), [message](<https://devfeed.tech/tags/message.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [more](<https://devfeed.tech/tags/more.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [platform](<https://devfeed.tech/tags/platform.md>)

### AI overview

Debezium 3.8.0.Alpha2 adds initial and initial-only snapshots to the TiDB connector, which capture a consistent view without locking tables before streaming continues. The Oracle connector changes snapshot locking: shared mode holds table locks while data is read, so most DDL waits; extended mode blocks all DDL and pauses writes, subject to `snapshot.lock.timeout.ms`. Remove the no-longer-recognized `snapshot.database.errors.max.retries` setting. For databases using extended strings, a recovery snapshot rebuilds schema history with byte-size data. Other notable changes include configurable pipeline message formats and secured Prometheus monitoring in Debezium Platform; Oracle LogMiner query performance nearly tripled in a 1,000-table benchmark; and MongoDB change-stream buffers gain event and byte limits. This is an alpha preview with Oracle upgrade considerations and many connector fixes.

### Source excerpt

With November in full swing, we're pleased to announce Debezium 3.8.0.Alpha2, the second preview release of the 3.8 series. This release brings initial snapshots to the TiDB connector. Debezium Platform gains configurable message formats for pipelines and monitoring that works with secured Prometheus deployments. Oracle LogMiner users get faster and more reliable change capture, and there are many bug fixes across the connectors. There are a few breaking changes for the Oracle connector to review before upgrading. Read on for the full details!

## Orchestrate DuckDB analytics pipelines with Amazon MWAA

DevFeed: [Orchestrate DuckDB analytics pipelines with Amazon MWAA](<https://devfeed.tech/articles/orchestrate-duckdb-analytics-pipelines-with-amazon-mwaa-141881.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/orchestrate-duckdb-analytics-pipelines-with-amazon-mwaa/>)

Author: Vinod Jayendra

Published: 2026-11-18T20:57:09Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [parquet](<https://devfeed.tech/topics/parquet.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-managed-workflows-for-apache-airflow-amazon-mwaa](<https://devfeed.tech/tags/amazon-managed-workflows-for-apache-airflow-amazon-mwaa.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [data](<https://devfeed.tech/tags/data.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [effective](<https://devfeed.tech/tags/effective.md>), [embedded](<https://devfeed.tech/tags/embedded.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [learn-how-to](<https://devfeed.tech/tags/learn-how-to.md>), [model](<https://devfeed.tech/tags/model.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [retry](<https://devfeed.tech/tags/retry.md>), [s3](<https://devfeed.tech/tags/s3.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Amazon MWAA can orchestrate bounded SQL analytics by running DuckDB inside Airflow worker tasks, without a separate query cluster. The walkthrough reads S3 Parquet files, deduplicates events, uses ASOF LEFT JOINs to apply rates and ownership effective at each event time, and merges results into three Iceberg tables in Amazon S3 Tables. A second task validates counts, uniqueness, and allocation states. On supported MWAA Provisioned environments running Airflow 3.3.1, DuckDB and its providers plus the duckdb_default AWS-aware connection are preinstalled; the execution role supplies temporary credentials. The example's unchanged-input reruns retained 300,000 unique events and duplicate-free summaries, but the excerpt says this does not establish recovery from partial failures or concurrent external writes. Each task shares one worker's CPU, memory, and storage, so limits and concurrency need tuning with representative data. Use Glue or EMR for steps needing horizontal scaling or more capacity than one worker provides. Three table commits are not one transaction, and Iceberg update/delete support can vary by version.

### Source excerpt

Learn how to orchestrate embedded DuckDB analytics pipelines with Amazon MWAA. This post shows how to run DuckDB inside Airflow tasks to read Parquet data from Amazon S3, apply time-effective model rates, and publish Apache Iceberg tables to Amazon S3 Tables, with retry-safe publishing and worker-sizing considerations.

## India's Humyn Labs raises $47M to feed U.S. robotics companies the data they need

DevFeed: [India's Humyn Labs raises $47M to feed U.S. robotics companies the data they need](<https://devfeed.tech/articles/india-s-humyn-labs-raises-47m-to-feed-u-s-robotics-companies-the-data-they-need-140457.md>)

Original publisher: [Read original article](<https://techcrunch.com/2026/11/18/accel-backs-indias-humyn-labs-to-supply-training-data-needed-by-u-s-robotics-ai-developers/>)

Author: Jagmeet Singh

Published: 2026-11-18T11:30:00Z

Content type: news

Language: en

Sources: [TechCrunch | AI News & Artificial Intelligence](<https://devfeed.tech/sources/techcrunch-ai-news-artificial-intelligence.md>)

Topics: [Physical AI](<https://devfeed.tech/topics/physical-ai.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [recording](<https://devfeed.tech/topics/recording.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>)

Tags: [accel](<https://devfeed.tech/tags/accel.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-labs](<https://devfeed.tech/tags/ai-labs.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [asia](<https://devfeed.tech/tags/asia.md>), [camera](<https://devfeed.tech/tags/camera.md>), [co-founder](<https://devfeed.tech/tags/co-founder.md>), [collect](<https://devfeed.tech/tags/collect.md>), [collecting](<https://devfeed.tech/tags/collecting.md>), [consent](<https://devfeed.tech/tags/consent.md>), [contracts](<https://devfeed.tech/tags/contracts.md>), [contributors](<https://devfeed.tech/tags/contributors.md>), [customers](<https://devfeed.tech/tags/customers.md>), [data](<https://devfeed.tech/tags/data.md>), [data-collection](<https://devfeed.tech/tags/data-collection.md>), [humyn-labs](<https://devfeed.tech/tags/humyn-labs.md>), [india](<https://devfeed.tech/tags/india.md>), [revenue](<https://devfeed.tech/tags/revenue.md>), [robotics](<https://devfeed.tech/tags/robotics.md>), [startups](<https://devfeed.tech/tags/startups.md>)

### AI overview

Humyn Labs raised $47 million to supply robotics and AI companies with real-world training data: $26 million in equity led by Accel and $21 million in venture debt led by Lighthouse Canton. Its two-camera hardware records people doing everyday tasks; software tracks hand movements and object interactions and labels footage for model training. The company collects data across more than 20 countries, using local partners to identify sites and supervise collection. It reports 12 paying customers and says about 90% of revenue comes from the U.S. The debt helps cover a cash-flow gap: workers and partners are paid daily or weekly, while customer payments typically arrive in 30 to 60 days. The excerpt notes privacy and consent concerns around recording people at home; Humyn says it records usual work or chores rather than staged activities.

### Source excerpt

Founded in February, Humyn Labs competes with startups including Mecka AI and XDOF, with about 90% of its revenue coming from the U.S.

## Exclusive: Wherobots AI agents automate complex spatial data engineering

DevFeed: [Exclusive: Wherobots AI agents automate complex spatial data engineering](<https://devfeed.tech/articles/exclusive-wherobots-ai-agents-automate-complex-spatial-data-engineering-140316.md>)

Original publisher: [Read original article](<https://siliconangle.com/2026/11/18/exclusive-wherobots-ai-agents-automate-complex-spatial-data-engineering/>)

Author: Paul Gillin

Published: 2026-11-18T11:00:42Z

Content type: news

Language: en

Sources: [SiliconANGLE](<https://devfeed.tech/sources/siliconangle.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [geospatial](<https://devfeed.tech/topics/geospatial.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Geographic Information System](<https://devfeed.tech/topics/gis.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-sedona](<https://devfeed.tech/tags/apache-sedona.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [geographic-visualizations](<https://devfeed.tech/tags/geographic-visualizations.md>), [geospatial-data](<https://devfeed.tech/tags/geospatial-data.md>), [global-marketing-insights-inc](<https://devfeed.tech/tags/global-marketing-insights-inc.md>), [news](<https://devfeed.tech/tags/news.md>), [spatial](<https://devfeed.tech/tags/spatial.md>), [spatial-analytics](<https://devfeed.tech/tags/spatial-analytics.md>), [spatial-data-catalog](<https://devfeed.tech/tags/spatial-data-catalog.md>), [the-latest](<https://devfeed.tech/tags/the-latest.md>), [top-story-5](<https://devfeed.tech/tags/top-story-5.md>), [wherobots-havasu](<https://devfeed.tech/tags/wherobots-havasu.md>), [wherobots-inc](<https://devfeed.tech/tags/wherobots-inc.md>), [wherobots-rasterflow](<https://devfeed.tech/tags/wherobots-rasterflow.md>), [wherobots-waldo](<https://devfeed.tech/tags/wherobots-waldo.md>), [wherobotsdb](<https://devfeed.tech/tags/wherobotsdb.md>)

### AI overview

Wherobots' Waldo agent uses natural-language instructions to build spatial data pipelines, coordinating data discovery, query generation, analysis, visualization, and scheduled refreshes. Its platform combines Havasu for dataset discovery and preparation, RasterFlow for imagery processing, WherobotsDB with more than 300 spatial functions, and Wherobots Maps. Wherobots says it connects existing LLMs to specialized processing engines and can use existing cloud storage and catalogs such as AWS Glue and Databricks Unity Catalog, without requiring data migration. Users can inspect generated queries and processing steps, but the company cautions that prompts do not always produce accurate results. Waldo is in public preview; agent tokens are included at no extra charge in Professional Edition, while customers primarily pay for computing resources.

### Source excerpt

Cloud-native spatial analytics and artificial intelligence startup Wherobots Inc. today introduced an artificial intelligence agent that enables users to build and deploy spatial data pipelines using natural-language instructions, reducing the need for specialized programming expertise. Called Waldo, the agent can discover data sources, generate queries, perform analysis, visualize results and schedule recurring operations. It's designed [...] The post Exclusive: Wherobots AI agents automate complex spatial data engineering appeared first on SiliconANGLE.

## Simplify Brand Awareness Analysis With Fivetran Data Models | Blog | Fivetran

DevFeed: [Simplify Brand Awareness Analysis With Fivetran Data Models | Blog | Fivetran](<https://devfeed.tech/articles/simplify-brand-awareness-analysis-with-fivetran-data-models-blog-fivetran-141338.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/simplify-brand-awareness-analysis-with-fivetran-data-models>)

Author: Joe Markiewicz

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [etl](<https://devfeed.tech/topics/etl.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Instagram](<https://devfeed.tech/topics/instagram.md>)

Tags: [advertising](<https://devfeed.tech/tags/advertising.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [awareness](<https://devfeed.tech/tags/awareness.md>), [brand](<https://devfeed.tech/tags/brand.md>), [brand-awareness](<https://devfeed.tech/tags/brand-awareness.md>), [data](<https://devfeed.tech/tags/data.md>), [engagement](<https://devfeed.tech/tags/engagement.md>), [engagement-metrics](<https://devfeed.tech/tags/engagement-metrics.md>), [facebook](<https://devfeed.tech/tags/facebook.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [instagram](<https://devfeed.tech/tags/instagram.md>), [linkedin](<https://devfeed.tech/tags/linkedin.md>), [marketing-analytics](<https://devfeed.tech/tags/marketing-analytics.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [models](<https://devfeed.tech/tags/models.md>), [twitter](<https://devfeed.tech/tags/twitter.md>)

### AI overview

Fivetran's social media data model combines cleaned, tested, normalized engagement data from Facebook Pages, Instagram Business, Organic Twitter, and LinkedIn Company Pages into one schema and table. It rolls up clicks, impressions, shares, likes, and comments, supports combining multiple accounts on a connector, and helps compare engagement across profiles. Used alongside Fivetran's separate ad reporting model--which aggregates spend, clicks, and impressions across Facebook, Google, Pinterest, Bing, Twitter, and LinkedIn--analysts can compare daily reach and engagement. Both models use shared transformation logic maintained by Fivetran analytics engineers and community contributors.

### Source excerpt

Easily combine engagement metrics from Facebook, Instagram, Twitter and LinkedIn.

## Perform rapid analysis of your app platform data | Blog | Fivetran

DevFeed: [Perform rapid analysis of your app platform data | Blog | Fivetran](<https://devfeed.tech/articles/perform-rapid-analysis-of-your-app-platform-data-blog-fivetran-141335.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/perform-rapid-analysis-of-your-app-platform-data>)

Author: Jamie Rodriguez

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Google Play](<https://devfeed.tech/topics/google-play.md>), [etl](<https://devfeed.tech/topics/etl.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>)

Tags: [2022](<https://devfeed.tech/tags/2022.md>), [about](<https://devfeed.tech/tags/about.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [app](<https://devfeed.tech/tags/app.md>), [app-store](<https://devfeed.tech/tags/app-store.md>), [apple](<https://devfeed.tech/tags/apple.md>), [apple-app-store](<https://devfeed.tech/tags/apple-app-store.md>), [can](<https://devfeed.tech/tags/can.md>), [data](<https://devfeed.tech/tags/data.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [google](<https://devfeed.tech/tags/google.md>), [google-play](<https://devfeed.tech/tags/google-play.md>), [model](<https://devfeed.tech/tags/model.md>), [platform](<https://devfeed.tech/tags/platform.md>), [play](<https://devfeed.tech/tags/play.md>), [rapid](<https://devfeed.tech/tags/rapid.md>), [reporting](<https://devfeed.tech/tags/reporting.md>), [store](<https://devfeed.tech/tags/store.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Fivetran's App Reporting data models standardize Google Play and Apple App Store connector data into a unified schema, aggregating app activity daily by app and across country, OS version, app version, traffic source, and device. The models are intended to reduce the need for teams to learn and join numerous normalized source tables. They include a data dictionary and model lineage graph. With Fivetran Transformations for dbt Core, integrated scheduling (available as of February 2022) runs the reporting model after connector syncs complete, so the resulting models use the latest synced data.

### Source excerpt

With Fivetran's new App Reporting data model, you can easily roll your Apple App Store and Google Play data into a unified schema for seamless reporting.

## Best of 2021: Fivetran's Year in Review | Blog | Fivetran

DevFeed: [Best of 2021: Fivetran's Year in Review | Blog | Fivetran](<https://devfeed.tech/articles/best-of-2021-fivetran-s-year-in-review-blog-fivetran-141301.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/best-of-2021-fivetrans-year-in-review>)

Author: Cat Origitano

Published: 2026-11-18T06:13:15Z

Content type: news

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>), [Security](<https://devfeed.tech/topics/security.md>)

Tags: [2021](<https://devfeed.tech/tags/2021.md>), [access-control](<https://devfeed.tech/tags/access-control.md>), [acquisition](<https://devfeed.tech/tags/acquisition.md>), [acquisitions](<https://devfeed.tech/tags/acquisitions.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [alerts](<https://devfeed.tech/tags/alerts.md>), [best-of](<https://devfeed.tech/tags/best-of.md>), [blog](<https://devfeed.tech/tags/blog.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [data](<https://devfeed.tech/tags/data.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [integrations](<https://devfeed.tech/tags/integrations.md>), [models](<https://devfeed.tech/tags/models.md>), [partner](<https://devfeed.tech/tags/partner.md>), [partner-integrations](<https://devfeed.tech/tags/partner-integrations.md>), [review](<https://devfeed.tech/tags/review.md>), [security](<https://devfeed.tech/tags/security.md>), [security-and-compliance](<https://devfeed.tech/tags/security-and-compliance.md>), [year-in-review](<https://devfeed.tech/tags/year-in-review.md>)

### AI overview

Fivetran reports that in 2021 it added 40 connectors, bringing its total to more than 180, and launched 14 prebuilt data models. It also introduced integrated scheduling with dbt Core; Business Critical features including AWS or Azure PrivateLink, customer-managed keys, cloud and region selection, and PCI DSS Level 1 qualification; and governance tools such as granular RBAC, Okta and AzureAD user provisioning, and column masking across all connectors. The company says it averaged over 99.9% pipeline uptime and handled more than one million daily syncs. Other changes included integrations with Airflow, Terraform, Prefect, and Dagster, expanded cloud partnerships, interface and documentation updates, and the HVR and Teleport Data acquisitions. The article presents a company year-in-review, not independent evaluation of these claims.

### Source excerpt

Dozens of new connectors, partner integrations, security and compliance enhancements -- and the Fivetran data models you love.

## How to Build a Modern Data Stack From the Ground Up | Blog | Fivetran

DevFeed: [How to Build a Modern Data Stack From the Ground Up | Blog | Fivetran](<https://devfeed.tech/articles/how-to-build-a-modern-data-stack-from-the-ground-up-blog-fivetran-141324.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/how-to-build-a-modern-data-stack-from-the-ground-up>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: tutorial

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [elt](<https://devfeed.tech/topics/elt.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [Data serialization](<https://devfeed.tech/topics/data-serialization.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [bi-tools](<https://devfeed.tech/tags/bi-tools.md>), [business-intelligence](<https://devfeed.tech/tags/business-intelligence.md>), [business-logic](<https://devfeed.tech/tags/business-logic.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [data](<https://devfeed.tech/tags/data.md>), [data-driven](<https://devfeed.tech/tags/data-driven.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-platforms](<https://devfeed.tech/tags/data-platforms.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [data-stack](<https://devfeed.tech/tags/data-stack.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [decision-making](<https://devfeed.tech/tags/decision-making.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [modern-data-stack](<https://devfeed.tech/tags/modern-data-stack.md>), [plan](<https://devfeed.tech/tags/plan.md>)

### AI overview

Build the stack around a defined analytics use case: identify the data needed, then choose an ingestion tool that connects to those sources. Centralize data in a warehouse and consider a data lake for historical data; use transformation tools such as dbt with connectors such as Fivetran to prepare data for reporting. Keep business logic consistent and models representative of the real world. Choose a BI tool based on the tradeoff that matters most, such as usability or scalability, and consult the technical team to maintain a cohesive strategy. The article emphasizes data quality throughout: validate at ingestion and transformation stages, test continuously, and promote to production only when confident the data will not mislead users.

### Source excerpt

A data architecture expert gives a crash course on moving analytics to the cloud -- including a step-by-step plan to ensure success.

## Cookies Are Out, Conversions APIs Are In | Blog | Fivetran

DevFeed: [Cookies Are Out, Conversions APIs Are In | Blog | Fivetran](<https://devfeed.tech/articles/cookies-are-out-conversions-apis-are-in-blog-fivetran-141304.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/cookies-are-out-conversions-apis-are-in>)

Author: Lee Hammond

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [API Governance](<https://devfeed.tech/topics/api-governance.md>), [Cookies](<https://devfeed.tech/topics/cookies.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2022](<https://devfeed.tech/tags/2022.md>), [actions](<https://devfeed.tech/tags/actions.md>), [ad-performance](<https://devfeed.tech/tags/ad-performance.md>), [added](<https://devfeed.tech/tags/added.md>), [ads](<https://devfeed.tech/tags/ads.md>), [advertising](<https://devfeed.tech/tags/advertising.md>), [advice](<https://devfeed.tech/tags/advice.md>), [apis](<https://devfeed.tech/tags/apis.md>), [blog](<https://devfeed.tech/tags/blog.md>), [change](<https://devfeed.tech/tags/change.md>), [cookies](<https://devfeed.tech/tags/cookies.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [media](<https://devfeed.tech/tags/media.md>), [regulations](<https://devfeed.tech/tags/regulations.md>), [roi](<https://devfeed.tech/tags/roi.md>), [social](<https://devfeed.tech/tags/social.md>), [social-media](<https://devfeed.tech/tags/social-media.md>), [technology](<https://devfeed.tech/tags/technology.md>)

### AI overview

Server-side conversions APIs can capture conversion events more reliably than browser pixels, which are limited by privacy changes, ad blockers and browser restrictions. They also accept events from offline and other parts of the customer journey, but take more engineering effort to set up. The article recommends centralizing events and customer data from business systems in a data warehouse, then using reverse ETL to send modeled data to advertising platforms. It says a fuller set of customer information can improve event matching and attribution, while direct integrations across separate systems risk brittle pipelines, incomplete data and lower match rates. Fivetran is presented for loading source data into the warehouse and Hightouch for syncing warehouse models to APIs such as Meta and Google; the article does not compare these products with alternatives.

### Source excerpt

How to improve ROI on social media ads as regulations and technology change.

## Fivetran now available in Microsoft Azure Marketplace | Blog | Fivetran

DevFeed: [Fivetran now available in Microsoft Azure Marketplace | Blog | Fivetran](<https://devfeed.tech/articles/fivetran-now-available-in-microsoft-azure-marketplace-blog-fivetran-141315.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/fivetran-now-available-in-microsoft-azure-marketplace>)

Author: Logan Welley

Published: 2026-11-18T06:13:15Z

Content type: release

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [azure marketplace](<https://devfeed.tech/topics/azure-marketplace.md>), [Azure](<https://devfeed.tech/topics/azure.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [applications](<https://devfeed.tech/tags/applications.md>), [automated](<https://devfeed.tech/tags/automated.md>), [azure](<https://devfeed.tech/tags/azure.md>), [azure-databricks](<https://devfeed.tech/tags/azure-databricks.md>), [azure-marketplace](<https://devfeed.tech/tags/azure-marketplace.md>), [azure-synapse](<https://devfeed.tech/tags/azure-synapse.md>), [business](<https://devfeed.tech/tags/business.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [data-engineers](<https://devfeed.tech/tags/data-engineers.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-teams](<https://devfeed.tech/tags/data-teams.md>), [databases](<https://devfeed.tech/tags/databases.md>), [deploy](<https://devfeed.tech/tags/deploy.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [managed](<https://devfeed.tech/tags/managed.md>), [marketplace](<https://devfeed.tech/tags/marketplace.md>), [microsoft](<https://devfeed.tech/tags/microsoft.md>), [microsoft-azure](<https://devfeed.tech/tags/microsoft-azure.md>)

### AI overview

Fivetran Data Pipelines is available through the Microsoft Azure Marketplace, where Microsoft customers can discover, try, and deploy solutions certified and optimized for Azure. The fully managed pipelines automate data extraction and loading into cloud warehouses; Fivetran says its 180+ connectors centralize data from SaaS applications and databases into destinations including Azure Synapse, Azure Databricks, and Snowflake. Marketplace access is presented as simplifying procurement, helping customers get more value from Azure, and speeding integration with supported sources. Fivetran Business Critical is also available for enterprises with sensitive data and strict compliance requirements.

### Source excerpt

Microsoft customers can discover, transact and deploy Fivetran's fully managed and automated data pipelines in Azure and more.

## The Data Integration Revolution: A Solution Architect's Perspective | Blog | Fivetran

DevFeed: [The Data Integration Revolution: A Solution Architect's Perspective | Blog | Fivetran](<https://devfeed.tech/articles/the-data-integration-revolution-a-solution-architect-s-perspective-blog-fivetran-141341.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/the-data-integration-revolution-a-solution-architects-perspective>)

Author: Dom Orsini

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [elt](<https://devfeed.tech/topics/elt.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>), [data migrations](<https://devfeed.tech/topics/data-migrations.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>), [Tool](<https://devfeed.tech/topics/tool.md>)

Tags: [architect](<https://devfeed.tech/tags/architect.md>), [batch](<https://devfeed.tech/tags/batch.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloudera](<https://devfeed.tech/tags/cloudera.md>), [coding](<https://devfeed.tech/tags/coding.md>), [collaboration](<https://devfeed.tech/tags/collaboration.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [data](<https://devfeed.tech/tags/data.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [data-migration](<https://devfeed.tech/tags/data-migration.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-teams](<https://devfeed.tech/tags/data-teams.md>), [databases](<https://devfeed.tech/tags/databases.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [elt](<https://devfeed.tech/tags/elt.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [etl](<https://devfeed.tech/tags/etl.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [solution-architect](<https://devfeed.tech/tags/solution-architect.md>)

### AI overview

The article's central finding is that managed SaaS data integration with ELT and cloud data warehouses has made data delivery faster and less engineering-intensive, freeing data teams to focus on business use. It contrasts this with on-premise ETL, which required manual coding, hardware tuning, and close operational support, and big data/IaaS systems, which sped delivery but added interdependent services, maintenance burdens, sync bottlenecks, and technology churn. In the modern approach described, fully managed pipelines need minimal configuration, while analysts build SQL transformations; dbt adds version control and collaboration practices. The author says this reduces coordination and schema-change work, though the piece presents a solution architect's perspective and specifically promotes Fivetran.

### Source excerpt

Life for data professionals keeps getting better -- and more interesting.

## How data-driven strategies can reverse the Great Resignation | Blog | Fivetran

DevFeed: [How data-driven strategies can reverse the Great Resignation | Blog | Fivetran](<https://devfeed.tech/articles/how-data-driven-strategies-can-reverse-the-great-resignation-blog-fivetran-141322.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/how-data-driven-strategies-can-reverse-the-great-resignation>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [business-intelligence](<https://devfeed.tech/topics/business-intelligence.md>), [Data visualization](<https://devfeed.tech/topics/data-visualization.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [ats](<https://devfeed.tech/tags/ats.md>), [bi](<https://devfeed.tech/tags/bi.md>), [business-intelligence](<https://devfeed.tech/tags/business-intelligence.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-based](<https://devfeed.tech/tags/cloud-based.md>), [cloud-data-platforms](<https://devfeed.tech/tags/cloud-data-platforms.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [data-analytics](<https://devfeed.tech/tags/data-analytics.md>), [data-driven](<https://devfeed.tech/tags/data-driven.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-stack](<https://devfeed.tech/tags/data-stack.md>), [diversity](<https://devfeed.tech/tags/diversity.md>), [diversity-and-inclusion](<https://devfeed.tech/tags/diversity-and-inclusion.md>), [employee](<https://devfeed.tech/tags/employee.md>), [pain](<https://devfeed.tech/tags/pain.md>), [people](<https://devfeed.tech/tags/people.md>), [talent](<https://devfeed.tech/tags/talent.md>), [top](<https://devfeed.tech/tags/top.md>)

### AI overview

People analytics can help HR identify employee pain points and intervene to improve retention, though the article presents these approaches as opportunities rather than guaranteed results. It recommends analyzing compensation and benefits against relevant benchmarks, tracking diversity through recruiting and tenure data, and using employee relationship maps to spot weak connections. Sentiment analysis, including automated staff surveys, can help identify employees who may leave so HR can respond proactively. These methods depend on integrating HR data sources such as HRIS and making the data available for analysis; the excerpt also describes ATS, dashboards, data warehouses and BI software as parts of the HR analytics landscape.

### Source excerpt

People analytics can help you understand employee pain points and take steps to retain top talent.

## Three Must-Haves for Your Modern Retail Data Stack | Blog | Fivetran

DevFeed: [Three Must-Haves for Your Modern Retail Data Stack | Blog | Fivetran](<https://devfeed.tech/articles/three-must-haves-for-your-modern-retail-data-stack-blog-fivetran-141344.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/three-must-haves-for-modern-retail-data-stack>)

Author: Raman Kaler

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [3](<https://devfeed.tech/tags/3.md>), [act](<https://devfeed.tech/tags/act.md>), [all](<https://devfeed.tech/tags/all.md>), [allows](<https://devfeed.tech/tags/allows.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [application](<https://devfeed.tech/tags/application.md>), [application-development](<https://devfeed.tech/tags/application-development.md>), [blog](<https://devfeed.tech/tags/blog.md>), [culture](<https://devfeed.tech/tags/culture.md>), [customer](<https://devfeed.tech/tags/customer.md>), [data](<https://devfeed.tech/tags/data.md>), [data-stack](<https://devfeed.tech/tags/data-stack.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [invest](<https://devfeed.tech/tags/invest.md>), [modern](<https://devfeed.tech/tags/modern.md>), [quickly](<https://devfeed.tech/tags/quickly.md>), [retail](<https://devfeed.tech/tags/retail.md>), [retailers](<https://devfeed.tech/tags/retailers.md>), [stack](<https://devfeed.tech/tags/stack.md>)

### AI overview

A modern retail data stack combines automated connectors, a cloud warehouse and visual analytics to make current information available for faster retail decisions. The article describes Fivetran connectors as syncing data from applications, databases, events and files, adapting to source changes and requiring no maintenance; Snowflake as providing governed access to centralized data; and Tableau as helping users explore performance. It cites a dashboard using more than 500 million transactions and 3 billion inventory records. Effective use also depends on data access, transparency and literacy, supported by coordination and a plan. Retail data needs vary by sector, including personalization, inventory and supply-chain optimization, consumer trend analysis, and ecommerce demand forecasting.

### Source excerpt

When retailers invest in a culture of data, they gain the ability to quickly navigate shifting customer priorities.

## How to Develop a Data-Driven Marketing Culture | Blog | Fivetran

DevFeed: [How to Develop a Data-Driven Marketing Culture | Blog | Fivetran](<https://devfeed.tech/articles/how-to-develop-a-data-driven-marketing-culture-blog-fivetran-141326.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/how-to-develop-a-data-driven-marketing-culture>)

Author: Cat Origitano

Published: 2026-11-18T06:13:15Z

Content type: tutorial

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [business-intelligence](<https://devfeed.tech/topics/business-intelligence.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>)

Tags: [2022](<https://devfeed.tech/tags/2022.md>), [actions](<https://devfeed.tech/tags/actions.md>), [ads](<https://devfeed.tech/tags/ads.md>), [advertising](<https://devfeed.tech/tags/advertising.md>), [agile](<https://devfeed.tech/tags/agile.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analysts](<https://devfeed.tech/tags/analysts.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [blog](<https://devfeed.tech/tags/blog.md>), [culture](<https://devfeed.tech/tags/culture.md>), [data](<https://devfeed.tech/tags/data.md>), [data-driven](<https://devfeed.tech/tags/data-driven.md>), [develop](<https://devfeed.tech/tags/develop.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [help](<https://devfeed.tech/tags/help.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [marketing](<https://devfeed.tech/tags/marketing.md>), [skills](<https://devfeed.tech/tags/skills.md>), [team](<https://devfeed.tech/tags/team.md>)

### AI overview

The article argues that data-driven marketing requires organizational change alongside analytics tools. As third-party cookies disappear and first-party data remains scattered across teams, it recommends setting measurable goals for data culture, securing leadership support, holding teams accountable for evidence-based decisions, and sharing lessons from wins and failures. Practical steps include reducing silos through embedded or rotating analysts, assessing marketers' skills and planning training, and ensuring analytics cover six source categories: email, CRM, paid digital advertising, organic social and digital, conversions, and website or engagement analytics. For cross-platform analysis, it recommends a modern data stack: source platforms, an automated pipeline such as Fivetran, a cloud warehouse such as Snowflake, and a BI tool such as Looker or Tableau. This can replace error-prone, stale manual spreadsheet merging, though the article presents centralization as a foundation rather than a guarantee of effective decisions.

### Source excerpt

Help your team develop the appropriate skills to win in the age of data-driven marketing.

## From Months to Days: How Autodesk Reinvented Its Data Ingestion Strategy | Blog | Fivetran

DevFeed: [From Months to Days: How Autodesk Reinvented Its Data Ingestion Strategy | Blog | Fivetran](<https://devfeed.tech/articles/from-months-to-days-how-autodesk-reinvented-its-data-ingestion-strategy-blog-fivetran-141321.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/how-autodesk-reinvented-its-data-ingestion-strategy>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [elt](<https://devfeed.tech/topics/elt.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [copy](<https://devfeed.tech/topics/copy.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [autodesk](<https://devfeed.tech/tags/autodesk.md>), [aws](<https://devfeed.tech/tags/aws.md>), [data](<https://devfeed.tech/tags/data.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-platform](<https://devfeed.tech/tags/data-platform.md>), [data-stack](<https://devfeed.tech/tags/data-stack.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [join](<https://devfeed.tech/tags/join.md>), [keep](<https://devfeed.tech/tags/keep.md>), [kinesis](<https://devfeed.tech/tags/kinesis.md>), [learning](<https://devfeed.tech/tags/learning.md>), [looker](<https://devfeed.tech/tags/looker.md>), [process](<https://devfeed.tech/tags/process.md>), [strategy](<https://devfeed.tech/tags/strategy.md>)

### AI overview

Autodesk replaced a patchwork of ingestion tools and custom scripts with two defined routes: Fivetran ingests structured sources into Snowflake, while AWS Kinesis sends high-volume unstructured product usage data to S3. Data moves between the repositories when needed; usage data can be promoted to Snowflake for analytics subject to privacy controls, and Snowflake snapshots are stored in S3 for history and machine learning. Autodesk's stated goal was to cut onboarding a dataset, including requirements, vendor setup, and security and privacy controls, from six months to six business days. The change reportedly freed engineers to focus on business problems and reduced urgent pipeline-breakage emails. The account attributes the approach to Autodesk's buy-over-build, simplicity, speed-to-impact, and privacy principles.

### Source excerpt

By replacing homemade ETL with automated ELT, Autodesk cut its data ingestion process from six months to six days.

## ETL Platform Alternatives: Fivetran vs. Informatica and More | Blog | Fivetran

DevFeed: [ETL Platform Alternatives: Fivetran vs. Informatica and More | Blog | Fivetran](<https://devfeed.tech/articles/etl-platform-alternatives-fivetran-vs-informatica-and-more-blog-fivetran-141305.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/etl-platform-alternatives-fivetran-vs-informatica-and-more>)

Author: Charles Wang

Published: 2026-11-18T06:13:15Z

Content type: comparison

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [etl](<https://devfeed.tech/topics/etl.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>)

Tags: [alternatives](<https://devfeed.tech/tags/alternatives.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [business-intelligence](<https://devfeed.tech/tags/business-intelligence.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [cloud-native](<https://devfeed.tech/tags/cloud-native.md>), [compare](<https://devfeed.tech/tags/compare.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [etl](<https://devfeed.tech/tags/etl.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [informatica](<https://devfeed.tech/tags/informatica.md>), [integration](<https://devfeed.tech/tags/integration.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [security](<https://devfeed.tech/tags/security.md>), [sources](<https://devfeed.tech/tags/sources.md>), [tools](<https://devfeed.tech/tags/tools.md>)

### AI overview

The excerpt compares ETL platforms on destinations, connectors, pricing, trials, compliance and replication behavior. It says Informatica has opaque, sales-led pricing and a 30-day trial; Matillion has just over 100 connectors across five destinations, usage-based tiers and a 14-day trial; Xplenty offers almost 100 integrations and broad destination types but charges per source-destination connection and lacks change data capture; Stitch offers 130+ connectors, consumption-based tiers and a 14-day trial. The article also recommends testing platforms with live data for volume, speed, types, record structures and documentation, and asking how they handle source schema changes at the destination. It claims Fivetran supports 700+ sources and 15+ destinations, uses monthly-active-row pricing, and automatically migrates warehouse schemas when source schemas change.

### Source excerpt

Compare popular data integration tools on cost, security, reliability and number of supported sources.

## Analysis: Fivetran Is the Fastest-Growing Cloud ELT Provider | Blog | Fivetran

DevFeed: [Analysis: Fivetran Is the Fastest-Growing Cloud ELT Provider | Blog | Fivetran](<https://devfeed.tech/articles/analysis-fivetran-is-the-fastest-growing-cloud-elt-provider-blog-fivetran-141312.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/fivetran-is-fastest-growing-cloud-elt-provider>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [elt](<https://devfeed.tech/topics/elt.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data transformation](<https://devfeed.tech/topics/data-transformation.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [automation](<https://devfeed.tech/tags/automation.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-teams](<https://devfeed.tech/tags/data-teams.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [elt](<https://devfeed.tech/tags/elt.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [gartner](<https://devfeed.tech/tags/gartner.md>)

### AI overview

The text attributes Fivetran's growth to managed cloud data ingestion and ease of use. It quotes Gartner's 2020 market-share report, which says Fivetran nearly doubled revenue from 2019 to 2020 and identifies it as a vendor of choice for moving data from popular SaaS applications and databases into cloud warehouses. The described service uses prebuilt connectors, self-healing pipelines, minimal infrastructure management, and automatic normalization. It also offers in-warehouse SQL transformations powered by dbt, with version control, testing, and documentation. Gartner is quoted as calling this a more complete offering that could support future enterprise adoption and revenue growth.

### Source excerpt

Gartner recognizes Fivetran as the "vendor of choice" for enterprises looking to modernize their data infrastructure.

[Next page](<https://devfeed.tech/topics/data-pipelines.md?cursor=WyIyMDI2LTExLTE4VDA2OjEzOjE1KzAwOjAwIiwgIjMwN2MyNjdkLWQ2MzAtNGEwNy05MDM3LTZjMzZkM2Y4ZTdlYiJd>)