# data-engineering

Data engineering is a discipline within computer science and software engineering concerned with designing, building, and operating systems that collect, store, transform, and deliver data for downstream use.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Prompt engineering won't make enterprise AI trustworthy. Data engineering will.

DevFeed: [Prompt engineering won't make enterprise AI trustworthy. Data engineering will.](<https://devfeed.tech/articles/prompt-engineering-won-t-make-enterprise-ai-trustworthy-data-engineering-will-146812.md>)

Original publisher: [Read original article](<https://sdtimes.com/data/prompt-engineering-wont-make-enterprise-ai-trustworthy-data-engineering-will/>)

Author: Arjun Danda Sureshbabu

Published: 2026-11-20T16:50:33Z

Content type: opinion

Language: en

Sources: [SD Times](<https://devfeed.tech/sources/sd-times.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [data-governance](<https://devfeed.tech/topics/data-governance.md>), [metric-standardization](<https://devfeed.tech/topics/metric-standardization.md>), [knowledge-engineering](<https://devfeed.tech/topics/knowledge-engineering.md>), [business logic](<https://devfeed.tech/topics/business-logic.md>), [enterprise data protection](<https://devfeed.tech/topics/enterprise-data-protection.md>), [trust](<https://devfeed.tech/topics/trust.md>)

Tags: [access-control](<https://devfeed.tech/tags/access-control.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-assistant](<https://devfeed.tech/tags/ai-assistant.md>), [ai-projects](<https://devfeed.tech/tags/ai-projects.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [code-review](<https://devfeed.tech/tags/code-review.md>), [column](<https://devfeed.tech/tags/column.md>), [context](<https://devfeed.tech/tags/context.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-engineers](<https://devfeed.tech/tags/data-engineers.md>), [data-layer](<https://devfeed.tech/tags/data-layer.md>), [database](<https://devfeed.tech/tags/database.md>), [definition](<https://devfeed.tech/tags/definition.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [enterprise-ai](<https://devfeed.tech/tags/enterprise-ai.md>), [enterprise-data](<https://devfeed.tech/tags/enterprise-data.md>), [finance](<https://devfeed.tech/tags/finance.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompt-engineering](<https://devfeed.tech/tags/prompt-engineering.md>)

### AI overview

The excerpt argues that confident wrong answers often come from missing business meaning and controls in enterprise data, not from poorly worded prompts. It describes a finance query that summed amounts without accounting for reversals, remapped region codes, or the narrower definition of "bookings." Its proposed approach is to give models a maintained semantic layer with agreed business definitions, allowed values, synonyms, and accurate joins; enforce row- and column-level access in the database; route common or consequential questions to reviewed queries; and show the query, definitions, and filters used for new answers. Test sets with known answers help check changes. The article says this puts definitions and policies under engineering review and monitoring, while prompts still provide context and guide follow-up questions. This is based on the supplied excerpt.

### Source excerpt

A few months into building an AI assistant for our finance teams, someone asked it what bookings looked like for one region last quarter. It answered in a few minutes, with a table and a summary explaining how it got there. The number was inaccurate. My first instinct was to blame the prompt, and we'd... ... continue reading The post Prompt engineering won't make enterprise AI trustworthy. Data engineering will. appeared first on SD Times.

## A Neurosymbolic Multimodal Framework for 30-Day Diabetic Readmission Prediction: Integrating Structured EHR Data, Dense Clinical Embeddings, and Knowledge-Grounded Symbolic Extraction

DevFeed: [A Neurosymbolic Multimodal Framework for 30-Day Diabetic Readmission Prediction: Integrating Structured EHR Data, Dense Clinical Embeddings, and Knowledge-Grounded Symbolic Extraction](<https://devfeed.tech/articles/a-neurosymbolic-multimodal-framework-for-30-day-diabetic-readmission-prediction-integrating-structured-ehr-data-dense-clinical-embeddings-and-knowledge-grounded-symbolic-145488.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25236>)

Author: Una Suman Kumar Patro, Sibangi Subhadarsani, Narendra Badam

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Vectors & LLMs](<https://devfeed.tech/topics/vectors-llms.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [AI for data automation](<https://devfeed.tech/topics/ai-for-data-automation.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>)

Tags: [cross-encoder](<https://devfeed.tech/tags/cross-encoder.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [hallucinations](<https://devfeed.tech/tags/hallucinations.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [retrieval-augmented-generation](<https://devfeed.tech/tags/retrieval-augmented-generation.md>), [structured](<https://devfeed.tech/tags/structured.md>), [xgboost](<https://devfeed.tech/tags/xgboost.md>)

### AI overview

On a held-out cohort of 20,153 encounters, the multimodal model improved sensitivity over the structured EHR baseline (0.6667 vs. 0.5839), achieved the best F2-score (0.3824) and lower Brier score (0.0977 vs. 0.1080), but had slightly lower AUROC (0.6375 vs. 0.6477) and specificity (0.5161). It combines engineered structured features, 32 compressed Bio_ClinicalBERT embedding components, and symbolic entities extracted through hybrid retrieval and negation auditing, then predicts with cost-sensitive XGBoost. In a separate 500-note code-switched benchmark, grounded retrieval raised Hinglish symptom recall and precision and reduced entity hallucinations. Prior utilization was the dominant risk signal. The reported findings are from the abstract's dataset and evaluations.

### Source excerpt

arXiv:2611.25236v1 Announce Type: new Abstract: Unplanned 30-day readmissions among patients with diabetes mellitus impose a substantial clinical and economic burden, yet most predictive models use only structured electronic health record (EHR) data and ignore free-text clinical notes, which often contain code-mixed Hindi-English (Hinglish) phrasing and negation that general-purpose language models parse unreliably. We propose a neurosymbolic multimodal framework evaluated on the public Diabetes 130-US Hospitals dataset (101,766 encounters; 71,518 patients) with a patient-level grouped 80/20 split. Three streams are fused: engineered structured features; 32 components of Bio_ClinicalBERT note embeddings obtained by truncated singular value decomposition; and symbolic entities extracted by an open-weight clinical language model within a three-stage hybrid retrieval-augmented generation pipeline (dense retrieval, sparse lexical retrieval, and cross-encoder re-ranking) with reflective negation auditing. A cost-sensitive XGBoost classifier with a decision threshold calibrated by Youden's J statistic produces the prediction. On the held-out test cohort (n = 20,153), the structured baseline achieved an area under the receiver operating characteristic curve (AUROC) of 0.6477, sensitivity of 0.5839, and Brier score of 0.1080. The full multimodal model achieved the highest sensitivity (0.6667; +8.28 percentage points), the best F2-score (0.3824), and the lowest Brier score (0.0977), at a slightly lower AUROC (0.6375) and specificity (0.5161). Shapley additive explanations showed that prior utilization dominates risk, with compressed narrative components among the top predictors. On a 500-note code-switched benchmark, grounded retrieval raised Hinglish symptom recall from 4.8% to 40.2% and precision from 3.1% to 45.2%, while reducing entity hallucinations from 1.74 to 0.00 per note.

## How to aggregate test results across teams with Temporal Nexus

DevFeed: [How to aggregate test results across teams with Temporal Nexus](<https://devfeed.tech/articles/how-to-aggregate-test-results-across-teams-with-temporal-nexus-146865.md>)

Original publisher: [Read original article](<https://temporal.io/blog/how-to-aggregate-test-results-across-teams-with-temporal-nexus>)

Author: Brooks Newberry

Published: 2026-11-20T00:00:00Z

Content type: article

Language: en

Sources: [Temporal](<https://devfeed.tech/sources/temporal-blog.md>)

Topics: [Test automation](<https://devfeed.tech/topics/test-automation.md>), [Integration testing](<https://devfeed.tech/topics/integration-testing.md>), [Remote Procedure Call (RPC)](<https://devfeed.tech/topics/rpc.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>)

Tags: [aggregate](<https://devfeed.tech/tags/aggregate.md>), [collect](<https://devfeed.tech/tags/collect.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [format](<https://devfeed.tech/tags/format.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [nexus](<https://devfeed.tech/tags/nexus.md>), [release](<https://devfeed.tech/tags/release.md>), [release-engineering](<https://devfeed.tech/tags/release-engineering.md>), [report](<https://devfeed.tech/tags/report.md>), [results](<https://devfeed.tech/tags/results.md>), [teams](<https://devfeed.tech/tags/teams.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [temporal-voices](<https://devfeed.tech/tags/temporal-voices.md>), [test](<https://devfeed.tech/tags/test.md>)

### AI overview

Temporal's Release Engineering team built a shared report format and Nexus Operation to collect test evidence from cloud validation and open source test Workflows while leaving tests and verdicts with their owners. Producers call a named Operation through an allowed Temporal Endpoint using their existing Temporal connections; a direct gRPC write API would require a reachable service and separate authentication and access controls. The synchronous handler validates and stores reports, keyed by (source, business_id), then returns the record ID. That key makes retries after a lost response return the same record, but repeated submissions can replace contents, and older payloads are not rejected by timestamp. Each report and result uses pass, warn, fail, or skipped; a test failure can coexist with successful ingestion. The team reads reports through a separate gRPC API. Nexus access is limited to explicitly allowed Namespaces in the same Cloud Account, and the synchronous database work must finish within ten seconds; Temporal recommends asynchronous Operations for uncertain database latency or availability. Reports support release decisions but do not automatically determine readiness,

### Source excerpt

How Temporal's Release Engineering team used a Nexus Operation and a shared report format to collect test results from many teams in one place.

## How to Generate Synthetic Healthcare Data Using Synthea: A Hands On Tutorial

DevFeed: [How to Generate Synthetic Healthcare Data Using Synthea: A Hands On Tutorial](<https://devfeed.tech/articles/how-to-generate-synthetic-healthcare-data-using-synthea-a-hands-on-tutorial-144721.md>)

Original publisher: [Read original article](<https://www.freecodecamp.org/news/how-to-generate-synthetic-healthcare-data-using-synthea-a-hands-on-tutorial/>)

Author: Lakshmi Mahabaleshwara

Published: 2026-11-19T22:00:42Z

Content type: tutorial

Language: en

Sources: [Free Code Camp](<https://devfeed.tech/sources/freecodecamp-programming-tutorials-python-javascript-git-more.md>)

Topics: [Tutorial](<https://devfeed.tech/topics/tutorial.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Structured-data](<https://devfeed.tech/topics/structured-data.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-pipeline](<https://devfeed.tech/tags/analytics-pipeline.md>), [applications](<https://devfeed.tech/tags/applications.md>), [csv](<https://devfeed.tech/tags/csv.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [dataengineering](<https://devfeed.tech/tags/dataengineering.md>), [generate](<https://devfeed.tech/tags/generate.md>), [healthcare](<https://devfeed.tech/tags/healthcare.md>), [healthcareai](<https://devfeed.tech/tags/healthcareai.md>), [healthcaredata](<https://devfeed.tech/tags/healthcaredata.md>), [healthtech](<https://devfeed.tech/tags/healthtech.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [synthetichealthcaredata](<https://devfeed.tech/tags/synthetichealthcaredata.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>), [using](<https://devfeed.tech/tags/using.md>)

### AI overview

Synthea simulates longitudinal patient records, making it useful for prototyping and testing healthcare data workflows without using real patient histories. The tutorial shows how to build it with Java 17 or newer, generate about 100 patients, enable CSV export explicitly, and explore linked patient, encounter, condition, observation, medication, and procedure files with Python and Pandas. A fixed seed helps reproduce runs; version, configuration, and reference date also affect results. Default population overflow can produce more records than requested (115 in the described run); disabling it targets the requested count but may change how closely survivor demographics match population targets. The tutorial demonstrates basic relationship and missing-value checks and reconstructing a patient's timeline. Synthetic data is not necessarily representative, clinically accurate, unbiased, or suitable evidence for real-world model performance, so validate concepts, data quality, and representativeness before use.

### Source excerpt

Healthcare generates huge amounts of data every day, from patient visits and lab results to diagnoses, medications, and medical imaging. For developers and data engineers building healthcare applicati

## Enabling Real-Time AI for Scientific Instruments with GPU-Native Data Acquisition

DevFeed: [Enabling Real-Time AI for Scientific Instruments with GPU-Native Data Acquisition](<https://devfeed.tech/articles/enabling-real-time-ai-for-scientific-instruments-with-gpu-native-data-acquisition-143888.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/enabling-real-time-ai-for-scientific-instruments-with-gpu-native-data-acquisition/>)

Author: Cara Laasch

Published: 2026-11-19T17:00:00Z

Content type: article

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [Holoscan](<https://devfeed.tech/topics/holoscan.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>)

Tags: [acquisition](<https://devfeed.tech/tags/acquisition.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [batch](<https://devfeed.tech/tags/batch.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [communication](<https://devfeed.tech/tags/communication.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [connectx](<https://devfeed.tech/tags/connectx.md>), [control](<https://devfeed.tech/tags/control.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [data](<https://devfeed.tech/tags/data.md>), [developer-tools-techniques](<https://devfeed.tech/tags/developer-tools-techniques.md>), [dgx-spark](<https://devfeed.tech/tags/dgx-spark.md>), [edge-computing](<https://devfeed.tech/tags/edge-computing.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [instruments](<https://devfeed.tech/tags/instruments.md>), [measurements](<https://devfeed.tech/tags/measurements.md>), [networking-communications](<https://devfeed.tech/tags/networking-communications.md>), [processes](<https://devfeed.tech/tags/processes.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [sc26](<https://devfeed.tech/tags/sc26.md>), [science](<https://devfeed.tech/tags/science.md>), [sensor](<https://devfeed.tech/tags/sensor.md>)

### AI overview

NVIDIA DAQIRI moves high-bandwidth sensor data directly into GPU memory so it can be processed during acquisition, enabling real-time inspection and feedback without first writing measurements to disk. At Lawrence Berkeley National Laboratory's Molecular Foundry, a high-dynamic-range electron microscopy camera streams over 500 Gbps; DAQIRI and GPU processing assemble and correct detector frames, derive energy-loss spectra, and improve spatial resolution from 5-micrometer sensor pixels to submicron resolution. The latest release adds packet reordering on supported NVIDIA ConnectX-7-or-newer NICs, an ibverbs backend, and direct polling. The ibverbs engine benchmarks at about twice the previous DPDK backend's per-core throughput for packets up to 1,024 bytes at rates above 200 Gbps on specified NVIDIA platforms. Direct polling avoids a separate I/O-worker handoff; cabled loopback tests demonstrated roughly 2-4 μs application send-to-receive latency. These results are benchmark-specific, and the excerpt recommends evaluating throughput and latency on the target hardware.

### Source excerpt

In science and industry, analyzing sensor measurements in real time can enable AI to adjust processes, guide experiments, and search for better operating...

## Biomedical Knowledge Composition: A Software Engineering Perspective

DevFeed: [Biomedical Knowledge Composition: A Software Engineering Perspective](<https://devfeed.tech/articles/biomedical-knowledge-composition-a-software-engineering-perspective-142115.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.08927>)

Author: Natallia Kokash, Adam S. Z. Bellouma, Paola Grosso

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [knowledge-engineering](<https://devfeed.tech/topics/knowledge-engineering.md>), [knowledge-management](<https://devfeed.tech/topics/knowledge-management.md>)

Tags: [biomedical](<https://devfeed.tech/tags/biomedical.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [data](<https://devfeed.tech/tags/data.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [development](<https://devfeed.tech/tags/development.md>), [digital-twins](<https://devfeed.tech/tags/digital-twins.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [drug-discovery](<https://devfeed.tech/tags/drug-discovery.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [entity-resolution](<https://devfeed.tech/tags/entity-resolution.md>), [governance](<https://devfeed.tech/tags/governance.md>), [graphs](<https://devfeed.tech/tags/graphs.md>), [knowledge-graphs](<https://devfeed.tech/tags/knowledge-graphs.md>), [package-management](<https://devfeed.tech/tags/package-management.md>), [perspectives](<https://devfeed.tech/tags/perspectives.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [practices](<https://devfeed.tech/tags/practices.md>), [software](<https://devfeed.tech/tags/software.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>)

### AI overview

The article argues that biomedical knowledge graphs are central to integrating molecular, clinical, and population data, but their construction is hindered by identifier mapping, entity resolution, schema alignment, evidence integration, and provenance tracking. It connects these domain challenges to limited adoption of mature software engineering practices, including package management, typed namespaces, canonical interchange formats, service composition, reproducible pipelines, and lifecycle governance. It catalogs eight open integration challenges, noting that partial solutions exist but no universally adopted stack. Its main emphasis is on reproducible KG assembly: reusable build pipelines and versioned dependencies that let others compile and customize graphs from source, rather than relying only on static deployed artifacts.

### Source excerpt

arXiv:2608.08927v2 Announce Type: replace Abstract: Biomedical research has accumulated vast molecular, clinical, and population data, yet translating this wealth into actionable knowledge remains constrained by technical and organizational difficulties. This article presents a unified treatment of two perspectives on biomedical knowledge infrastructure. The first introduces the biomedical domain to software engineers: it explains why knowledge graphs (KGs) are the central integrative data structure in modern biomedicine, characterizes five data harmonization challenges (identifier mapping, entity resolution, schema alignment, evidence integration, and provenance tracking), surveys application domains from drug discovery to digital twins, and profiles six representative KG systems with contrasting choices. The second perspective asks why engineering biomedical knowledge infrastructure remains so difficult. We argue that a contributing root cause is limited adoption of software tooling and practices that make development in other mature domains - particularly web engineering - reliably composable and reproducible: package management, typed namespaces, canonical interchange formats, service composition protocols, reproducible pipelines, and lifecycle governance. Against this backdrop, eight open engineering challenges for biomedical data integration are catalogued, each with partial solutions but no universally adopted stack. Crucially, the article shifts emphasis from describing deployed KG instances toward the reproducible process of assembling them: reusable build pipelines, versioned dependencies, and engineering practices that let others compile and customize a KG from source rather than consuming a static artifact. Together, the two perspectives provide domain grounding for newcomers and a research agenda for software engineers seeking to make transformative contributions to biomedical knowledge infrastructure.

## KnowledgeClaw: Towards Agentic Knowledge Engineering via Omnimodal Hypergraphs

DevFeed: [KnowledgeClaw: Towards Agentic Knowledge Engineering via Omnimodal Hypergraphs](<https://devfeed.tech/articles/knowledgeclaw-towards-agentic-knowledge-engineering-via-omnimodal-hypergraphs-142774.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24294>)

Author: Haoran Luo, Jiapu Wang, Linhao Luo, Luu Anh Tuan

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [knowledge-engineering](<https://devfeed.tech/topics/knowledge-engineering.md>), [Agentic AI](<https://devfeed.tech/topics/what-is-agentic-ai.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [application](<https://devfeed.tech/tags/application.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [bench](<https://devfeed.tech/tags/bench.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [data](<https://devfeed.tech/tags/data.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [files](<https://devfeed.tech/tags/files.md>), [graph](<https://devfeed.tech/tags/graph.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [knowledge-engineering](<https://devfeed.tech/tags/knowledge-engineering.md>), [lifecycle](<https://devfeed.tech/tags/lifecycle.md>), [live](<https://devfeed.tech/tags/live.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [memory](<https://devfeed.tech/tags/memory.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [query](<https://devfeed.tech/tags/query.md>), [rag](<https://devfeed.tech/tags/rag.md>)

### AI overview

KnowledgeClaw combines an omnimodal typed hypergraph, a closed-loop agent that keeps the knowledge base current, and query routing to the needed granularity. The paper introduces OmniKE as the combination of omnimodality, lifecycle management, and multi-granular use, and reports that existing systems address at most two of these together. Its OmniKE-Bench contains six tasks and 1,829 model-free items drawn from 3,960 real files across five modalities. Against 30 baselines, KnowledgeClaw ranks first on all six tasks, with an overall OKE score of 55.9--14.6 above the strongest baseline. These are results reported in the supplied abstract excerpt.

### Source excerpt

arXiv:2611.24294v1 Announce Type: new Abstract: Knowledge engineering (KE) turns raw files into a reasoning-ready knowledge base, a task now within reach of LLM agents. KE has progressed from chunk and graph retrieval-augmented generation (RAG) to memory agents, agentic KE and multimodal KE. However, a live knowledge base needs omnimodality, a closed-loop lifecycle and multi-granular application at once, yet existing systems meet at most two and no benchmark measures all three. We formalize Omnimodal Knowledge Engineering (OmniKE) and release OmniKE-Bench, six tasks with 1,829 model-free items over 3,960 real files in five modalities. We further propose KnowledgeClaw, an agentic system that builds an omnimodal typed hypergraph, keeps it current with a closed-loop lifecycle agent and routes each query to the granularity it needs. Against 30 baselines, it ranks first on all six tasks with an overall OKE of 55.9, +14.6 over the strongest baseline. Our software and data are publicly available at https://github.com/LHRLAB/KnowledgeClaw.

## Piccolo: Co-Evolving Decision Models and Data through Recursive Self-Improvement

DevFeed: [Piccolo: Co-Evolving Decision Models and Data through Recursive Self-Improvement](<https://devfeed.tech/articles/piccolo-co-evolving-decision-models-and-data-through-recursive-self-improvement-142820.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24651>)

Author: Zhenqing Ling, Qirui Jiao, Daoyuan Chen, Cheng Chen, Xinji Mai, Haowen Gao, Zhuo Zhang, Xuchen Pan, Mengsu Ding, Jiaji Deng, Yuexiang Xie, Xiaoming Qin, Weirui Kuang, Zhaoyang Liu, Yaliang Li, Bolin D

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Advanced AI workflows](<https://devfeed.tech/topics/advanced-ai-workflows.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-harnesses](<https://devfeed.tech/tags/agent-harnesses.md>), [agentic-data-engineering](<https://devfeed.tech/tags/agentic-data-engineering.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [curation](<https://devfeed.tech/tags/curation.md>), [data](<https://devfeed.tech/tags/data.md>), [data-driven](<https://devfeed.tech/tags/data-driven.md>), [data-workflows](<https://devfeed.tech/tags/data-workflows.md>), [decision-models](<https://devfeed.tech/tags/decision-models.md>), [family](<https://devfeed.tech/tags/family.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [first](<https://devfeed.tech/tags/first.md>), [games](<https://devfeed.tech/tags/games.md>), [leaderboard](<https://devfeed.tech/tags/leaderboard.md>), [memory](<https://devfeed.tech/tags/memory.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [recursive-self-improvement](<https://devfeed.tech/tags/recursive-self-improvement.md>), [research](<https://devfeed.tech/tags/research.md>), [rsi](<https://devfeed.tech/tags/rsi.md>)

### AI overview

Piccolo is a 0.8B-27B multimodal decision-model family developed through recursive co-improvement of models and training data. Research agents analyze behavior, build data, run experiments, and feed results into later proposals, linked by semantic slices and shared research memory; the training recipes use hundreds of thousands of examples. Piccolo-27B scores 75.59 on Decision Index Vision Full, with 9B and 4B variants scoring 68.35 and 67.46; all three rank first on their size categories' Private benchmark. The authors report that earlier backbones can outperform newer ones on visual tasks under the same recipe, targeted curation can let 4B-9B models rival 27B baselines on specialized workflows, and lower training loss may not improve--and can harm--discrete decision accuracy. The evidence is the supplied abstract, not full paper text.

### Source excerpt

arXiv:2611.24651v1 Announce Type: cross Abstract: We present Piccolo, a family of multimodal decision models (0.8B to 27B) that coevolves model capabilities and training data via Agentic Data Engineering and recursive self-improvement (RSI). The models predict probabilities over candidate answers for general-purpose text and visual judgments, agent harnesses, data workflows, and interactive games. Piccolo-27B tops the Decision Index Vision Full leaderboard with a score of 75.59, while Piccolo-9B and Piccolo-4B achieve 68.35 and 67.46, respectively. All three rank first on the Private benchmark in their respective model-size categories. Research agents analyze model behavior, construct training data, conduct experiments, and use the results to refine subsequent proposals. Semantic slices and a shared research memory connect these actions across the model family's development. With training recipes comprising hundreds of thousands of examples, data selection and feedback-driven refinement are central to the self-improvement process. Our large-scale RSI exploration yields three counter-intuitive insights for decision-model development: (1) Backbone-recipe co-adaptation outweighs raw model recency: earlier backbones can surpass newer generations on visual tasks under identical recipes; (2) Targeted data curation enables 4B-9B models to rival 27B baselines on specialized workflows; and (3) Continuous loss reduction can decouple from and even degrade discrete decision accuracy. Together with the released weights and tools from the Data-Juicer and AgentScope ecosystems, Piccolo provides an actionable blueprint for building frontier decision models and scaling data-driven RSI.

## Building Predictive AI Models When Synthetic Data Is All You Have

DevFeed: [Building Predictive AI Models When Synthetic Data Is All You Have](<https://devfeed.tech/articles/building-predictive-ai-models-when-synthetic-data-is-all-you-have-144066.md>)

Original publisher: [Read original article](<https://engineering.salesforce.com/building-predictive-ai-models-when-synthetic-data-is-all-you-have/>)

Author: Taniya Ghosh

Published: 2026-11-19T00:59:15Z

Content type: article

Language: en

Sources: [Salesforce](<https://devfeed.tech/sources/salesforce-engineering-blog.md>)

Topics: [Predictive AI](<https://devfeed.tech/topics/predictive-ai.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Advanced AI workflows](<https://devfeed.tech/topics/advanced-ai-workflows.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [AI Models](<https://devfeed.tech/topics/ai-models.md>), [Feature Engineering](<https://devfeed.tech/topics/feature-engineering.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [customer-data](<https://devfeed.tech/tags/customer-data.md>), [data](<https://devfeed.tech/tags/data.md>), [data-360](<https://devfeed.tech/tags/data-360.md>), [predictive-ai](<https://devfeed.tech/tags/predictive-ai.md>), [synthetic-data](<https://devfeed.tech/tags/synthetic-data.md>), [training-data](<https://devfeed.tech/tags/training-data.md>)

### AI overview

Salesforce's team built reusable IT service management prediction templates before real customer incidents were available for training. Product-manager scenarios, varied within guardrails and expanded with open-source LLMs, supplied synthetic data for testing. To avoid being misled by aggregate accuracy on rare SLA breaches, the team checked whether scores changed as expected when one risk factor changed and whether the top three predictors made sense to service managers; these scenario tests did not establish production accuracy. Data 360 transforms customer incident histories into training data, Einstein on Data 360 trains an XGBoost model, and a scoring transform writes predictions for display on incident records. Templates carry the data contract and model structure, while each customer's history supplies its own learned parameters. The excerpt says training requires about 400 completed incidents and scoring runs in scheduled batches about hourly; thin or poor-quality histories may reduce reliability, and immediate scoring remained a need. At interview time, the team had received no reports of the anticipated challenges becoming production problems and was awaiting more user反馈.

### Source excerpt

In our Engineering Energizers Q&A series, we highlight the engineering minds driving innovation across Salesforce. Today, we spotlight Taniya Ghosh, Senior Manager of Software Engineering, whose team faced a brutal engineering contradiction: build predictive AI models that warn of escalating incidents before they had real customer incidents to learn from. Synthetic training data gave them [...] The post Building Predictive AI Models When Synthetic Data Is All You Have appeared first on Salesforce Engineering Blog.

## Thoughts on Verification Engineering in the Agentic SDLC

DevFeed: [Thoughts on Verification Engineering in the Agentic SDLC](<https://devfeed.tech/articles/thoughts-on-verification-engineering-in-the-agentic-sdlc-142073.md>)

Original publisher: [Read original article](<https://www.dataengineeringweekly.com/p/thoughts-on-verification-engineering>)

Author: Ananth Packkildurai

Published: 2026-11-18T21:59:55Z

Content type: article

Language: en

Sources: [Data Engineering Weekly](<https://devfeed.tech/sources/data-engineering-weekly.md>)

Topics: [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Spec Driven Development](<https://devfeed.tech/topics/spec-driven-development.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-sdlc](<https://devfeed.tech/tags/agentic-sdlc.md>), [agentic-software](<https://devfeed.tech/tags/agentic-software.md>), [agentic-software-development](<https://devfeed.tech/tags/agentic-software-development.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-coding-assistants](<https://devfeed.tech/tags/ai-coding-assistants.md>), [code](<https://devfeed.tech/tags/code.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [manufacturing](<https://devfeed.tech/tags/manufacturing.md>), [sdlc](<https://devfeed.tech/tags/sdlc.md>), [software](<https://devfeed.tech/tags/software.md>), [software-supply-chain](<https://devfeed.tech/tags/software-supply-chain.md>), [supply-chain](<https://devfeed.tech/tags/supply-chain.md>), [thoughts](<https://devfeed.tech/tags/thoughts.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

The article argues that AI accelerates code production faster than existing review and testing loops can verify it, shifting the engineering bottleneck toward proving behavior before release. Static analysis and human diff reviews may miss cross-service runtime failures, while relying on canaries risks exposing users to too many regressions. It proposes Verification Engineering built around disposable production-equivalent sandboxes, synthetic test data shaped by domain distributions and failure patterns, automatically generated test harnesses and correctness/performance/resource assertions, and simulated end-to-end user workflows. Requirements and designs would become executable specifications, while production logs, traces, and interactions would feed anomalies back into future sandbox tests. These are presented as a proposed discipline, not as a comparison of specific products or evidence of measured results.

### Source excerpt

When AI accelerates code manufacturing, the software supply chain bottleneck shifts from writing code to building the feedback loop that proves it works.

## Exclusive: Wherobots AI agents automate complex spatial data engineering

DevFeed: [Exclusive: Wherobots AI agents automate complex spatial data engineering](<https://devfeed.tech/articles/exclusive-wherobots-ai-agents-automate-complex-spatial-data-engineering-140316.md>)

Original publisher: [Read original article](<https://siliconangle.com/2026/11/18/exclusive-wherobots-ai-agents-automate-complex-spatial-data-engineering/>)

Author: Paul Gillin

Published: 2026-11-18T11:00:42Z

Content type: news

Language: en

Sources: [SiliconANGLE](<https://devfeed.tech/sources/siliconangle.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [geospatial](<https://devfeed.tech/topics/geospatial.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Geographic Information System](<https://devfeed.tech/topics/gis.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-sedona](<https://devfeed.tech/tags/apache-sedona.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [geographic-visualizations](<https://devfeed.tech/tags/geographic-visualizations.md>), [geospatial-data](<https://devfeed.tech/tags/geospatial-data.md>), [global-marketing-insights-inc](<https://devfeed.tech/tags/global-marketing-insights-inc.md>), [news](<https://devfeed.tech/tags/news.md>), [spatial](<https://devfeed.tech/tags/spatial.md>), [spatial-analytics](<https://devfeed.tech/tags/spatial-analytics.md>), [spatial-data-catalog](<https://devfeed.tech/tags/spatial-data-catalog.md>), [the-latest](<https://devfeed.tech/tags/the-latest.md>), [top-story-5](<https://devfeed.tech/tags/top-story-5.md>), [wherobots-havasu](<https://devfeed.tech/tags/wherobots-havasu.md>), [wherobots-inc](<https://devfeed.tech/tags/wherobots-inc.md>), [wherobots-rasterflow](<https://devfeed.tech/tags/wherobots-rasterflow.md>), [wherobots-waldo](<https://devfeed.tech/tags/wherobots-waldo.md>), [wherobotsdb](<https://devfeed.tech/tags/wherobotsdb.md>)

### AI overview

Wherobots' Waldo agent uses natural-language instructions to build spatial data pipelines, coordinating data discovery, query generation, analysis, visualization, and scheduled refreshes. Its platform combines Havasu for dataset discovery and preparation, RasterFlow for imagery processing, WherobotsDB with more than 300 spatial functions, and Wherobots Maps. Wherobots says it connects existing LLMs to specialized processing engines and can use existing cloud storage and catalogs such as AWS Glue and Databricks Unity Catalog, without requiring data migration. Users can inspect generated queries and processing steps, but the company cautions that prompts do not always produce accurate results. Waldo is in public preview; agent tokens are included at no extra charge in Professional Edition, while customers primarily pay for computing resources.

### Source excerpt

Cloud-native spatial analytics and artificial intelligence startup Wherobots Inc. today introduced an artificial intelligence agent that enables users to build and deploy spatial data pipelines using natural-language instructions, reducing the need for specialized programming expertise. Called Waldo, the agent can discover data sources, generate queries, perform analysis, visualize results and schedule recurring operations. It's designed [...] The post Exclusive: Wherobots AI agents automate complex spatial data engineering appeared first on SiliconANGLE.

## Simplify Brand Awareness Analysis With Fivetran Data Models | Blog | Fivetran

DevFeed: [Simplify Brand Awareness Analysis With Fivetran Data Models | Blog | Fivetran](<https://devfeed.tech/articles/simplify-brand-awareness-analysis-with-fivetran-data-models-blog-fivetran-141338.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/simplify-brand-awareness-analysis-with-fivetran-data-models>)

Author: Joe Markiewicz

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [etl](<https://devfeed.tech/topics/etl.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Instagram](<https://devfeed.tech/topics/instagram.md>)

Tags: [advertising](<https://devfeed.tech/tags/advertising.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [awareness](<https://devfeed.tech/tags/awareness.md>), [brand](<https://devfeed.tech/tags/brand.md>), [brand-awareness](<https://devfeed.tech/tags/brand-awareness.md>), [data](<https://devfeed.tech/tags/data.md>), [engagement](<https://devfeed.tech/tags/engagement.md>), [engagement-metrics](<https://devfeed.tech/tags/engagement-metrics.md>), [facebook](<https://devfeed.tech/tags/facebook.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [instagram](<https://devfeed.tech/tags/instagram.md>), [linkedin](<https://devfeed.tech/tags/linkedin.md>), [marketing-analytics](<https://devfeed.tech/tags/marketing-analytics.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [models](<https://devfeed.tech/tags/models.md>), [twitter](<https://devfeed.tech/tags/twitter.md>)

### AI overview

Fivetran's social media data model combines cleaned, tested, normalized engagement data from Facebook Pages, Instagram Business, Organic Twitter, and LinkedIn Company Pages into one schema and table. It rolls up clicks, impressions, shares, likes, and comments, supports combining multiple accounts on a connector, and helps compare engagement across profiles. Used alongside Fivetran's separate ad reporting model--which aggregates spend, clicks, and impressions across Facebook, Google, Pinterest, Bing, Twitter, and LinkedIn--analysts can compare daily reach and engagement. Both models use shared transformation logic maintained by Fivetran analytics engineers and community contributors.

### Source excerpt

Easily combine engagement metrics from Facebook, Instagram, Twitter and LinkedIn.

## Fivetran Makes Big Strides in India | Blog | Fivetran

DevFeed: [Fivetran Makes Big Strides in India | Blog | Fivetran](<https://devfeed.tech/articles/fivetran-makes-big-strides-in-india-blog-fivetran-141314.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/fivetran-makes-big-strides-in-india>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: news

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [big-data](<https://devfeed.tech/topics/big-data.md>)

Tags: [2021](<https://devfeed.tech/tags/2021.md>), [2022](<https://devfeed.tech/tags/2022.md>), [about](<https://devfeed.tech/tags/about.md>), [addition](<https://devfeed.tech/tags/addition.md>), [after](<https://devfeed.tech/tags/after.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [analysts](<https://devfeed.tech/tags/analysts.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [annual-recurring-revenue](<https://devfeed.tech/tags/annual-recurring-revenue.md>), [aws](<https://devfeed.tech/tags/aws.md>), [blog](<https://devfeed.tech/tags/blog.md>), [country](<https://devfeed.tech/tags/country.md>), [data](<https://devfeed.tech/tags/data.md>), [data-centers](<https://devfeed.tech/tags/data-centers.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [gcp](<https://devfeed.tech/tags/gcp.md>), [india](<https://devfeed.tech/tags/india.md>), [staff](<https://devfeed.tech/tags/staff.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Within 11 months of launching its product in India in January 2021, Fivetran reported $1 million in local ARR, ninefold revenue growth and more than twice as many customers. It cited over 180 connectors for centralizing data in warehouses such as Snowflake. In one example, Cars24 connected MySQL data to Snowflake and had a test database running in two days; the company said an internal pipeline would have required five or six engineers for up to four months. Fivetran also reported reaching 100 employees in India and opening an AWS data center alongside its in-country GCP data center. Following its HVR integration, it said it could serve large enterprises needing high-volume, real-time, multidirectional synchronization from sources such as Oracle and SAP.

### Source excerpt

With 100+ staff and dedicated in-country GCP and AWS data centers, Fivetran is growing rapidly in the country.

## Submit Your Nomination for the Modern Data Stack Awards | Blog | Fivetran

DevFeed: [Submit Your Nomination for the Modern Data Stack Awards | Blog | Fivetran](<https://devfeed.tech/articles/submit-your-nomination-for-the-modern-data-stack-awards-blog-fivetran-141339.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/submit-your-nomination-for-the-modern-data-stack-awards>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [data transformation](<https://devfeed.tech/topics/data-transformation.md>)

Tags: [2021](<https://devfeed.tech/tags/2021.md>), [analyst](<https://devfeed.tech/tags/analyst.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [announce](<https://devfeed.tech/tags/announce.md>), [app](<https://devfeed.tech/tags/app.md>), [application](<https://devfeed.tech/tags/application.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [awards](<https://devfeed.tech/tags/awards.md>), [blog](<https://devfeed.tech/tags/blog.md>), [categories](<https://devfeed.tech/tags/categories.md>), [conference](<https://devfeed.tech/tags/conference.md>), [data](<https://devfeed.tech/tags/data.md>), [data-stack](<https://devfeed.tech/tags/data-stack.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [good](<https://devfeed.tech/tags/good.md>), [hero](<https://devfeed.tech/tags/hero.md>), [modern](<https://devfeed.tech/tags/modern.md>), [modern-data](<https://devfeed.tech/tags/modern-data.md>), [modern-data-stack](<https://devfeed.tech/tags/modern-data-stack.md>), [transformation](<https://devfeed.tech/tags/transformation.md>)

### AI overview

The inaugural Modern Data Stack Awards recognize innovative data teams and people across seven categories: Modern Data Team, Mentor, Hero, Project, App, Leader, and Transformation for Good. Any business impacted by the modern data stack may submit a nomination, regardless of whether it uses a particular service or product. A hand-selected panel of industry experts will choose the winners, announced on the second day of the Modern Data Stack Conference on Fivetran's blog and in a social post. Submissions were due September 9, 2021.

### Source excerpt

Winners in seven categories -- including Data Hero and Transformation for Good -- will be announced at the Modern Data Stack Conference.

## Perform rapid analysis of your app platform data | Blog | Fivetran

DevFeed: [Perform rapid analysis of your app platform data | Blog | Fivetran](<https://devfeed.tech/articles/perform-rapid-analysis-of-your-app-platform-data-blog-fivetran-141335.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/perform-rapid-analysis-of-your-app-platform-data>)

Author: Jamie Rodriguez

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Google Play](<https://devfeed.tech/topics/google-play.md>), [etl](<https://devfeed.tech/topics/etl.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>)

Tags: [2022](<https://devfeed.tech/tags/2022.md>), [about](<https://devfeed.tech/tags/about.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [app](<https://devfeed.tech/tags/app.md>), [app-store](<https://devfeed.tech/tags/app-store.md>), [apple](<https://devfeed.tech/tags/apple.md>), [apple-app-store](<https://devfeed.tech/tags/apple-app-store.md>), [can](<https://devfeed.tech/tags/can.md>), [data](<https://devfeed.tech/tags/data.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [google](<https://devfeed.tech/tags/google.md>), [google-play](<https://devfeed.tech/tags/google-play.md>), [model](<https://devfeed.tech/tags/model.md>), [platform](<https://devfeed.tech/tags/platform.md>), [play](<https://devfeed.tech/tags/play.md>), [rapid](<https://devfeed.tech/tags/rapid.md>), [reporting](<https://devfeed.tech/tags/reporting.md>), [store](<https://devfeed.tech/tags/store.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Fivetran's App Reporting data models standardize Google Play and Apple App Store connector data into a unified schema, aggregating app activity daily by app and across country, OS version, app version, traffic source, and device. The models are intended to reduce the need for teams to learn and join numerous normalized source tables. They include a data dictionary and model lineage graph. With Fivetran Transformations for dbt Core, integrated scheduling (available as of February 2022) runs the reporting model after connector syncs complete, so the resulting models use the latest synced data.

### Source excerpt

With Fivetran's new App Reporting data model, you can easily roll your Apple App Store and Google Play data into a unified schema for seamless reporting.

## Fivetran Launches Support for New Databricks + GCP Offering | Blog | Fivetran

DevFeed: [Fivetran Launches Support for New Databricks + GCP Offering | Blog | Fivetran](<https://devfeed.tech/articles/fivetran-launches-support-for-new-databricks-gcp-offering-blog-fivetran-141313.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/fivetran-launches-support-for-new-databricks-gcp-offering>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: release

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [billing](<https://devfeed.tech/tags/billing.md>), [choice](<https://devfeed.tech/tags/choice.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-marketplace](<https://devfeed.tech/tags/cloud-marketplace.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [data-science-and-analytics](<https://devfeed.tech/tags/data-science-and-analytics.md>), [data-scientists](<https://devfeed.tech/tags/data-scientists.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [gcp](<https://devfeed.tech/tags/gcp.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [launches](<https://devfeed.tech/tags/launches.md>), [platform](<https://devfeed.tech/tags/platform.md>), [support](<https://devfeed.tech/tags/support.md>)

### AI overview

Fivetran now supports Databricks on Google Cloud. Customers can purchase Fivetran through Google Cloud Marketplace, use Google Cloud credits, and receive unified billing for Google Cloud products and Fivetran purchases made there. The article also says all Fivetran connectors support Databricks Delta Lake and that Marketplace purchasing simplifies procurement. The offering is presented as expanding customer choice across Google Cloud and Databricks; no technical setup details are provided.

### Source excerpt

Businesses can now use Fivetran with the Lakehouse Platform on Google Cloud.

## Three Must-Haves for Your Modern Retail Data Stack | Blog | Fivetran

DevFeed: [Three Must-Haves for Your Modern Retail Data Stack | Blog | Fivetran](<https://devfeed.tech/articles/three-must-haves-for-your-modern-retail-data-stack-blog-fivetran-141344.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/three-must-haves-for-modern-retail-data-stack>)

Author: Raman Kaler

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [3](<https://devfeed.tech/tags/3.md>), [act](<https://devfeed.tech/tags/act.md>), [all](<https://devfeed.tech/tags/all.md>), [allows](<https://devfeed.tech/tags/allows.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [application](<https://devfeed.tech/tags/application.md>), [application-development](<https://devfeed.tech/tags/application-development.md>), [blog](<https://devfeed.tech/tags/blog.md>), [culture](<https://devfeed.tech/tags/culture.md>), [customer](<https://devfeed.tech/tags/customer.md>), [data](<https://devfeed.tech/tags/data.md>), [data-stack](<https://devfeed.tech/tags/data-stack.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [invest](<https://devfeed.tech/tags/invest.md>), [modern](<https://devfeed.tech/tags/modern.md>), [quickly](<https://devfeed.tech/tags/quickly.md>), [retail](<https://devfeed.tech/tags/retail.md>), [retailers](<https://devfeed.tech/tags/retailers.md>), [stack](<https://devfeed.tech/tags/stack.md>)

### AI overview

A modern retail data stack combines automated connectors, a cloud warehouse and visual analytics to make current information available for faster retail decisions. The article describes Fivetran connectors as syncing data from applications, databases, events and files, adapting to source changes and requiring no maintenance; Snowflake as providing governed access to centralized data; and Tableau as helping users explore performance. It cites a dashboard using more than 500 million transactions and 3 billion inventory records. Effective use also depends on data access, transparency and literacy, supported by coordination and a plan. Retail data needs vary by sector, including personalization, inventory and supply-chain optimization, consumer trend analysis, and ecommerce demand forecasting.

### Source excerpt

When retailers invest in a culture of data, they gain the ability to quickly navigate shifting customer priorities.

## Trends that will dominate analytics & BI in 2022 | Blog | Fivetran

DevFeed: [Trends that will dominate analytics & BI in 2022 | Blog | Fivetran](<https://devfeed.tech/articles/trends-that-will-dominate-analytics-bi-in-2022-blog-fivetran-141345.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/trends-that-will-dominate-analytics-bi-2022>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [DataOps](<https://devfeed.tech/topics/dataops.md>), [elt](<https://devfeed.tech/topics/elt.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [etl](<https://devfeed.tech/topics/etl.md>), [databricks](<https://devfeed.tech/topics/databricks.md>)

Tags: [2022](<https://devfeed.tech/tags/2022.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [automation](<https://devfeed.tech/tags/automation.md>), [beam](<https://devfeed.tech/tags/beam.md>), [bi](<https://devfeed.tech/tags/bi.md>), [bi-tools](<https://devfeed.tech/tags/bi-tools.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [boost](<https://devfeed.tech/tags/boost.md>), [building](<https://devfeed.tech/tags/building.md>), [business](<https://devfeed.tech/tags/business.md>), [business-intelligence](<https://devfeed.tech/tags/business-intelligence.md>), [chaos](<https://devfeed.tech/tags/chaos.md>), [dataops](<https://devfeed.tech/tags/dataops.md>), [elt](<https://devfeed.tech/tags/elt.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [managed](<https://devfeed.tech/tags/managed.md>), [trends](<https://devfeed.tech/tags/trends.md>)

### AI overview

The article predicts five shifts in analytics and BI for 2022: ELT pipelines becoming more continuous and fully managed, reducing reliance on brittle batch ETL; DataOps practices and features becoming more central, with dbt Labs cited for SQL-based warehouse transformations, lineage tracking, and version control; operational analytics making data available in frontline tools to support day-to-day decisions; BI tools enabling more collaborative work between data and business teams; and wider use of distributed processing frameworks such as Apache Spark, Beam, Flink, and PySpark. These are presented as forecasts, not measured outcomes.

### Source excerpt

Fully managed ELT, DataOps and more trends that will change the way we use data this year.

## Fivetran Named Databricks Innovation Partner | Blog | Fivetran

DevFeed: [Fivetran Named Databricks Innovation Partner | Blog | Fivetran](<https://devfeed.tech/articles/fivetran-named-databricks-innovation-partner-blog-fivetran-141310.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/fivetran-databricks-innovation-partner-award-winner-announces-sql-analytics-plans>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: news

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [award](<https://devfeed.tech/tags/award.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [innovation](<https://devfeed.tech/tags/innovation.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [lakehouse-architecture](<https://devfeed.tech/tags/lakehouse-architecture.md>), [partner](<https://devfeed.tech/tags/partner.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

Fivetran received Databricks' 2020 ISV Partners Innovation Award for its collaboration around Delta Lake and the Lakehouse architecture. The article says Fivetran automates ingestion, including handling changing SaaS APIs and schemas, and can create normalized source tables, capture historical changes through history mode, and provide prebuilt dbt transformations. It says setup requires minimal configuration and does not require customers to create a staging area. Planned work included dbt packages and job orchestration for Databricks SQL, which was presented as a way for analysts to run SQL-based pipelines and access data through BI tools; those capabilities were described as upcoming. The article also reports customer claims of near-real-time syncing and savings equivalent to two to three data engineers per year.

### Source excerpt

We're honored to win this prestigious award, and we're doubling down on the Lakehouse architecture with Databricks SQL plans.

## How to Use Fivetran Data Models to Accelerate Analytics | Blog | Fivetran

DevFeed: [How to Use Fivetran Data Models to Accelerate Analytics | Blog | Fivetran](<https://devfeed.tech/articles/how-to-use-fivetran-data-models-to-accelerate-analytics-blog-fivetran-141330.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/how-to-use-fivetran-data-models-to-accelerate-analytics>)

Author: Dayna Shoemaker

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [data-modeling](<https://devfeed.tech/topics/data-modeling.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [business-logic](<https://devfeed.tech/tags/business-logic.md>), [code-modularity](<https://devfeed.tech/tags/code-modularity.md>), [core](<https://devfeed.tech/tags/core.md>), [custom](<https://devfeed.tech/tags/custom.md>), [data-modeling](<https://devfeed.tech/tags/data-modeling.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [financial](<https://devfeed.tech/tags/financial.md>), [financial-reporting](<https://devfeed.tech/tags/financial-reporting.md>), [firefly](<https://devfeed.tech/tags/firefly.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [reporting](<https://devfeed.tech/tags/reporting.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

Fivetran's open-source dbt data models provide maintained SQL that works with data ingested through Fivetran, reducing the work of turning raw schemas into analytics-ready tables. Source packages document and standardize underlying schemas; transform packages create analytics-ready models. For standard reporting such as financial reports, packages may be used directly. Where business logic is more specialized, dbt users can combine them with custom models and schedule dependencies through Fivetran's dbt Core integration. Users who prefer to build transformations themselves can inspect the SQL as a template and contribute code to future package versions.

### Source excerpt

Fivetran pre-built data models package SQL to accelerate analytics with near instant reporting.

## What is Idempotence? | Blog | Fivetran

DevFeed: [What is Idempotence? | Blog | Fivetran](<https://devfeed.tech/articles/what-is-idempotence-blog-fivetran-140270.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/what-is-idempotence>)

Author: Charles Wang

Published: 2026-11-18T05:58:59Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [blog](<https://devfeed.tech/tags/blog.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [what-is](<https://devfeed.tech/tags/what-is.md>)

### AI overview

Idempotence means repeating an operation leaves the result unchanged after its first execution. In incremental data integration, a pipeline reads a saved cursor, syncs later records, then advances the cursor. If a sync saves a row but fails before updating the cursor, the next attempt may load that row again. An idempotent pipeline still produces the intended dataset; a non-idempotent one can duplicate the row. The excerpt says duplicates can distort analytics, disrupt downstream systems that expect unique records, and waste storage and computing resources. It does not describe specific implementation methods.

### Source excerpt

Idempotence is an important characteristic of many systems, including Fivetran. Learn what it means.

## Understanding Enterprise Customer Engagement With the Modern Data Stack | Blog | Fivetran

DevFeed: [Understanding Enterprise Customer Engagement With the Modern Data Stack | Blog | Fivetran](<https://devfeed.tech/articles/understanding-enterprise-customer-engagement-with-the-modern-data-stack-blog-fivetran-140265.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/understanding-enterprise-customer-engagement-with-the-modern-data-stack>)

Author: Ross Katz

Published: 2026-11-18T05:58:59Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [elt](<https://devfeed.tech/topics/elt.md>), [etl](<https://devfeed.tech/topics/etl.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>)

Tags: [analytics-workloads](<https://devfeed.tech/tags/analytics-workloads.md>), [api](<https://devfeed.tech/tags/api.md>), [azure-synapse](<https://devfeed.tech/tags/azure-synapse.md>), [b2b](<https://devfeed.tech/tags/b2b.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [cloud-data-warehouse](<https://devfeed.tech/tags/cloud-data-warehouse.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [crm](<https://devfeed.tech/tags/crm.md>), [customer-engagement](<https://devfeed.tech/tags/customer-engagement.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [modern-data-stack](<https://devfeed.tech/tags/modern-data-stack.md>)

### AI overview

Companies serving both individual and enterprise customers need separate, evolving data models and processes to understand leads, customer journeys, renewals, and retention. The article argues that operational databases, CRMs, and ERPs are built to drive actions, so customizing them for changing analytics needs can increase overhead, maintenance, and performance problems. It recommends moving operational data into a cloud data warehouse, analyzing it there, and sending useful results back to business systems. Custom pipelines can require substantial setup and upkeep, with challenges including API limits, change data capture, source schema changes, and expanding access needs. The proposed Modern Data Stack uses Fivetran connectors to load source data into warehouses such as BigQuery, Snowflake, Redshift, or Azure Synapse, alongside analytics tools and reverse ETL. This can shift engineering effort from migration plumbing to analysis, though the article presents these benefits as its recommendation rather than quantified results.

### Source excerpt

Companies with B2B and B2C channels face complex management challenges that a modern data stack can solve.

## Operating an Agent-Driven Dataset Factory: Discovery, Construction, and Public Release

DevFeed: [Operating an Agent-Driven Dataset Factory: Discovery, Construction, and Public Release](<https://devfeed.tech/articles/operating-an-agent-driven-dataset-factory-discovery-construction-and-public-release-139597.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21957>)

Author: Eduardo Ari\\~no de la Rubia, Szilard Pafka

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [construction](<https://devfeed.tech/tags/construction.md>), [cs-ma](<https://devfeed.tech/tags/cs-ma.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [factory](<https://devfeed.tech/tags/factory.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [public](<https://devfeed.tech/tags/public.md>), [publication](<https://devfeed.tech/tags/publication.md>), [release](<https://devfeed.tech/tags/release.md>), [release-workflow](<https://devfeed.tech/tags/release-workflow.md>), [releases](<https://devfeed.tech/tags/releases.md>), [review](<https://devfeed.tech/tags/review.md>), [selection](<https://devfeed.tech/tags/selection.md>), [software](<https://devfeed.tech/tags/software.md>), [source](<https://devfeed.tech/tags/source.md>), [sources](<https://devfeed.tech/tags/sources.md>), [state](<https://devfeed.tech/tags/state.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [workflow](<https://devfeed.tech/tags/workflow.md>), [write](<https://devfeed.tech/tags/write.md>)

### AI overview

Dataset Factory continuously turns public sources into prediction datasets using bounded language-model tasks to interpret sources, write construction code, and review task meaning, while deterministic software handles state, workers, retries, and publication. Over 81.53 hours it published 14 datasets at an estimated recurring cost of about $2.74 each, including failed attempts, inference, workers, and hosting. Separate agents completed prediction experiments on all 12 evaluated releases. Source review still found a task whose construction filters conflicted with its declared population; an earlier review had accepted later-record exclusions without reconciling them with the published selection rule. The account highlights the need to check task meaning separately from successful execution and downstream use.

### Source excerpt

arXiv:2611.21957v1 Announce Type: new Abstract: Turning public data into prediction datasets requires connecting source discovery, task definition, construction, and distribution. We deployed Dataset Factory to perform these steps continuously under a standing release policy. Bounded language-model tasks interpret sources, write construction code, and review task meaning; deterministic software manages state, workers, retries, and publication. Over 81.53 hours, the factory published 14 datasets at an estimated recurring operating expenditure of about \$2.74 per publication. The accounting includes failed attempts, inference, workers, and hosting. Separate agents completed prediction experiments on all twelve evaluated releases. Source review nevertheless identified a task whose construction filters contradicted its population declaration. The original reviewer accepted later-record exclusions without reconciling them with the published selection rule. This experience documents an operating source-to-release workflow and shows why successful execution and consumer use need separate checks of task meaning.

## How To Build a Data Pipeline: A Step-by-Step Guide

DevFeed: [How To Build a Data Pipeline: A Step-by-Step Guide](<https://devfeed.tech/articles/how-to-build-a-data-pipeline-a-step-by-step-guide-140200.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/build-a-data-pipeline>)

Author: Fivetran

Published: 2026-11-18T04:57:01Z

Content type: tutorial

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [elt](<https://devfeed.tech/topics/elt.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [automation](<https://devfeed.tech/tags/automation.md>), [batch](<https://devfeed.tech/tags/batch.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [cloud-based](<https://devfeed.tech/tags/cloud-based.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [how-to](<https://devfeed.tech/tags/how-to.md>)

### AI overview

A data pipeline moves data from sources through ingestion, processing, storage and workflows, with monitoring to catch system and data quality problems. The guide recommends inventorying sources, choosing batch or streaming ingestion based on needs, validating and transforming data, selecting destinations such as a warehouse, lake or lakehouse, and monitoring health, anomalies, alerts and logs. It contrasts ETL, which transforms before storage and can support masking before loading but becomes harder to scale as sources grow, with ELT, which loads first for faster availability and easier cloud scaling but can analyze large volumes more slowly. It also highlights automation, performance, reliability, scalability and security; ELT can still protect sensitive data through isolation, encryption and blocking or hashing before storage. The excerpt concludes that many organizations buy managed pipelines because building and maintaining connectors and components is labor-intensive.

### Source excerpt

Learn what a data pipeline is and how to build one.

[Next page](<https://devfeed.tech/topics/data-engineering.md?cursor=WyIyMDI2LTExLTE4VDA0OjU3OjAxKzAwOjAwIiwgIjZjNWY3MTY1LWJlN2EtNDcyZC1hNjg2LTY1N2ExYWYxYmM1NSJd>)