# DataOps

DataOps is a set of principles for developing and delivering analytics, including the orchestration of data, tools, code, and environments.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## DataPraxis: Scaling Analytics Tasks via Verified Workspace Evolution

DevFeed: [DataPraxis: Scaling Analytics Tasks via Verified Workspace Evolution](<https://devfeed.tech/articles/datapraxis-scaling-analytics-tasks-via-verified-workspace-evolution-142581.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.26924>)

Author: Nan Jia, Wenxuan Shi, Jingqing Ruan, Zhizhao Zeng, Tianyi Zhai, Zhongda Su, Feng Hong, Zhengyu Chen, Wenhao Yan, Haojin Yang, Siqi Dai, Weipeng Zhang, Ke Zeng, Xunliang Cai

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [Frontier AI](<https://devfeed.tech/topics/frontier-ai.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [assembly](<https://devfeed.tech/tags/assembly.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [dabstep](<https://devfeed.tech/tags/dabstep.md>), [data](<https://devfeed.tech/tags/data.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [domains](<https://devfeed.tech/tags/domains.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [formats](<https://devfeed.tech/tags/formats.md>), [framework](<https://devfeed.tech/tags/framework.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [frontier-models](<https://devfeed.tech/tags/frontier-models.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [workspace](<https://devfeed.tech/tags/workspace.md>)

### AI overview

DataPraxis synthesizes analytics tasks by progressively enriching workspaces with validated findings and artifacts, then inheriting and combining workspaces to surface deeper questions. It builds tasks from accumulated findings and their dependencies, checking analytical results during workspace evolution and again after task assembly. The resulting DataPraxis-5K dataset contains 5,000 verified trajectories across domains and output formats and was used to train DataPraxis-27B and DataPraxis-35B-A3B. The abstract reports DataPraxis-27B scores of 55.11 on DABStep and 51.9 on JobBench, matching or surpassing frontier models such as GLM-5.2.

### Source excerpt

arXiv:2610.26924v3 Announce Type: replace Abstract: General-purpose agents are increasingly expected to perform professional analytics work, which requires sustained reasoning, iterative investigation, and reliable evidence. Training these capabilities requires diverse and challenging tasks. However, generating tasks directly from raw data often overlooks deeper questions that emerge through analysis and provides limited evidence for verification. To address these limitations, we introduce DataPraxis, an analytics task synthesis framework based on verified workspace evolution. The framework progressively enriches workspaces with validated findings and artifacts, then inherits and combines these workspaces to uncover deeper questions. It constructs tasks from accumulated findings and their dependencies, verifying analytical results both during evolution and after task assembly. Using this framework, we construct DataPraxis-5K, a dataset of 5,000 verified trajectories spanning diverse domains and output formats, to train DataPraxis-27B and DataPraxis-35B-A3B. DataPraxis-27B achieves 55.11 on DABStep and 51.9 on JobBench, matching or surpassing frontier models such as GLM-5.2. DataPraxis-5K is available at huggingface.co/datasets/stargazerhust/DataPraxis-5k

## How to Build a Modern Data Stack From the Ground Up | Blog | Fivetran

DevFeed: [How to Build a Modern Data Stack From the Ground Up | Blog | Fivetran](<https://devfeed.tech/articles/how-to-build-a-modern-data-stack-from-the-ground-up-blog-fivetran-141324.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/how-to-build-a-modern-data-stack-from-the-ground-up>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: tutorial

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [elt](<https://devfeed.tech/topics/elt.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [Data serialization](<https://devfeed.tech/topics/data-serialization.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [bi-tools](<https://devfeed.tech/tags/bi-tools.md>), [business-intelligence](<https://devfeed.tech/tags/business-intelligence.md>), [business-logic](<https://devfeed.tech/tags/business-logic.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [data](<https://devfeed.tech/tags/data.md>), [data-driven](<https://devfeed.tech/tags/data-driven.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-platforms](<https://devfeed.tech/tags/data-platforms.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [data-stack](<https://devfeed.tech/tags/data-stack.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [decision-making](<https://devfeed.tech/tags/decision-making.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [modern-data-stack](<https://devfeed.tech/tags/modern-data-stack.md>), [plan](<https://devfeed.tech/tags/plan.md>)

### AI overview

Build the stack around a defined analytics use case: identify the data needed, then choose an ingestion tool that connects to those sources. Centralize data in a warehouse and consider a data lake for historical data; use transformation tools such as dbt with connectors such as Fivetran to prepare data for reporting. Keep business logic consistent and models representative of the real world. Choose a BI tool based on the tradeoff that matters most, such as usability or scalability, and consult the technical team to maintain a cohesive strategy. The article emphasizes data quality throughout: validate at ingestion and transformation stages, test continuously, and promote to production only when confident the data will not mislead users.

### Source excerpt

A data architecture expert gives a crash course on moving analytics to the cloud -- including a step-by-step plan to ensure success.

## The Modern Data Stack in 2025 | Blog | Fivetran

DevFeed: [The Modern Data Stack in 2025 | Blog | Fivetran](<https://devfeed.tech/articles/the-modern-data-stack-in-2025-blog-fivetran-141342.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/the-modern-data-stack-in-2025>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>)

Tags: [2025](<https://devfeed.tech/tags/2025.md>), [aggregates](<https://devfeed.tech/tags/aggregates.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [agile](<https://devfeed.tech/tags/agile.md>), [analysts](<https://devfeed.tech/tags/analysts.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [application](<https://devfeed.tech/tags/application.md>), [applications](<https://devfeed.tech/tags/applications.md>), [business](<https://devfeed.tech/tags/business.md>), [cases](<https://devfeed.tech/tags/cases.md>), [ceo](<https://devfeed.tech/tags/ceo.md>), [co-founder](<https://devfeed.tech/tags/co-founder.md>), [conference](<https://devfeed.tech/tags/conference.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [continuous-integration](<https://devfeed.tech/tags/continuous-integration.md>), [data](<https://devfeed.tech/tags/data.md>), [data-stack](<https://devfeed.tech/tags/data-stack.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [modern](<https://devfeed.tech/tags/modern.md>), [modern-data-stack](<https://devfeed.tech/tags/modern-data-stack.md>)

### AI overview

Fivetran CEO George Fraser expects the modern data stack to evolve into a hybrid analytical-operational system, with data warehouses supporting both traditional aggregate analysis and operational applications. As warehouse workloads expand, he anticipates more production use cases served directly from warehouses and interfaces for exploring individual entities and their relationships. Analytics teams operating production workloads will increasingly need engineering practices such as testing and continuous integration. Fraser also says organizations will need to serve both highly data-literate business users who can self-serve and users with other strengths. These are his expectations for the coming years, not reported outcomes.

### Source excerpt

Fivetran CEO George Fraser contemplates how analytics technology is likely to evolve over the next five years.

## Agenda Released for the Modern Data Stack Conference 2021 | Blog | Fivetran

DevFeed: [Agenda Released for the Modern Data Stack Conference 2021 | Blog | Fivetran](<https://devfeed.tech/articles/agenda-released-for-the-modern-data-stack-conference-2021-blog-fivetran-141297.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/agenda-released-for-the-modern-data-stack-conference-2021>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: news

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [Data visualization](<https://devfeed.tech/topics/data-visualization.md>)

Tags: [2021](<https://devfeed.tech/tags/2021.md>), [ali-ghodsi](<https://devfeed.tech/tags/ali-ghodsi.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [aws](<https://devfeed.tech/tags/aws.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [conference](<https://devfeed.tech/tags/conference.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analytics](<https://devfeed.tech/tags/data-analytics.md>), [data-driven](<https://devfeed.tech/tags/data-driven.md>), [data-engineers](<https://devfeed.tech/tags/data-engineers.md>), [data-lakes](<https://devfeed.tech/tags/data-lakes.md>), [data-stack](<https://devfeed.tech/tags/data-stack.md>), [data-visualization](<https://devfeed.tech/tags/data-visualization.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [modern-data-stack](<https://devfeed.tech/tags/modern-data-stack.md>), [sessions](<https://devfeed.tech/tags/sessions.md>), [virtual](<https://devfeed.tech/tags/virtual.md>)

### AI overview

The free Modern Data Stack Conference 2021 runs September 22-23, with 40+ sessions across four virtual stages. Its focus is transformation in analytics teams, including the insights and tools enabled and practices for driving data-informed work. Topics span the future of the modern data stack, avoiding analysis paralysis, choosing a stack for company type and growth stage, security and other stack best practices, and supporting different departments. The opening keynote features Fivetran CEO George Fraser and Databricks CEO Ali Ghodsi on how analytics teams and tools are changing. Day two includes NYU professor Jonathan Haidt on using moral psychology to support data-driven presentations.

### Source excerpt

This year's conference includes 40+ sessions across four virtual stages and features some of the brightest minds in the data community.

## How to Modernize Your Analytics Department | Blog | Fivetran

DevFeed: [How to Modernize Your Analytics Department | Blog | Fivetran](<https://devfeed.tech/articles/how-to-modernize-your-analytics-department-blog-fivetran-141328.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/how-to-modernize-your-analytics-department>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>)

Tags: [advice](<https://devfeed.tech/tags/advice.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [automate](<https://devfeed.tech/tags/automate.md>), [basics](<https://devfeed.tech/tags/basics.md>), [benefits](<https://devfeed.tech/tags/benefits.md>), [blog](<https://devfeed.tech/tags/blog.md>), [business](<https://devfeed.tech/tags/business.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [company](<https://devfeed.tech/tags/company.md>), [company-culture](<https://devfeed.tech/tags/company-culture.md>), [conference](<https://devfeed.tech/tags/conference.md>), [culture](<https://devfeed.tech/tags/culture.md>), [customer](<https://devfeed.tech/tags/customer.md>), [data](<https://devfeed.tech/tags/data.md>), [department](<https://devfeed.tech/tags/department.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [modern](<https://devfeed.tech/tags/modern.md>), [modernize](<https://devfeed.tech/tags/modernize.md>)

### AI overview

The leaders' central recommendation is to build the fundamentals before pursuing advanced analytics: establish data governance, documentation and ownership, and treat data as a product that business teams can use themselves. Rankin says poor governance left the Guardian with an undocumented "data swamp." To gain support, secure buy-in from users and stakeholders, start with small projects, and track KPIs such as platform usage; Penguin Random House saw usage rise after moving from an outsourced solution to its own stack, driven largely by visualization tools. The leaders also recommend building data literacy through accessible decision-focused conversations, workshops and regular engagement. Rashedi cautions against rushing into machine learning before the organization is ready. Williams identifies automation and rapid scaling as modern data stacks' advantage over on-premises systems and recommends moving to cloud where possible.

### Source excerpt

Three leading data practitioners share advice on building a modern analytics practice -- including how to avoid costly missteps.

## Trends that will dominate analytics & BI in 2022 | Blog | Fivetran

DevFeed: [Trends that will dominate analytics & BI in 2022 | Blog | Fivetran](<https://devfeed.tech/articles/trends-that-will-dominate-analytics-bi-in-2022-blog-fivetran-141345.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/trends-that-will-dominate-analytics-bi-2022>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [DataOps](<https://devfeed.tech/topics/dataops.md>), [elt](<https://devfeed.tech/topics/elt.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [etl](<https://devfeed.tech/topics/etl.md>), [databricks](<https://devfeed.tech/topics/databricks.md>)

Tags: [2022](<https://devfeed.tech/tags/2022.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [automation](<https://devfeed.tech/tags/automation.md>), [beam](<https://devfeed.tech/tags/beam.md>), [bi](<https://devfeed.tech/tags/bi.md>), [bi-tools](<https://devfeed.tech/tags/bi-tools.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [boost](<https://devfeed.tech/tags/boost.md>), [building](<https://devfeed.tech/tags/building.md>), [business](<https://devfeed.tech/tags/business.md>), [business-intelligence](<https://devfeed.tech/tags/business-intelligence.md>), [chaos](<https://devfeed.tech/tags/chaos.md>), [dataops](<https://devfeed.tech/tags/dataops.md>), [elt](<https://devfeed.tech/tags/elt.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [managed](<https://devfeed.tech/tags/managed.md>), [trends](<https://devfeed.tech/tags/trends.md>)

### AI overview

The article predicts five shifts in analytics and BI for 2022: ELT pipelines becoming more continuous and fully managed, reducing reliance on brittle batch ETL; DataOps practices and features becoming more central, with dbt Labs cited for SQL-based warehouse transformations, lineage tracking, and version control; operational analytics making data available in frontline tools to support day-to-day decisions; BI tools enabling more collaborative work between data and business teams; and wider use of distributed processing frameworks such as Apache Spark, Beam, Flink, and PySpark. These are presented as forecasts, not measured outcomes.

### Source excerpt

Fully managed ELT, DataOps and more trends that will change the way we use data this year.

## How to Use Fivetran Data Models to Accelerate Analytics | Blog | Fivetran

DevFeed: [How to Use Fivetran Data Models to Accelerate Analytics | Blog | Fivetran](<https://devfeed.tech/articles/how-to-use-fivetran-data-models-to-accelerate-analytics-blog-fivetran-141330.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/how-to-use-fivetran-data-models-to-accelerate-analytics>)

Author: Dayna Shoemaker

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [data-modeling](<https://devfeed.tech/topics/data-modeling.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [business-logic](<https://devfeed.tech/tags/business-logic.md>), [code-modularity](<https://devfeed.tech/tags/code-modularity.md>), [core](<https://devfeed.tech/tags/core.md>), [custom](<https://devfeed.tech/tags/custom.md>), [data-modeling](<https://devfeed.tech/tags/data-modeling.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [financial](<https://devfeed.tech/tags/financial.md>), [financial-reporting](<https://devfeed.tech/tags/financial-reporting.md>), [firefly](<https://devfeed.tech/tags/firefly.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [reporting](<https://devfeed.tech/tags/reporting.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

Fivetran's open-source dbt data models provide maintained SQL that works with data ingested through Fivetran, reducing the work of turning raw schemas into analytics-ready tables. Source packages document and standardize underlying schemas; transform packages create analytics-ready models. For standard reporting such as financial reports, packages may be used directly. Where business logic is more specialized, dbt users can combine them with custom models and schedule dependencies through Fivetran's dbt Core integration. Users who prefer to build transformations themselves can inspect the SQL as a template and contribute code to future package versions.

### Source excerpt

Fivetran pre-built data models package SQL to accelerate analytics with near instant reporting.

## Unleashing the power of Fivetran and Azure Synapse for accelerated data movement | Blog | Fivetran

DevFeed: [Unleashing the power of Fivetran and Azure Synapse for accelerated data movement | Blog | Fivetran](<https://devfeed.tech/articles/unleashing-the-power-of-fivetran-and-azure-synapse-for-accelerated-data-movement-blog-fivetran-140266.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/unleashing-the-power-of-fivetran-and-azure-synapse-for-accelerated-data-movement>)

Author: Kelly Kohlleffel

Published: 2026-11-18T04:00:45Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [data mobility](<https://devfeed.tech/topics/data-mobility.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>), [Azure](<https://devfeed.tech/topics/azure.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>)

Tags: [ai-ml](<https://devfeed.tech/tags/ai-ml.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [azure](<https://devfeed.tech/tags/azure.md>), [azure-synapse](<https://devfeed.tech/tags/azure-synapse.md>), [data](<https://devfeed.tech/tags/data.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>)

### AI overview

Fivetran can move database data into Azure Synapse through managed connectors, handling historical loads and incremental change capture while automatically replicating schema changes and recovering from failures. In the Oracle RDS example, setup requires source credentials, a connection method, an update method, and selected schemas and tables; the initial historical sync reportedly finished in under a minute. Standard connectors offer Log Miner or Teleport sync and can run at intervals as short as five minutes; High Volume Agent (HVA) requires deploying an agent at the source and offers one-minute intervals. HVR also uses a source agent and supports continuous replication, including an initial refresh followed by ongoing change capture. The article presents Synapse Data Studio, SQLPro Studio, and Preset.io as ways to query or visualize the resulting data. It frames Fivetran and Azure Data Factory as tools to evaluate against delivery speed, latency, scale, maintenance, security, usability, and total costs, rather than recommending one tool for every case.

### Source excerpt

Learn how the power of Fivetran and Azure Synapse work to accelerate data movement

## Fivetran gives analysts a new way to transform data | Blog | Fivetran

DevFeed: [Fivetran gives analysts a new way to transform data | Blog | Fivetran](<https://devfeed.tech/articles/fivetran-gives-analysts-a-new-way-to-transform-data-blog-fivetran-140224.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/fivetran-gives-analysts-a-new-way-to-transform-data>)

Author: Fivetran

Published: 2026-11-18T03:07:10Z

Content type: tutorial

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [elt](<https://devfeed.tech/topics/elt.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>)

Tags: [analysts](<https://devfeed.tech/tags/analysts.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [cli](<https://devfeed.tech/tags/cli.md>), [code](<https://devfeed.tech/tags/code.md>), [command-line](<https://devfeed.tech/tags/command-line.md>), [compute-costs](<https://devfeed.tech/tags/compute-costs.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [data-freshness](<https://devfeed.tech/tags/data-freshness.md>), [data-integrity](<https://devfeed.tech/tags/data-integrity.md>), [data-orchestration](<https://devfeed.tech/tags/data-orchestration.md>), [data-teams](<https://devfeed.tech/tags/data-teams.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [platform](<https://devfeed.tech/tags/platform.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>)

### AI overview

Fivetran Transformations adds a UI orchestration layer for dbt Core, letting teams coordinate connector syncs and SQL model runs from the same platform. Its integrated schedule runs transformations after upstream connectors finish syncing and in dependency order; the article says this can improve freshness, avoid unnecessary runs, reduce compute costs, and help keep related tables consistent. Users can instead choose a custom time-based schedule. Initial setup still requires dbt Core, an IDE, version control, a supported destination (BigQuery, Snowflake, Redshift, Databricks, or PostgreSQL), command-line project creation, warehouse credentials, and a populated dbt project. The GUI simplifies ongoing scheduling and monitoring, while dbt development and setup remain code-based.

### Source excerpt

With integrated scheduling, Fivetran Transformations enables data orchestration from a single platform with minimal configuration and code.

## Introducing Fivetran Transformations: Powerful Automated Orchestration | Blog | Fivetran

DevFeed: [Introducing Fivetran Transformations: Powerful Automated Orchestration | Blog | Fivetran](<https://devfeed.tech/articles/introducing-fivetran-transformations-powerful-automated-orchestration-blog-fivetran-140263.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/transformations>)

Author: Fivetran

Published: 2026-11-18T02:47:08Z

Content type: release

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [etl](<https://devfeed.tech/topics/etl.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [automated](<https://devfeed.tech/tags/automated.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [warehouse](<https://devfeed.tech/tags/warehouse.md>)

### AI overview

Fivetran Transformations lets analysts orchestrate SQL transformations inside a data warehouse, with runs triggered by new data loads or a time-based schedule. Fivetran verifies warehouse ingestion and runs the prepared SQL through its scheduling system, avoiding the separate trigger solution the article says was traditionally needed. A dashboard manages reporting logic, and Run History shows current status and earlier attempts. Combined with Fivetran Connectors, it is presented as a way for data teams to manage the pipeline without additional engineering resources. The article describes early access for all customers.

### Source excerpt

Fivetran now provides everything you need for powerful and automated in-warehouse transformations.

## Databricks is a Leader and positioned highest in execution in the 2026 Gartner® Magic Quadrant™ for Data Lakehouse Platforms

DevFeed: [Databricks is a Leader and positioned highest in execution in the 2026 Gartner® Magic Quadrant™ for Data Lakehouse Platforms](<https://devfeed.tech/articles/databricks-is-a-leader-and-positioned-highest-in-execution-in-the-2026-gartner-magic-quadranttm-for-data-lakehouse-platforms-136417.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/databricks-leader-and-positioned-highest-execution-2026-gartnerr-magic-quadranttm-data>)

Author: Kaitlin Baumgardner; Kevin Balaji; Brianna Helm; Jonathan Keller; Susan Pierce

Published: 2026-11-14T18:15:02Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2026](<https://devfeed.tech/tags/2026.md>), [3](<https://devfeed.tech/tags/3.md>), [about](<https://devfeed.tech/tags/about.md>), [acid](<https://devfeed.tech/tags/acid.md>), [affiliates](<https://devfeed.tech/tags/affiliates.md>), [ai](<https://devfeed.tech/tags/ai.md>), [company](<https://devfeed.tech/tags/company.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [decades](<https://devfeed.tech/tags/decades.md>), [enterprises](<https://devfeed.tech/tags/enterprises.md>), [execution](<https://devfeed.tech/tags/execution.md>), [gartner](<https://devfeed.tech/tags/gartner.md>), [gartner-magic-quadranttm](<https://devfeed.tech/tags/gartner-magic-quadranttm.md>), [lakebase](<https://devfeed.tech/tags/lakebase.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [lakehouse-architecture](<https://devfeed.tech/tags/lakehouse-architecture.md>), [leader](<https://devfeed.tech/tags/leader.md>), [ltap](<https://devfeed.tech/tags/ltap.md>), [magic](<https://devfeed.tech/tags/magic.md>), [operational](<https://devfeed.tech/tags/operational.md>), [platforms](<https://devfeed.tech/tags/platforms.md>)

### AI overview

Databricks argues that AI workloads need operational and analytical data on one governed foundation. Its Lake Transactional/Analytical Processing (LTAP) architecture pairs Lakebase Postgres for ACID transactions with lakehouse engines for analytics, both reading one durable copy in open formats such as Delta Lake and Apache Iceberg under Unity Catalog. The company says this avoids pipelines and mirrored tables while giving analytics fresh application data without slowing transactions. Lakebase Postgres uses separated, serverless compute and storage on cloud object storage, scales compute to zero, and supports instant database branching. Databricks cites PRADA Group's reported KPI delivery at about 130 times faster and infrastructure costs about 50% lower. It also says Gartner named Databricks a Leader with the highest execution scores in its 2026 Data Lakehouse Platforms Magic Quadrant and the only Visionary in its Cloud Operational Database Management Systems Magic Quadrant. These are vendor-reported claims; the text includes Gartner's disclaimer that its research is not an endorsement.

### Source excerpt

AI is pulling analytical and operational data togetherFor decades, enterprises have...

## PostgreSQL + ClickHouse Recovery: Test the Whole Data Service

DevFeed: [PostgreSQL + ClickHouse Recovery: Test the Whole Data Service](<https://devfeed.tech/articles/postgresql-clickhouse-recovery-test-the-whole-data-service-132583.md>)

Original publisher: [Read original article](<https://severalnines.com/blog/postgresql-clickhouse-recovery-test-the-whole-data-service/>)

Author: Jessie Baynas

Published: 2026-11-13T11:47:17Z

Content type: tutorial

Language: en

Sources: [SeveralNines](<https://devfeed.tech/sources/severalnines.md>)

Topics: [Disaster Recovery](<https://devfeed.tech/topics/disaster-recovery.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Synthetic Data Generation](<https://devfeed.tech/topics/synthetic-data-generation.md>)

Tags: [acceptance](<https://devfeed.tech/tags/acceptance.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [application](<https://devfeed.tech/tags/application.md>), [backup](<https://devfeed.tech/tags/backup.md>), [backup-management](<https://devfeed.tech/tags/backup-management.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [clustercontrol](<https://devfeed.tech/tags/clustercontrol.md>), [data](<https://devfeed.tech/tags/data.md>), [debezium](<https://devfeed.tech/tags/debezium.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [modern-data-stack-operations](<https://devfeed.tech/tags/modern-data-stack-operations.md>), [offset](<https://devfeed.tech/tags/offset.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [reconciliation](<https://devfeed.tech/tags/reconciliation.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [restoration](<https://devfeed.tech/tags/restoration.md>), [restore](<https://devfeed.tech/tags/restore.md>), [service](<https://devfeed.tech/tags/service.md>), [test](<https://devfeed.tech/tags/test.md>), [updates](<https://devfeed.tech/tags/updates.md>)

### AI overview

A green restore and zero Kafka lag do not prove that ClickHouse analytics are correct. In this PostgreSQL-Debezium-Kafka-ClickHouse exercise, the team records the backup's last applied Kafka offset, rewinds the consumer group before restoring, then waits for a source-backed, stable recovery boundary. It reconciles PostgreSQL and ClickHouse against independently generated expected results, checking keys, values, updates, deletes, outage-window inserts, and the dashboard query. ReplacingMergeTree checks use FINAL or version-based selection and filter delete markers; LSN versions make replayed events idempotent. The walkthrough's negative test skips the rewind: ingestion resumes and lag reaches zero, yet pre-outage updates and a delete are missing. PostgreSQL can rebuild ClickHouse only while it retains all needed rows; purged history depends on ClickHouse backups and retained Kafka events. The lab timings are not production targets, and PostgreSQL loss, Kafka loss, node loss with on-node backups, and retained-history recovery are outside the tested sequence.

### Source excerpt

A successful restore does not prove that an analytics service is delivering correct data. This PostgreSQL + ClickHouse walkthrough uses Debezium and Kafka to test backup restoration, offset replay, updates, deletes, and application-level reconciliation--including a negative test that catches a seemingly healthy but incomplete recovery. The post PostgreSQL + ClickHouse Recovery: Test the Whole Data Service appeared first on Severalnines.

## Top 10 IT certifications in demand today

DevFeed: [Top 10 IT certifications in demand today](<https://devfeed.tech/articles/top-10-it-certifications-in-demand-today-130038.md>)

Original publisher: [Read original article](<https://www.cio.com/article/193586/top-15-it-certifications-in-demand-today.html>)

Author: CIO

Published: 2026-11-12T09:30:00Z

Content type: article

Language: en

Sources: [CIO](<https://devfeed.tech/sources/cio.md>)

Topics: [Learning](<https://devfeed.tech/topics/learning.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Cryptography](<https://devfeed.tech/topics/cryptography.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-skills](<https://devfeed.tech/tags/ai-skills.md>), [boost](<https://devfeed.tech/tags/boost.md>), [career](<https://devfeed.tech/tags/career.md>), [careers](<https://devfeed.tech/tags/careers.md>), [careers-certifications-it-skills-and-training](<https://devfeed.tech/tags/careers-certifications-it-skills-and-training.md>), [certifications](<https://devfeed.tech/tags/certifications.md>), [course](<https://devfeed.tech/tags/course.md>), [demand](<https://devfeed.tech/tags/demand.md>), [edition-us](<https://devfeed.tech/tags/edition-us.md>), [education](<https://devfeed.tech/tags/education.md>), [exam](<https://devfeed.tech/tags/exam.md>), [experience](<https://devfeed.tech/tags/experience.md>), [help](<https://devfeed.tech/tags/help.md>), [industry](<https://devfeed.tech/tags/industry.md>), [it](<https://devfeed.tech/tags/it.md>), [it-skills-and-training](<https://devfeed.tech/tags/it-skills-and-training.md>), [job-search](<https://devfeed.tech/tags/job-search.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [language-en](<https://devfeed.tech/tags/language-en.md>), [learn](<https://devfeed.tech/tags/learn.md>), [resume](<https://devfeed.tech/tags/resume.md>), [salaries](<https://devfeed.tech/tags/salaries.md>), [salary](<https://devfeed.tech/tags/salary.md>), [storytype-feature](<https://devfeed.tech/tags/storytype-feature.md>), [top](<https://devfeed.tech/tags/top.md>)

### AI overview

The excerpt presents certifications as a way to validate or build IT skills, strengthen a résumé, and potentially improve job prospects and pay. It cites Robert Half survey figures indicating that technology leaders expect higher offers for specialized skills, especially AI expertise, and lists ten credentials spanning cloud, security, data, support, AI, Azure architecture, auditing, and project management. Requirements and costs vary: AWS Cloud Practitioner recommends basic IT knowledge and six months of AWS experience ($100); CCSP recommends substantial IT and security experience ($599); CISA requires five years in IS audit, control, or security ($575 for members, $760 for nonmembers); and CompTIA A+ requires two exams, with 9-12 months' hands-on experience recommended ($274). Other examples include vendor-neutral CompTIA Cloud+ for cloud infrastructure ($399), a free IBM AI Engineering course series, and Azure Solutions Architect credentials with associate and expert levels ($165). The excerpt also gives salary estimates from ZipRecruiter or Payscale, which are reported averages rather than guaranteed outcomes.

### Source excerpt

What are IT certifications? IT certifications are professional designations offered by organizations and companies to help you demonstrate to employers your qualifications in specific skills, tools, and IT areas. To gain certifications, you'll need to pass an exam, along with any other requirements set by the accrediting body. Gaining the right IT certifications for your role and industry can help you advance your career. Certifications give you the opportunity to earn a credential in skills you already have or to grow your competencies by developing new skills you'd like to use in your career. Whatever your reason for getting certified, certifications are a great way to strengthen your resume, set yourself apart from other candidates in a job search, and boost your salary. The significance of IT certifications IT certifications not only help solidify the skills on your resume, but they can also offer education and hands-on experience. Many certifications require completing a course or training program, allowing you to learn new skills and hone existing ones. This can be a great opportunity for entry-level workers who need to bulk up their resumes or for more seasoned IT pros to set themselves apart from other candidates by validating their expertise and skills. While some certifications consist of only an exam, many teach you new concepts, ideas, and expand your knowledge prior to certifying your skills and abilities. This can give you the opportunity to not only obtain an industry-recognized designation but also help you gain new skills and expertise for your resume. IT certifications can boost salary in 2027 Overall, 72% of technology leaders say they're planning to make more competitive offers for IT professionals in the new year, with 65% willing to offer salaries higher than the planned range to "secure candidates with specialized skills." In 2027, 92% of IT leaders plan to offer higher salaries for relevant AI skills, and 60% plan to offer "more for AI-relate

## Fivetran Recognized as a Leader in the 2026 Gartner® Magic Quadrant™ for Data Integration Tools | Blog | Fivetran

DevFeed: [Fivetran Recognized as a Leader in the 2026 Gartner® Magic Quadrant™ for Data Integration Tools | Blog | Fivetran](<https://devfeed.tech/articles/fivetran-recognized-as-a-leader-in-the-2026-gartner-magic-quadranttm-for-data-integration-tools-blog-fivetran-136774.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/fivetran-recognized-as-a-leader-in-the-2026-gartner-r-magic-quadrant-tm-for-data-integration-tools>)

Author: Anjan Kundavaram

Published: 2026-11-11T23:21:52Z

Content type: release

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [DataOps](<https://devfeed.tech/topics/dataops.md>), [AI-Ready Data](<https://devfeed.tech/topics/ai-ready-data.md>), [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [elt](<https://devfeed.tech/topics/elt.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2026](<https://devfeed.tech/tags/2026.md>), [3](<https://devfeed.tech/tags/3.md>), [ai](<https://devfeed.tech/tags/ai.md>), [blog](<https://devfeed.tech/tags/blog.md>), [data](<https://devfeed.tech/tags/data.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [enterprise-ai](<https://devfeed.tech/tags/enterprise-ai.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [gartner](<https://devfeed.tech/tags/gartner.md>), [gartner-magic-quadranttm](<https://devfeed.tech/tags/gartner-magic-quadranttm.md>), [integration](<https://devfeed.tech/tags/integration.md>), [labs](<https://devfeed.tech/tags/labs.md>), [leader](<https://devfeed.tech/tags/leader.md>), [magic](<https://devfeed.tech/tags/magic.md>), [tools](<https://devfeed.tech/tags/tools.md>), [trusted-data](<https://devfeed.tech/tags/trusted-data.md>)

### AI overview

Fivetran says it was named a Leader in Gartner's 2026 Magic Quadrant for Data Integration Tools for the seventh consecutive year, with improved Ability to Execute and Completeness of Vision. Following its merger with dbt Labs, the company presents a unified workflow for moving data, building and testing transformations, defining shared business metrics, and making context available to analytics and AI. It cites more than 750 managed connectors, a Connector SDK, Activations, dbt modeling and semantic capabilities, a Context Layer, and dbt State, which uses change awareness to avoid unnecessary transformation work. AI features include dbt Wizard for model generation and refactoring, an agent that builds connectors from URLs and API documentation, and coding-agent plugins for custom connectors. The article emphasizes reusable pipelines and definitions, open standards, and customer choice of platforms and tools, while describing reduced maintenance and compute costs as goals. It is a company account of Gartner recognition; Gartner's disclaimer says its research reflects opinions and does not endorse vendors.

### Source excerpt

Fivetran + dbt Labs is the trusted data foundation for enterprise AI.

## The dawn of the Data Intelligence Engineer

DevFeed: [The dawn of the Data Intelligence Engineer](<https://devfeed.tech/articles/the-dawn-of-the-data-intelligence-engineer-130712.md>)

Original publisher: [Read original article](<https://motherduck.com/blog/the-data-intelligence-engineer>)

Author: Jordan Tigani

Published: 2026-11-11T00:00:00Z

Content type: opinion

Language: en

Sources: [MotherDuck](<https://devfeed.tech/sources/motherduck.md>)

Topics: [AI for data automation](<https://devfeed.tech/topics/ai-for-data-automation.md>), [business-intelligence](<https://devfeed.tech/topics/business-intelligence.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>), [AI-Ready Data](<https://devfeed.tech/topics/ai-ready-data.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [bi-tools](<https://devfeed.tech/topics/bi-tools.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>)

Tags: [2025](<https://devfeed.tech/tags/2025.md>), [about](<https://devfeed.tech/tags/about.md>), [accountability](<https://devfeed.tech/tags/accountability.md>), [act](<https://devfeed.tech/tags/act.md>), [admin](<https://devfeed.tech/tags/admin.md>), [age-of-ai](<https://devfeed.tech/tags/age-of-ai.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-evals](<https://devfeed.tech/tags/agent-evals.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [can](<https://devfeed.tech/tags/can.md>), [data](<https://devfeed.tech/tags/data.md>), [data-teams](<https://devfeed.tech/tags/data-teams.md>), [engineer](<https://devfeed.tech/tags/engineer.md>), [intelligence](<https://devfeed.tech/tags/intelligence.md>), [motherduck](<https://devfeed.tech/tags/motherduck.md>), [next](<https://devfeed.tech/tags/next.md>), [teams](<https://devfeed.tech/tags/teams.md>), [this](<https://devfeed.tech/tags/this.md>), [work](<https://devfeed.tech/tags/work.md>)

### AI overview

The proposed Data Intelligence Engineer shifts data work from producing every query and dashboard to making agent-driven self-service reliable. The role maintains clear data models and shared business context, builds frameworks, monitors systems, and uses known-answer evaluations to check agent results. Human accountability remains essential because agents may misinterpret company-specific definitions or select the wrong data. The author recommends supervision proportional to an output's impact: the data team can leave low-stakes experiments largely open, make a best effort to fix important curated dashboards and apps, and keep core metrics highly reliable. In the author's MotherDuck example, Flights handle ingestion pipelines, Guides provide shared model and business context, Dives are agent-created visualizations, and each user or agent works in a dedicated DuckDB instance called a Duckling; these are reachable through SQL to reduce tool handoffs. The excerpt presents self-service as a way for stakeholders to ask follow-up questions and make timely decisions, while acknowledging the risk of more people producing errors.

### Source excerpt

For Data Teams, AI has changed everything. The traditional work can be run by agents, leaving folks wondering "what's next"? In this blog, MotherDuck CEO Jordan Tigani unpacks the possibilities.

## AI's Efficiency Challenge For The Semiconductor Industry

DevFeed: [AI's Efficiency Challenge For The Semiconductor Industry](<https://devfeed.tech/articles/ai-s-efficiency-challenge-for-the-semiconductor-industry-126570.md>)

Original publisher: [Read original article](<https://semiengineering.com/ais-efficiency-challenge-for-the-semiconductor-industry/>)

Author: John Kibarian

Published: 2026-11-10T08:03:58Z

Content type: article

Language: en

Sources: [Semiconductor Engineering](<https://devfeed.tech/sources/semiconductor-engineering.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [Chip design](<https://devfeed.tech/topics/chip-design.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>)

Tags: [2025](<https://devfeed.tech/tags/2025.md>), [2026](<https://devfeed.tech/tags/2026.md>), [3d](<https://devfeed.tech/tags/3d.md>), [3d-ic](<https://devfeed.tech/tags/3d-ic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [blogs-ptes](<https://devfeed.tech/tags/blogs-ptes.md>), [bringing-ai-to-scale](<https://devfeed.tech/tags/bringing-ai-to-scale.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [collaboration](<https://devfeed.tech/tags/collaboration.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analytics](<https://devfeed.tech/tags/data-analytics.md>), [design](<https://devfeed.tech/tags/design.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [energy-efficiency](<https://devfeed.tech/tags/energy-efficiency.md>), [first](<https://devfeed.tech/tags/first.md>), [industry](<https://devfeed.tech/tags/industry.md>), [llms](<https://devfeed.tech/tags/llms.md>), [models](<https://devfeed.tech/tags/models.md>), [pdf-solutions](<https://devfeed.tech/tags/pdf-solutions.md>), [post](<https://devfeed.tech/tags/post.md>), [semiconductor](<https://devfeed.tech/tags/semiconductor.md>), [semiconductor-industry](<https://devfeed.tech/tags/semiconductor-industry.md>), [silicon](<https://devfeed.tech/tags/silicon.md>), [supply-chains](<https://devfeed.tech/tags/supply-chains.md>), [test](<https://devfeed.tech/tags/test.md>), [test-measurement-and-analytics](<https://devfeed.tech/tags/test-measurement-and-analytics.md>), [tools](<https://devfeed.tech/tags/tools.md>), [used](<https://devfeed.tech/tags/used.md>)

### AI overview

AI is increasing demand for chips while pushing manufacturers to get more output and insight from existing tools, data and compute. The keynote describes three pressures: energy efficient custom designs are multiplying products and widening test workloads; manufacturing data is vast and company specific, which general purpose LLMs lack context to interpret reliably; and repeated LLM analysis can be costly. PDF Solutions' described approach uses Python workflows for repeatable analytics, with LLMs to create workflows and customer data kept isolated from the model. It also combines characterization and test analytics, process control, equipment connectivity and secureWISE for controlled movement of data, rules and algorithms across supply chain sites. The excerpt says secureWISE uses double encryption and lets factory teams control what leaves and enters, with virus scanning and audit trails. Performance and scale figures are presented as company claims.

### Source excerpt

AI is reshaping the tools, data, and collaboration models used to design, manufacture, and test silicon. The post AI's Efficiency Challenge For The Semiconductor Industry appeared first on Semiconductor Engineering.

## Fivetran Recognized as a Leader in the 2026 Gartner® Magic Quadrant™ for Data Integration Tools | Blog | Fivetran

DevFeed: [Fivetran Recognized as a Leader in the 2026 Gartner® Magic Quadrant™ for Data Integration Tools | Blog | Fivetran](<https://devfeed.tech/articles/fivetran-recognized-as-a-leader-in-the-2026-gartner-magic-quadranttm-for-data-integration-tools-blog-fivetran-116601.md>)

Original publisher: [Read original article](<https://www.fivetran.com/blog/fivetran-recognized-as-a-leader-in-the-2026-gartner-r-magic-quadrant-tm-for-data-integration-tools>)

Author: Anjan Kundavaram

Published: 2026-11-04T16:15:01Z

Content type: news

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Data Infrastructure](<https://devfeed.tech/topics/data-infrastructure.md>), [AI-Ready Data](<https://devfeed.tech/topics/ai-ready-data.md>), [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [databricks](<https://devfeed.tech/topics/databricks.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2026](<https://devfeed.tech/tags/2026.md>), [3](<https://devfeed.tech/tags/3.md>), [ai](<https://devfeed.tech/tags/ai.md>), [blog](<https://devfeed.tech/tags/blog.md>), [data](<https://devfeed.tech/tags/data.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [enterprise-ai](<https://devfeed.tech/tags/enterprise-ai.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [gartner](<https://devfeed.tech/tags/gartner.md>), [gartner-magic-quadranttm](<https://devfeed.tech/tags/gartner-magic-quadranttm.md>), [integration](<https://devfeed.tech/tags/integration.md>), [labs](<https://devfeed.tech/tags/labs.md>), [leader](<https://devfeed.tech/tags/leader.md>), [magic](<https://devfeed.tech/tags/magic.md>), [tools](<https://devfeed.tech/tags/tools.md>), [trusted-data](<https://devfeed.tech/tags/trusted-data.md>)

### AI overview

Fivetran says it was named a Leader in the 2026 Gartner Magic Quadrant for Data Integration Tools for the seventh consecutive year, with improved positions in Ability to Execute and Completeness of Vision. The article presents the June 2026 merger with dbt Labs as a way to connect data movement, transformation, business definitions, activation, and context for analytics and AI. It highlights 750+ managed connectors, dbt's framework for tested and documented transformations, semantic definitions, a Context Layer for AI, and dbt State's change awareness to avoid unnecessary processing. It also describes AI-assisted tools for dbt modeling and connector development, alongside open standards and customer choice of platforms. The stated aims are trusted, reusable data, lower processing costs, and flexibility as AI tools change; these are company claims and goals, not measured outcomes in the excerpt.

### Source excerpt

Fivetran + dbt Labs is the trusted data foundation for enterprise AI.

## How Procore is powering construction analytics with OpenSharing

DevFeed: [How Procore is powering construction analytics with OpenSharing](<https://devfeed.tech/articles/how-procore-is-powering-construction-analytics-with-opensharing-105667.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/how-procore-powering-construction-analytics-opensharing-0>)

Author: Krishan Kumar

Published: 2026-10-29T00:29:00Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [protocols](<https://devfeed.tech/topics/protocols.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-use](<https://devfeed.tech/tags/ai-use.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [api](<https://devfeed.tech/tags/api.md>), [apis](<https://devfeed.tech/tags/apis.md>), [app](<https://devfeed.tech/tags/app.md>), [application](<https://devfeed.tech/tags/application.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [asia](<https://devfeed.tech/tags/asia.md>), [australia](<https://devfeed.tech/tags/australia.md>), [automated](<https://devfeed.tech/tags/automated.md>), [availability](<https://devfeed.tech/tags/availability.md>), [azure](<https://devfeed.tech/tags/azure.md>), [bi](<https://devfeed.tech/tags/bi.md>), [bi-tools](<https://devfeed.tech/tags/bi-tools.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [budget](<https://devfeed.tech/tags/budget.md>), [business](<https://devfeed.tech/tags/business.md>), [company](<https://devfeed.tech/tags/company.md>), [customers](<https://devfeed.tech/tags/customers.md>), [data](<https://devfeed.tech/tags/data.md>)

### AI overview

Procore replaced its API-driven Analytics 1.0 pipeline, limited to at most three daily refreshes, with Analytics 2.0 built on Databricks and OpenSharing. Curated gold-layer data is shared across regions, with Unity Catalog governance and automated per-user, per-project entitlements. Databricks customers can receive shares natively; others connect through OpenSharing's protocol and client libraries, including from BI tools and other platforms. Recipients can consume full tables or Change Data Feed updates. Procore says the new pipeline expanded its extract catalog to hundreds of tables in less than half the time the first version took to build its baseline, and customers can schedule refreshes according to their tools' capabilities. To manage downstream disruption, it gives six weeks' notice for four annual breaking-change releases and three weeks for twelve non-breaking releases.

### Source excerpt

Connecting an industry built on disconnected dataConstruction runs on data that rarely sits in one place...

## How Procore is powering construction analytics with OpenSharing

DevFeed: [How Procore is powering construction analytics with OpenSharing](<https://devfeed.tech/articles/how-procore-is-powering-construction-analytics-with-opensharing-112964.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/how-procore-powering-construction-analytics-opensharing>)

Author: Krishan Kumar

Published: 2026-10-29T00:29:00Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [protocols](<https://devfeed.tech/topics/protocols.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-use](<https://devfeed.tech/tags/ai-use.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-pipeline](<https://devfeed.tech/tags/analytics-pipeline.md>), [api](<https://devfeed.tech/tags/api.md>), [apis](<https://devfeed.tech/tags/apis.md>), [app](<https://devfeed.tech/tags/app.md>), [application](<https://devfeed.tech/tags/application.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [area](<https://devfeed.tech/tags/area.md>), [asia](<https://devfeed.tech/tags/asia.md>), [australia](<https://devfeed.tech/tags/australia.md>), [company](<https://devfeed.tech/tags/company.md>), [construction](<https://devfeed.tech/tags/construction.md>), [customers](<https://devfeed.tech/tags/customers.md>), [data](<https://devfeed.tech/tags/data.md>), [one](<https://devfeed.tech/tags/one.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Procore Analytics 2.0 replaces a SQL Server pipeline fed by public API extractions--which allowed at most three refreshes per day--with Databricks OpenSharing. Procore cleans, validates, and models data in a multi-region medallion pipeline, then shares gold tables to customer environments. Customers without Databricks connect with an OpenSharing token and client libraries; Databricks customers can receive shares natively. Unity Catalog permissions are reconciled automatically per user and project, and Change Data Feed lets recipients sync only updates or the full dataset. Procore reports expanding its extract catalog to hundreds of tables in less than half the time taken to build the first version, while customers can schedule refreshes to suit their tools. It also sets release notice periods: six weeks for four annual breaking changes and three weeks for twelve non-breaking releases. The source describes the architecture and reported benefits but provides no measured refresh latency or quantified cost comparison.

### Source excerpt

Connecting an industry built on disconnected dataConstruction runs on data that rarely sits in one place...

## Build a Reliable Analytics Pipeline Before Migrating to a Data Warehouse

DevFeed: [Build a Reliable Analytics Pipeline Before Migrating to a Data Warehouse](<https://devfeed.tech/articles/build-a-reliable-analytics-pipeline-before-migrating-to-a-data-warehouse-105467.md>)

Original publisher: [Read original article](<https://www.sitepoint.com/build-a-reliable-analytics-pipeline-before-migrating-to-a-data-warehouse/>)

Author: tbutt4438

Published: 2026-10-28T04:28:05Z

Content type: tutorial

Language: en

Sources: [SitePoint](<https://devfeed.tech/sources/sitepoint.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [etl](<https://devfeed.tech/topics/etl.md>), [metric-standardization](<https://devfeed.tech/topics/metric-standardization.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>)

Tags: [access-controls](<https://devfeed.tech/tags/access-controls.md>), [agile-development](<https://devfeed.tech/tags/agile-development.md>), [ai](<https://devfeed.tech/tags/ai.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-pipeline](<https://devfeed.tech/tags/analytics-pipeline.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [batch](<https://devfeed.tech/tags/batch.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [dashboard](<https://devfeed.tech/tags/dashboard.md>), [data](<https://devfeed.tech/tags/data.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [database](<https://devfeed.tech/tags/database.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [warehouse](<https://devfeed.tech/tags/warehouse.md>)

### AI overview

Build and test a small analytics pipeline before choosing a warehouse: define each metric's date, currency, status and reporting grain, then preserve raw deliveries, resolve versions in staging, and aggregate in reporting models. In the PostgreSQL-compatible example, rank all versions before filtering status so a later refund supersedes an earlier paid state; deduplicate retries and flag conflicting payloads for the same source version. Add failing-row checks for data assumptions, and test late older updates and repeated deliveries. Incremental runs should recompute changed IDs and affected old and new reporting groups, handle deletions by an explicit policy, and advance checkpoints only after outputs commit. Track ingestion, transformation and validation freshness separately; a last validated result with a warning may be safer than serving an unverified total. Measure performance, concurrency, freshness needs and application impact before migration. The excerpt notes that dbt can organize tests and Airflow guidance addresses retryable tasks; neither tool alone ensures source completeness or correct metric definitions.

### Source excerpt

null Continue reading Build a Reliable Analytics Pipeline Before Migrating to a Data Warehouse on SitePoint.

## How KOHO built a Claude-powered self-serve analytics tool on years of dbt governance

DevFeed: [How KOHO built a Claude-powered self-serve analytics tool on years of dbt governance](<https://devfeed.tech/articles/how-koho-built-a-claude-powered-self-serve-analytics-tool-on-years-of-dbt-governance-102640.md>)

Original publisher: [Read original article](<https://www.getdbt.com/blog/claude-self-serve-analytics-dbt-governance>)

Author: Daniel Poppy

Published: 2026-10-27T18:44:00Z

Content type: news

Language: en

Sources: [dbt Labs](<https://devfeed.tech/sources/dbt-labs.md>)

Topics: [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>), [metric-standardization](<https://devfeed.tech/topics/metric-standardization.md>), [Claude](<https://devfeed.tech/topics/claude.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [bedrock](<https://devfeed.tech/tags/bedrock.md>), [claude](<https://devfeed.tech/tags/claude.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [context](<https://devfeed.tech/tags/context.md>), [context-libraries](<https://devfeed.tech/tags/context-libraries.md>), [culture](<https://devfeed.tech/tags/culture.md>), [data](<https://devfeed.tech/tags/data.md>), [data-team](<https://devfeed.tech/tags/data-team.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fintech](<https://devfeed.tech/tags/fintech.md>), [foundations](<https://devfeed.tech/tags/foundations.md>), [governance](<https://devfeed.tech/tags/governance.md>), [learn](<https://devfeed.tech/tags/learn.md>), [self-serve-analytics](<https://devfeed.tech/tags/self-serve-analytics.md>)

### AI overview

KOHO's KoDa lets employees ask analytics questions in Claude and receive answers from certified dbt marts in Redshift, using Bedrock and Claude. KOHO says the tool depended on years of data culture and governance: its dbt foundation has about 1,400 tested and documented models, with PII removed. KoDa is limited to analytics, uses no raw data, is read-only, and applies shared company-wide metric definitions. It connects a markdown-based "KoDa skill" and pre- and post-hook MCP calls through Claude Code to a Redshift MCP server. The team's substantial work went into evaluation: a library of about 375 real analyst questions supports offline quality checks and weekly human review of logged questions. KOHO still reviews self-serve results intended for the board; the speakers identify company-wide data literacy and knowing when to trust an answer as ongoing challenges.

### Source excerpt

KOHO explains why AI self-serve analytics needs a strong data culture and a governed dbt foundation first.

## How DocuSign cut analytics engineering time in half by pairing Claude Code with dbt v2, dbt MCP, and agent skills

DevFeed: [How DocuSign cut analytics engineering time in half by pairing Claude Code with dbt v2, dbt MCP, and agent skills](<https://devfeed.tech/articles/how-docusign-cut-analytics-engineering-time-in-half-by-pairing-claude-code-with-dbt-v2-dbt-mcp-and-agent-skills-101834.md>)

Original publisher: [Read original article](<https://www.getdbt.com/blog/docusign-claude-code-dbt-v2-mcp-agent-skills>)

Author: Daniel Poppy

Published: 2026-10-26T14:04:00Z

Content type: news

Language: en

Sources: [dbt Labs](<https://devfeed.tech/sources/dbt-labs.md>)

Topics: [Claude Code](<https://devfeed.tech/topics/claude-code.md>), [Agent Skills](<https://devfeed.tech/topics/agent-skills.md>), [developer-productivity](<https://devfeed.tech/topics/developer-productivity.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [Pull Request](<https://devfeed.tech/topics/pull-request.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-skills](<https://devfeed.tech/tags/agent-skills.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [ai](<https://devfeed.tech/tags/ai.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-engineering](<https://devfeed.tech/tags/analytics-engineering.md>), [approach](<https://devfeed.tech/tags/approach.md>), [asset](<https://devfeed.tech/tags/asset.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [docusign](<https://devfeed.tech/tags/docusign.md>), [half](<https://devfeed.tech/tags/half.md>), [learn](<https://devfeed.tech/tags/learn.md>), [mcp](<https://devfeed.tech/tags/mcp.md>), [pairing](<https://devfeed.tech/tags/pairing.md>), [productivity](<https://devfeed.tech/tags/productivity.md>), [skills](<https://devfeed.tech/tags/skills.md>), [time](<https://devfeed.tech/tags/time.md>), [v2](<https://devfeed.tech/tags/v2.md>), [workflow](<https://devfeed.tech/tags/workflow.md>)

### AI overview

DocuSign's workflow uses Claude Code to coordinate analytics engineering across three phases: discovery, dbt development, and orchestration. dbt Agent Skills provide project conventions, dbt MCP supplies metadata and lineage, and dbt v2 compiles and validates SQL before a pull request. Phase-specific retries can address failures such as compile errors or skipped unit tests without restarting the full workflow. The demo produced a quarterly aggregation model, YAML, and orchestration job in an isolated branch, with human review required before merge. Gupta reported a 50% productivity gain from requirements to validated, review-ready assets in a pilot, plus more consistent naming, testing, and semantic patterns. Guardrails included capping test runs at 1,000 rows; governance for teams' custom skills was described as early and still evolving.

### Source excerpt

DocuSign wired Claude Code, dbt Agent Skills, dbt MCP, and dbt v2 into one workflow and measured a 50% productivity gain.

## Build a Reliable Analytics Pipeline Before Moving to a Data Warehouse

DevFeed: [Build a Reliable Analytics Pipeline Before Moving to a Data Warehouse](<https://devfeed.tech/articles/build-a-reliable-analytics-pipeline-before-moving-to-a-data-warehouse-92895.md>)

Original publisher: [Read original article](<https://www.sitepoint.com/build-a-reliable-analytics-pipeline-before-moving-to-a-data-warehouse/>)

Author: tbutt4438

Published: 2026-10-21T13:16:13Z

Content type: tutorial

Language: en

Sources: [SitePoint](<https://devfeed.tech/sources/sitepoint.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [metric-standardization](<https://devfeed.tech/topics/metric-standardization.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>)

Tags: [agile-development](<https://devfeed.tech/tags/agile-development.md>), [ai](<https://devfeed.tech/tags/ai.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-pipeline](<https://devfeed.tech/tags/analytics-pipeline.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>)

### AI overview

Define a metric contract and reporting grain before choosing a warehouse. In the example, raw order deliveries are preserved, duplicates and versions are resolved in staging, and reporting includes only each order's latest state. Ranking versions before filtering status prevents a refunded order's older paid version from inflating sales: the example's correct total is 12,500 cents, not 27,500. Add failing-row and behavior tests for conflicting versions, late arrivals, and retries. Incremental pipelines must recompute changed orders and affected old and new reporting groups; deletions need explicit handling. PostgreSQL can prototype the pipeline, dbt can organize maintained SQL models and tests, and Airflow can coordinate dependent jobs and retries. These tools address different responsibilities and do not ensure source completeness. Measure performance, concurrency, freshness, and application impact before migrating, and expose freshness and validation status with dashboard results.

### Source excerpt

null Continue reading Build a Reliable Analytics Pipeline Before Moving to a Data Warehouse on SitePoint.

## NousDB - A Fast Query Engine for Semantic Analytics

DevFeed: [NousDB - A Fast Query Engine for Semantic Analytics](<https://devfeed.tech/articles/nousdb-a-fast-query-engine-for-semantic-analytics-93879.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.28551>)

Author: Christos Tsapelas (Department of Informatics and Telecommunications, National and Kapodistrian University of Athens, Archimedes, Athena Research Center), Georgia Koutrika (Archimedes, Athena Research

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [algebra](<https://devfeed.tech/tags/algebra.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [based](<https://devfeed.tech/tags/based.md>), [batch](<https://devfeed.tech/tags/batch.md>), [batching](<https://devfeed.tech/tags/batching.md>), [complex](<https://devfeed.tech/tags/complex.md>), [context](<https://devfeed.tech/tags/context.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [engine](<https://devfeed.tech/tags/engine.md>), [fast](<https://devfeed.tech/tags/fast.md>), [query](<https://devfeed.tech/tags/query.md>), [query-engine](<https://devfeed.tech/tags/query-engine.md>), [semantic](<https://devfeed.tech/tags/semantic.md>)

### AI overview

NousDB, built on Apache DataFusion, aims to make semantic analytics more scalable and optimizable. It adds three semantic operators with formal algebraic properties for query rewriting and cross-operator optimization. A cost model uses equi-depth token histograms to choose batch sizes within context-window limits, with statistical guarantees for safe batching. The abstract reports up to 86x speedups with local and enterprise LLMs and says NousDB leads competitors in token efficiency and accuracy; gains also extend to JOB and Stack workloads augmented with semantic operators. No specific measurements, competitor names, or implementation details are provided in the supplied excerpt.

### Source excerpt

arXiv:2610.28551v1 Announce Type: new Abstract: Semantic data analytics systems integrate AI-powered operators into query pipelines, enabling declarative queries over structured and unstructured data. However, existing systems rely on per-tuple model invocations, ad hoc operator designs, and lack principled cost models, resulting in poor scalability and limited optimization. We present NousDB, a high-performance semantic analytics engine built on Apache DataFusion\cite{10.1145/3626246.3653368}.Through NousDB we propose an extended relational algebra with three semantic operators and formal algebraic properties that enable principled query rewriting and cross-operator optimization. We further propose a token-based cost model using equi-depth token histograms to optimize batch sizing under context-window constraints and provide statistical guarantees for safe batching. Experiments show that NousDB achieves up to $\times 86$ speedups with both local and enterprise LLMs, while leading competitors in token efficiency and accuracy. These gains extend to complex JOB and Stack workloads augmented with semantic operators, supporting both traditional and semantic analytics.

[Next page](<https://devfeed.tech/topics/dataops.md?cursor=WyIyMDI2LTEwLTIxVDA0OjAwOjAwKzAwOjAwIiwgIjdlOWM3ZmNjLWI4YjEtNDdiOC1hY2IwLTQ3MzFmYmZhYzFhMCJd>)