# Analytics Pipeline

An analytics pipeline is a computing architecture that collects, stores, processes, analyzes, and visualizes large volumes of data from varied sources.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## NetBox Analytics: Get Insight from Your System of Record without Manual Effort

DevFeed: [NetBox Analytics: Get Insight from Your System of Record without Manual Effort](<https://devfeed.tech/articles/netbox-analytics-get-insight-from-your-system-of-record-without-manual-effort-146905.md>)

Original publisher: [Read original article](<https://netboxlabs.com/blog/netbox-analytics-ga/>)

Author: Josh Chase

Published: 2026-11-19T21:50:51Z

Content type: release

Language: en

Sources: [NetBox Labs](<https://devfeed.tech/sources/netbox-labs.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [NetBox](<https://devfeed.tech/topics/netbox.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>), [etl](<https://devfeed.tech/topics/etl.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-pipeline](<https://devfeed.tech/tags/analytics-pipeline.md>), [api](<https://devfeed.tech/tags/api.md>), [blog](<https://devfeed.tech/tags/blog.md>), [budget](<https://devfeed.tech/tags/budget.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [cios](<https://devfeed.tech/tags/cios.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [etl](<https://devfeed.tech/tags/etl.md>), [governance](<https://devfeed.tech/tags/governance.md>), [grafana](<https://devfeed.tech/tags/grafana.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [history](<https://devfeed.tech/tags/history.md>), [ipam](<https://devfeed.tech/tags/ipam.md>), [kentik](<https://devfeed.tech/tags/kentik.md>), [netbox](<https://devfeed.tech/tags/netbox.md>), [system-of-record](<https://devfeed.tech/tags/system-of-record.md>)

### AI overview

NetBox Analytics provides 16 pre-built dashboards for IP, rack, power, port, and hardware lifecycle capacity questions, with changes reflected within minutes. NetBox Labs operates the data pipeline and warehouse, so teams do not need to build or maintain ETL; engineers and managers can use the same NetBox record for detailed and rolled-up views. The Essentials bundle has eight single-domain dashboards, while Complete adds eight cross-domain views. Four dashboards were added at general availability, including two that require specific plugins; two preview dashboards were renamed or enhanced, with Audit Evidence adding field-level change details. It is a NetBox Cloud add-on requiring NetBox 4.5 or later, and the curated catalog has no self-service dashboard builder. Teams can use the free NetBox Data Source for Grafana for custom Grafana views. Analytics focuses on inventory and trends, complementing observability tools that report current network activity.

### Source excerpt

The post NetBox Analytics: Get Insight from Your System of Record without Manual Effort appeared first on NetBox Labs.

## Aderant modernizes analytics data pipeline with Apache Iceberg and AWS Lambda

DevFeed: [Aderant modernizes analytics data pipeline with Apache Iceberg and AWS Lambda](<https://devfeed.tech/articles/aderant-modernizes-analytics-data-pipeline-with-apache-iceberg-and-aws-lambda-141916.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/aderant-modernizes-analytics-data-pipeline-with-apache-iceberg-and-aws-lambda/>)

Author: Sham Birbahadur

Published: 2026-11-19T00:36:23Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [event-driven data processing](<https://devfeed.tech/topics/event-driven-data-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [AWS Database Migration Service](<https://devfeed.tech/topics/aws-database-migration-service.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [acid](<https://devfeed.tech/tags/acid.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [batch](<https://devfeed.tech/tags/batch.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [cost](<https://devfeed.tech/tags/cost.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [faster](<https://devfeed.tech/tags/faster.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [legal](<https://devfeed.tech/tags/legal.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [queries](<https://devfeed.tech/tags/queries.md>)

### AI overview

Aderant replaced Stridyn's 15-minute batch pipeline with an event-driven medallion architecture that processes CDC changes from 700 client environments in single-digit seconds to bronze-layer Apache Iceberg tables. AWS DMS delivers Parquet files to S3; S3 notifications and SQS invoke Lambda for ingestion, while scheduled AWS Glue jobs transform data through silver and gold layers. Incremental Glue jobs handle 77 standard gold tables; 34 complex tables use Iceberg materialized views and snapshot comparisons to send changed rows to native Amazon Redshift tables. Aderant reports 10-26x faster queries on native Redshift tables, under-100-millisecond dashboard queries, and 77 percent lower pipeline compute costs versus streaming alternatives. The account says this approach supports schema changes and concurrent tenant workloads; it also describes a possible future Kinesis path for high-volume tenants.

### Source excerpt

Learn how Aderant modernized the data pipeline behind its Stridyn legal analytics platform with Apache Iceberg, AWS Lambda, and AWS Glue, moving from 15-minute batch cycles to an event-driven medallion architecture that processes change data capture events in single-digit seconds for 700 tenants, with 10-26x faster queries and 77 percent lower cost.

## Orchestrate DuckDB analytics pipelines with Amazon MWAA

DevFeed: [Orchestrate DuckDB analytics pipelines with Amazon MWAA](<https://devfeed.tech/articles/orchestrate-duckdb-analytics-pipelines-with-amazon-mwaa-141881.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/orchestrate-duckdb-analytics-pipelines-with-amazon-mwaa/>)

Author: Vinod Jayendra

Published: 2026-11-18T20:57:09Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [parquet](<https://devfeed.tech/topics/parquet.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-managed-workflows-for-apache-airflow-amazon-mwaa](<https://devfeed.tech/tags/amazon-managed-workflows-for-apache-airflow-amazon-mwaa.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [data](<https://devfeed.tech/tags/data.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [effective](<https://devfeed.tech/tags/effective.md>), [embedded](<https://devfeed.tech/tags/embedded.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [learn-how-to](<https://devfeed.tech/tags/learn-how-to.md>), [model](<https://devfeed.tech/tags/model.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [retry](<https://devfeed.tech/tags/retry.md>), [s3](<https://devfeed.tech/tags/s3.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Amazon MWAA can orchestrate bounded SQL analytics by running DuckDB inside Airflow worker tasks, without a separate query cluster. The walkthrough reads S3 Parquet files, deduplicates events, uses ASOF LEFT JOINs to apply rates and ownership effective at each event time, and merges results into three Iceberg tables in Amazon S3 Tables. A second task validates counts, uniqueness, and allocation states. On supported MWAA Provisioned environments running Airflow 3.3.1, DuckDB and its providers plus the duckdb_default AWS-aware connection are preinstalled; the execution role supplies temporary credentials. The example's unchanged-input reruns retained 300,000 unique events and duplicate-free summaries, but the excerpt says this does not establish recovery from partial failures or concurrent external writes. Each task shares one worker's CPU, memory, and storage, so limits and concurrency need tuning with representative data. Use Glue or EMR for steps needing horizontal scaling or more capacity than one worker provides. Three table commits are not one transaction, and Iceberg update/delete support can vary by version.

### Source excerpt

Learn how to orchestrate embedded DuckDB analytics pipelines with Amazon MWAA. This post shows how to run DuckDB inside Airflow tasks to read Parquet data from Amazon S3, apply time-effective model rates, and publish Apache Iceberg tables to Amazon S3 Tables, with retry-safe publishing and worker-sizing considerations.

## How United Airlines streams database changes to Apache Iceberg with Amazon Data Firehose

DevFeed: [How United Airlines streams database changes to Apache Iceberg with Amazon Data Firehose](<https://devfeed.tech/articles/how-united-airlines-streams-database-changes-to-apache-iceberg-with-amazon-data-firehose-141918.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/how-united-airlines-streams-database-changes-to-apache-iceberg-with-amazon-data-firehose/>)

Author: Guillaume Dupont

Published: 2026-11-18T20:37:39Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Amazon Aurora](<https://devfeed.tech/topics/amazon-aurora.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [streaming data to S3 with Parquet](<https://devfeed.tech/topics/streaming-data-to-s3-with-parquet.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [Visual Studio Code](<https://devfeed.tech/topics/visual-studio-code.md>)

Tags: [airlines](<https://devfeed.tech/tags/airlines.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-aurora](<https://devfeed.tech/tags/amazon-aurora.md>), [amazon-data-firehose](<https://devfeed.tech/tags/amazon-data-firehose.md>), [amazon-kinesis](<https://devfeed.tech/tags/amazon-kinesis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cloudformation](<https://devfeed.tech/tags/aws-cloudformation.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [dms](<https://devfeed.tech/tags/dms.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [kinesis-data-streams](<https://devfeed.tech/tags/kinesis-data-streams.md>), [streams](<https://devfeed.tech/tags/streams.md>), [united-airlines](<https://devfeed.tech/tags/united-airlines.md>)

### AI overview

United Airlines describes a near-real-time CDC pipeline that moves Aurora MySQL changes through AWS DMS, Kinesis Data Streams, Lambda, and Amazon Data Firehose into Apache Iceberg tables on S3. Firehose applies inserts, updates, and deletes as Iceberg snapshots; the open tables can be queried through AWS Glue, Redshift, or Athena. United reports 5x lower average query latency and 2x shorter downstream Glue job processing time in its testing, with a low-volume test reaching about 10 seconds end-to-end. Buffer settings trade latency against cost and file size. The walkthrough uses DMS Serverless, Athena for validation, and a simplified single-table CloudFormation template; United's production setup uses provisioned DMS and primarily Redshift Serverless. The template's shard count and other demo settings are not production-tuned, and schema evolution and multi-table replication require additional handling.

### Source excerpt

United Airlines built a near-real-time change data capture pipeline that replicates reference data from Amazon Aurora MySQL to open Apache Iceberg tables using AWS DMS, Amazon Kinesis Data Streams, and Amazon Data Firehose. This post shows how to reproduce the serverless, multi-engine analytics pattern with AWS CloudFormation.

## Exclusive: Wherobots AI agents automate complex spatial data engineering

DevFeed: [Exclusive: Wherobots AI agents automate complex spatial data engineering](<https://devfeed.tech/articles/exclusive-wherobots-ai-agents-automate-complex-spatial-data-engineering-140316.md>)

Original publisher: [Read original article](<https://siliconangle.com/2026/11/18/exclusive-wherobots-ai-agents-automate-complex-spatial-data-engineering/>)

Author: Paul Gillin

Published: 2026-11-18T11:00:42Z

Content type: news

Language: en

Sources: [SiliconANGLE](<https://devfeed.tech/sources/siliconangle.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [geospatial](<https://devfeed.tech/topics/geospatial.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Geographic Information System](<https://devfeed.tech/topics/gis.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-sedona](<https://devfeed.tech/tags/apache-sedona.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [geographic-visualizations](<https://devfeed.tech/tags/geographic-visualizations.md>), [geospatial-data](<https://devfeed.tech/tags/geospatial-data.md>), [global-marketing-insights-inc](<https://devfeed.tech/tags/global-marketing-insights-inc.md>), [news](<https://devfeed.tech/tags/news.md>), [spatial](<https://devfeed.tech/tags/spatial.md>), [spatial-analytics](<https://devfeed.tech/tags/spatial-analytics.md>), [spatial-data-catalog](<https://devfeed.tech/tags/spatial-data-catalog.md>), [the-latest](<https://devfeed.tech/tags/the-latest.md>), [top-story-5](<https://devfeed.tech/tags/top-story-5.md>), [wherobots-havasu](<https://devfeed.tech/tags/wherobots-havasu.md>), [wherobots-inc](<https://devfeed.tech/tags/wherobots-inc.md>), [wherobots-rasterflow](<https://devfeed.tech/tags/wherobots-rasterflow.md>), [wherobots-waldo](<https://devfeed.tech/tags/wherobots-waldo.md>), [wherobotsdb](<https://devfeed.tech/tags/wherobotsdb.md>)

### AI overview

Wherobots' Waldo agent uses natural-language instructions to build spatial data pipelines, coordinating data discovery, query generation, analysis, visualization, and scheduled refreshes. Its platform combines Havasu for dataset discovery and preparation, RasterFlow for imagery processing, WherobotsDB with more than 300 spatial functions, and Wherobots Maps. Wherobots says it connects existing LLMs to specialized processing engines and can use existing cloud storage and catalogs such as AWS Glue and Databricks Unity Catalog, without requiring data migration. Users can inspect generated queries and processing steps, but the company cautions that prompts do not always produce accurate results. Waldo is in public preview; agent tokens are included at no extra charge in Professional Edition, while customers primarily pay for computing resources.

### Source excerpt

Cloud-native spatial analytics and artificial intelligence startup Wherobots Inc. today introduced an artificial intelligence agent that enables users to build and deploy spatial data pipelines using natural-language instructions, reducing the need for specialized programming expertise. Called Waldo, the agent can discover data sources, generate queries, perform analysis, visualize results and schedule recurring operations. It's designed [...] The post Exclusive: Wherobots AI agents automate complex spatial data engineering appeared first on SiliconANGLE.

## How to Build a Modern Data Stack From the Ground Up | Blog | Fivetran

DevFeed: [How to Build a Modern Data Stack From the Ground Up | Blog | Fivetran](<https://devfeed.tech/articles/how-to-build-a-modern-data-stack-from-the-ground-up-blog-fivetran-141324.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/how-to-build-a-modern-data-stack-from-the-ground-up>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: tutorial

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [elt](<https://devfeed.tech/topics/elt.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [Data serialization](<https://devfeed.tech/topics/data-serialization.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [bi-tools](<https://devfeed.tech/tags/bi-tools.md>), [business-intelligence](<https://devfeed.tech/tags/business-intelligence.md>), [business-logic](<https://devfeed.tech/tags/business-logic.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [data](<https://devfeed.tech/tags/data.md>), [data-driven](<https://devfeed.tech/tags/data-driven.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-platforms](<https://devfeed.tech/tags/data-platforms.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [data-stack](<https://devfeed.tech/tags/data-stack.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [decision-making](<https://devfeed.tech/tags/decision-making.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [modern-data-stack](<https://devfeed.tech/tags/modern-data-stack.md>), [plan](<https://devfeed.tech/tags/plan.md>)

### AI overview

Build the stack around a defined analytics use case: identify the data needed, then choose an ingestion tool that connects to those sources. Centralize data in a warehouse and consider a data lake for historical data; use transformation tools such as dbt with connectors such as Fivetran to prepare data for reporting. Keep business logic consistent and models representative of the real world. Choose a BI tool based on the tradeoff that matters most, such as usability or scalability, and consult the technical team to maintain a cohesive strategy. The article emphasizes data quality throughout: validate at ingestion and transformation stages, test continuously, and promote to production only when confident the data will not mislead users.

### Source excerpt

A data architecture expert gives a crash course on moving analytics to the cloud -- including a step-by-step plan to ensure success.

## Using Analytics to Disrupt the Home Insurance Industry | Blog | Fivetran

DevFeed: [Using Analytics to Disrupt the Home Insurance Industry | Blog | Fivetran](<https://devfeed.tech/articles/using-analytics-to-disrupt-the-home-insurance-industry-blog-fivetran-141346.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/using-analytics-to-disrupt-the-home-insurance-industry>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: news

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Business, Industry, Usability, User Experience, User Research, Web Strategy](<https://devfeed.tech/topics/business-industry-usability-user-experience-user-research-web-strategy.md>), [product analytics](<https://devfeed.tech/topics/product-analytics.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [analyst](<https://devfeed.tech/tags/analyst.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [blog](<https://devfeed.tech/tags/blog.md>), [business](<https://devfeed.tech/tags/business.md>), [data](<https://devfeed.tech/tags/data.md>), [data-stack](<https://devfeed.tech/tags/data-stack.md>), [disrupt](<https://devfeed.tech/tags/disrupt.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [home](<https://devfeed.tech/tags/home.md>), [industry](<https://devfeed.tech/tags/industry.md>), [insurance](<https://devfeed.tech/tags/insurance.md>), [leveraging](<https://devfeed.tech/tags/leveraging.md>), [model](<https://devfeed.tech/tags/model.md>), [modern](<https://devfeed.tech/tags/modern.md>), [modern-data](<https://devfeed.tech/tags/modern-data.md>), [modern-data-stack](<https://devfeed.tech/tags/modern-data-stack.md>), [stack](<https://devfeed.tech/tags/stack.md>), [startup](<https://devfeed.tech/tags/startup.md>), [using](<https://devfeed.tech/tags/using.md>)

### AI overview

Luko uses centralized data and analytics to support a preventive home insurance model focused on reducing claims. Its data stack brings together finance, claims, customer support and IoT data, giving teams a more reliable shared view for daily performance tracking and decisions. The article credits Fivetran connectors and dbt with making ingestion and transformation manageable for a company without data engineers, helping it measure services and automate Giveback calculations. Luko says it can nearly forecast claim reimbursements in real time; its previous year's Giveback totaled EUR 100,000.

### Source excerpt

A French startup is leveraging its modern data stack to overturn a longstanding business model.

## New Connector: Airtable | Blog | Fivetran

DevFeed: [New Connector: Airtable | Blog | Fivetran](<https://devfeed.tech/articles/new-connector-airtable-blog-fivetran-141334.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/new-connector-airtable>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: release

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [etl](<https://devfeed.tech/topics/etl.md>), [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [data transformation](<https://devfeed.tech/topics/data-transformation.md>)

Tags: [airtable](<https://devfeed.tech/tags/airtable.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [blog](<https://devfeed.tech/tags/blog.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [spreadsheets](<https://devfeed.tech/tags/spreadsheets.md>)

### AI overview

Fivetran's Airtable connector uses an API key to access tables in Airtable bases and their metadata, with webhooks enabling incremental table updates. The text says Fivetran's partnership with Airtable allows it to retrieve additional fields beyond the standard API and provide higher throughput for faster syncs. Airtable's flexible structure means organizations may use it differently across departments; examples cited include product launch tracking, marketing campaigns, and accounts payable and receivable monitoring.

### Source excerpt

Bring disparate spreadsheets into your centralized analytics destination.

## From Months to Days: How Autodesk Reinvented Its Data Ingestion Strategy | Blog | Fivetran

DevFeed: [From Months to Days: How Autodesk Reinvented Its Data Ingestion Strategy | Blog | Fivetran](<https://devfeed.tech/articles/from-months-to-days-how-autodesk-reinvented-its-data-ingestion-strategy-blog-fivetran-141321.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/how-autodesk-reinvented-its-data-ingestion-strategy>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [elt](<https://devfeed.tech/topics/elt.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [copy](<https://devfeed.tech/topics/copy.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [autodesk](<https://devfeed.tech/tags/autodesk.md>), [aws](<https://devfeed.tech/tags/aws.md>), [data](<https://devfeed.tech/tags/data.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-platform](<https://devfeed.tech/tags/data-platform.md>), [data-stack](<https://devfeed.tech/tags/data-stack.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [join](<https://devfeed.tech/tags/join.md>), [keep](<https://devfeed.tech/tags/keep.md>), [kinesis](<https://devfeed.tech/tags/kinesis.md>), [learning](<https://devfeed.tech/tags/learning.md>), [looker](<https://devfeed.tech/tags/looker.md>), [process](<https://devfeed.tech/tags/process.md>), [strategy](<https://devfeed.tech/tags/strategy.md>)

### AI overview

Autodesk replaced a patchwork of ingestion tools and custom scripts with two defined routes: Fivetran ingests structured sources into Snowflake, while AWS Kinesis sends high-volume unstructured product usage data to S3. Data moves between the repositories when needed; usage data can be promoted to Snowflake for analytics subject to privacy controls, and Snowflake snapshots are stored in S3 for history and machine learning. Autodesk's stated goal was to cut onboarding a dataset, including requirements, vendor setup, and security and privacy controls, from six months to six business days. The change reportedly freed engineers to focus on business problems and reduced urgent pipeline-breakage emails. The account attributes the approach to Autodesk's buy-over-build, simplicity, speed-to-impact, and privacy principles.

### Source excerpt

By replacing homemade ETL with automated ELT, Autodesk cut its data ingestion process from six months to six days.

## ETL Platform Alternatives: Fivetran vs. Informatica and More | Blog | Fivetran

DevFeed: [ETL Platform Alternatives: Fivetran vs. Informatica and More | Blog | Fivetran](<https://devfeed.tech/articles/etl-platform-alternatives-fivetran-vs-informatica-and-more-blog-fivetran-141305.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/etl-platform-alternatives-fivetran-vs-informatica-and-more>)

Author: Charles Wang

Published: 2026-11-18T06:13:15Z

Content type: comparison

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [etl](<https://devfeed.tech/topics/etl.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>)

Tags: [alternatives](<https://devfeed.tech/tags/alternatives.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [business-intelligence](<https://devfeed.tech/tags/business-intelligence.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [cloud-native](<https://devfeed.tech/tags/cloud-native.md>), [compare](<https://devfeed.tech/tags/compare.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [etl](<https://devfeed.tech/tags/etl.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [informatica](<https://devfeed.tech/tags/informatica.md>), [integration](<https://devfeed.tech/tags/integration.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [security](<https://devfeed.tech/tags/security.md>), [sources](<https://devfeed.tech/tags/sources.md>), [tools](<https://devfeed.tech/tags/tools.md>)

### AI overview

The excerpt compares ETL platforms on destinations, connectors, pricing, trials, compliance and replication behavior. It says Informatica has opaque, sales-led pricing and a 30-day trial; Matillion has just over 100 connectors across five destinations, usage-based tiers and a 14-day trial; Xplenty offers almost 100 integrations and broad destination types but charges per source-destination connection and lacks change data capture; Stitch offers 130+ connectors, consumption-based tiers and a 14-day trial. The article also recommends testing platforms with live data for volume, speed, types, record structures and documentation, and asking how they handle source schema changes at the destination. It claims Fivetran supports 700+ sources and 15+ destinations, uses monthly-active-row pricing, and automatically migrates warehouse schemas when source schemas change.

### Source excerpt

Compare popular data integration tools on cost, security, reliability and number of supported sources.

## Trends that will dominate analytics & BI in 2022 | Blog | Fivetran

DevFeed: [Trends that will dominate analytics & BI in 2022 | Blog | Fivetran](<https://devfeed.tech/articles/trends-that-will-dominate-analytics-bi-in-2022-blog-fivetran-141345.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/trends-that-will-dominate-analytics-bi-2022>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [DataOps](<https://devfeed.tech/topics/dataops.md>), [elt](<https://devfeed.tech/topics/elt.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [etl](<https://devfeed.tech/topics/etl.md>), [databricks](<https://devfeed.tech/topics/databricks.md>)

Tags: [2022](<https://devfeed.tech/tags/2022.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [automation](<https://devfeed.tech/tags/automation.md>), [beam](<https://devfeed.tech/tags/beam.md>), [bi](<https://devfeed.tech/tags/bi.md>), [bi-tools](<https://devfeed.tech/tags/bi-tools.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [boost](<https://devfeed.tech/tags/boost.md>), [building](<https://devfeed.tech/tags/building.md>), [business](<https://devfeed.tech/tags/business.md>), [business-intelligence](<https://devfeed.tech/tags/business-intelligence.md>), [chaos](<https://devfeed.tech/tags/chaos.md>), [dataops](<https://devfeed.tech/tags/dataops.md>), [elt](<https://devfeed.tech/tags/elt.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [managed](<https://devfeed.tech/tags/managed.md>), [trends](<https://devfeed.tech/tags/trends.md>)

### AI overview

The article predicts five shifts in analytics and BI for 2022: ELT pipelines becoming more continuous and fully managed, reducing reliance on brittle batch ETL; DataOps practices and features becoming more central, with dbt Labs cited for SQL-based warehouse transformations, lineage tracking, and version control; operational analytics making data available in frontline tools to support day-to-day decisions; BI tools enabling more collaborative work between data and business teams; and wider use of distributed processing frameworks such as Apache Spark, Beam, Flink, and PySpark. These are presented as forecasts, not measured outcomes.

### Source excerpt

Fully managed ELT, DataOps and more trends that will change the way we use data this year.

## How to Use Fivetran Data Models to Accelerate Analytics | Blog | Fivetran

DevFeed: [How to Use Fivetran Data Models to Accelerate Analytics | Blog | Fivetran](<https://devfeed.tech/articles/how-to-use-fivetran-data-models-to-accelerate-analytics-blog-fivetran-141330.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/how-to-use-fivetran-data-models-to-accelerate-analytics>)

Author: Dayna Shoemaker

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [data-modeling](<https://devfeed.tech/topics/data-modeling.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [business-logic](<https://devfeed.tech/tags/business-logic.md>), [code-modularity](<https://devfeed.tech/tags/code-modularity.md>), [core](<https://devfeed.tech/tags/core.md>), [custom](<https://devfeed.tech/tags/custom.md>), [data-modeling](<https://devfeed.tech/tags/data-modeling.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [financial](<https://devfeed.tech/tags/financial.md>), [financial-reporting](<https://devfeed.tech/tags/financial-reporting.md>), [firefly](<https://devfeed.tech/tags/firefly.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [reporting](<https://devfeed.tech/tags/reporting.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

Fivetran's open-source dbt data models provide maintained SQL that works with data ingested through Fivetran, reducing the work of turning raw schemas into analytics-ready tables. Source packages document and standardize underlying schemas; transform packages create analytics-ready models. For standard reporting such as financial reports, packages may be used directly. Where business logic is more specialized, dbt users can combine them with custom models and schedule dependencies through Fivetran's dbt Core integration. Users who prefer to build transformations themselves can inspect the SQL as a template and contribute code to future package versions.

### Source excerpt

Fivetran pre-built data models package SQL to accelerate analytics with near instant reporting.

## Fivetran vs. Supermetrics | Blog | Fivetran

DevFeed: [Fivetran vs. Supermetrics | Blog | Fivetran](<https://devfeed.tech/articles/fivetran-vs-supermetrics-blog-fivetran-140232.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/fivetran-vs-supermetrics>)

Author: Zarmina Niazie

Published: 2026-11-18T05:58:59Z

Content type: comparison

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [etl](<https://devfeed.tech/topics/etl.md>), [elt](<https://devfeed.tech/topics/elt.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Data visualization](<https://devfeed.tech/topics/data-visualization.md>)

Tags: [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [api](<https://devfeed.tech/tags/api.md>), [applications](<https://devfeed.tech/tags/applications.md>), [automated](<https://devfeed.tech/tags/automated.md>), [automatically](<https://devfeed.tech/tags/automatically.md>), [aws](<https://devfeed.tech/tags/aws.md>), [azure](<https://devfeed.tech/tags/azure.md>), [azure-synapse](<https://devfeed.tech/tags/azure-synapse.md>), [blog](<https://devfeed.tech/tags/blog.md>), [cloud-data-warehouse](<https://devfeed.tech/tags/cloud-data-warehouse.md>), [company](<https://devfeed.tech/tags/company.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [data](<https://devfeed.tech/tags/data.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [integrate](<https://devfeed.tech/tags/integrate.md>), [marketing](<https://devfeed.tech/tags/marketing.md>), [sources](<https://devfeed.tech/tags/sources.md>), [vs](<https://devfeed.tech/tags/vs.md>)

### AI overview

The article contrasts Supermetrics' emphasis on moving marketing data to Excel, Google Sheets, and CSV-based workflows with Fivetran's automated pipelines into data warehouses. It says Supermetrics also supports some warehouse and cloud storage destinations, but connector compatibility, warehouse pricing, and a limited warehouse selection may matter; premium connectors can add costs. Fivetran describes incremental loads designed around warehouse APIs, query-ready schemas, adjustable sync intervals, and a broader set of application, file, and database connectors for combining marketing data with finance, sales, or product data. The article also highlights Fivetran's connector documentation and support, while acknowledging both platforms serve different needs and recommending trialing a solution before buying. These are claims from Fivetran's comparison article.

### Source excerpt

Trying to integrate your marketing data sources? Here are the fundamental differences between Fivetran and Supermetrics.

## How To Build a Data Pipeline: A Step-by-Step Guide

DevFeed: [How To Build a Data Pipeline: A Step-by-Step Guide](<https://devfeed.tech/articles/how-to-build-a-data-pipeline-a-step-by-step-guide-140200.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/build-a-data-pipeline>)

Author: Fivetran

Published: 2026-11-18T04:57:01Z

Content type: tutorial

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [elt](<https://devfeed.tech/topics/elt.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [automation](<https://devfeed.tech/tags/automation.md>), [batch](<https://devfeed.tech/tags/batch.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [cloud-based](<https://devfeed.tech/tags/cloud-based.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [how-to](<https://devfeed.tech/tags/how-to.md>)

### AI overview

A data pipeline moves data from sources through ingestion, processing, storage and workflows, with monitoring to catch system and data quality problems. The guide recommends inventorying sources, choosing batch or streaming ingestion based on needs, validating and transforming data, selecting destinations such as a warehouse, lake or lakehouse, and monitoring health, anomalies, alerts and logs. It contrasts ETL, which transforms before storage and can support masking before loading but becomes harder to scale as sources grow, with ELT, which loads first for faster availability and easier cloud scaling but can analyze large volumes more slowly. It also highlights automation, performance, reliability, scalability and security; ELT can still protect sensitive data through isolation, encryption and blocking or hashing before storage. The excerpt concludes that many organizations buy managed pipelines because building and maintaining connectors and components is labor-intensive.

### Source excerpt

Learn what a data pipeline is and how to build one.

## Why Front migrated from Elasticsearch to ClickHouse for "pixel-perfect" analytics at scale

DevFeed: [Why Front migrated from Elasticsearch to ClickHouse for "pixel-perfect" analytics at scale](<https://devfeed.tech/articles/why-front-migrated-from-elasticsearch-to-clickhouse-for-pixel-perfect-analytics-at-scale-144379.md>)

Original publisher: [Read original article](<https://clickhouse.com/blog/front-elasticsearch-clickhouse>)

Author: ClickHouse

Published: 2026-11-18T00:00:00Z

Content type: news

Language: en

Sources: [ClickHouse Blog](<https://devfeed.tech/sources/clickhouse-blog.md>)

Topics: [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [metric-standardization](<https://devfeed.tech/topics/metric-standardization.md>), [product analytics](<https://devfeed.tech/topics/product-analytics.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [aws](<https://devfeed.tech/tags/aws.md>), [b2b](<https://devfeed.tech/tags/b2b.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [customer](<https://devfeed.tech/tags/customer.md>), [data](<https://devfeed.tech/tags/data.md>), [days](<https://devfeed.tech/tags/days.md>), [development](<https://devfeed.tech/tags/development.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [metric](<https://devfeed.tech/tags/metric.md>), [more](<https://devfeed.tech/tags/more.md>), [pixel](<https://devfeed.tech/tags/pixel.md>), [scale](<https://devfeed.tech/tags/scale.md>)

### AI overview

Front moved customer-facing analytics from Elasticsearch to ClickHouse Cloud as its 25 TB workload and specialized Painless queries made new metrics slow and dependent on senior analytics engineers. It migrated gradually, writing each new event to both systems and comparing metrics, then defined each metric once in a shared framework that generates visualizations. For mutable conversations, VersionedCollapsingMergeTree reconciles changes such as deleted messages and merged threads. In one large customer's tag report, a 30-second query grew from 11 days of history in Elasticsearch to a month and a half in ClickHouse, using half as many machines of equal performance. The first metric built in the new framework took a junior engineer three days, versus several weeks for a senior engineer with Elasticsearch. The account also reports cloud-region moves dropping from days to minutes. Front says the change enables more flexible dashboards and plans to explore backlog metrics and AI-assisted metric creation.

### Source excerpt

Front powers customer-facing analytics for 9,000+ B2B companies with ClickHouse Cloud, cutting metric development from three weeks to three days and querying 4x more historical data.

## Dispelling LTAP myths

DevFeed: [Dispelling LTAP myths](<https://devfeed.tech/articles/dispelling-ltap-myths-130978.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/dispelling-ltap-myths>)

Author: Sergey Melnik

Published: 2026-11-12T21:49:43Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [read-replica](<https://devfeed.tech/topics/read-replica.md>), [Debezium Connector](<https://devfeed.tech/topics/debezium-connector.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [JOIN](<https://devfeed.tech/topics/join.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [addition](<https://devfeed.tech/tags/addition.md>), [after](<https://devfeed.tech/tags/after.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [alter](<https://devfeed.tech/tags/alter.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [api](<https://devfeed.tech/tags/api.md>), [apis](<https://devfeed.tech/tags/apis.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [array](<https://devfeed.tech/tags/array.md>), [availability](<https://devfeed.tech/tags/availability.md>), [bookkeeping](<https://devfeed.tech/tags/bookkeeping.md>), [buffer](<https://devfeed.tech/tags/buffer.md>), [build](<https://devfeed.tech/tags/build.md>), [but](<https://devfeed.tech/tags/but.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [lakebase](<https://devfeed.tech/tags/lakebase.md>), [monolith](<https://devfeed.tech/tags/monolith.md>), [myths](<https://devfeed.tech/tags/myths.md>), [post](<https://devfeed.tech/tags/post.md>)

### AI overview

LTAP stores one logical copy of each Postgres table in the lake, while heap pages, indexes, and caches serve as tiers for fast operational access. The excerpt distinguishes these tiers from CDC replicas, which create a separately queryable database that can drift, omit tables, or fall behind. For analytics, heap pages support point lookups while Parquet supports scans; queries merge recent committed pages with the materialized snapshot at a shared LSN, keeping multi-table reads consistent even when Parquet lags. LTAP compacts changed pages in parallel rather than replaying every WAL row event. The author says logical replication remains useful for migrations, upgrades, and cutovers, and notes that APIs for reading changes since a watermark are still being developed.

### Source excerpt

Reynold Xin's post "From monolith to Lakebase to LTAP" explained what LTAP is and...

## Get more from OpenTelemetry: From connected data to faster root cause analysis

DevFeed: [Get more from OpenTelemetry: From connected data to faster root cause analysis](<https://devfeed.tech/articles/get-more-from-opentelemetry-from-connected-data-to-faster-root-cause-analysis-130715.md>)

Original publisher: [Read original article](<https://www.dynatrace.com/news/blog/get-more-from-opentelemetry-from-connected-data-to-faster-root-cause-analysis/>)

Author: Kayla Bondy

Published: 2026-11-12T17:31:07Z

Content type: article

Language: en

Sources: [Dynatrace news](<https://devfeed.tech/sources/dynatrace-news.md>)

Topics: [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Instrumentation](<https://devfeed.tech/topics/instrumentation.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>)

Tags: [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-workflows](<https://devfeed.tech/tags/ai-workflows.md>), [causal-ai](<https://devfeed.tech/tags/causal-ai.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [context](<https://devfeed.tech/tags/context.md>), [control](<https://devfeed.tech/tags/control.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [dependency](<https://devfeed.tech/tags/dependency.md>), [dependency-graph](<https://devfeed.tech/tags/dependency-graph.md>), [digital-experience](<https://devfeed.tech/tags/digital-experience.md>), [dynatrace](<https://devfeed.tech/tags/dynatrace.md>), [dynatrace-grail](<https://devfeed.tech/tags/dynatrace-grail.md>), [dynatrace-intelligence](<https://devfeed.tech/tags/dynatrace-intelligence.md>), [effectively](<https://devfeed.tech/tags/effectively.md>), [instrumentation](<https://devfeed.tech/tags/instrumentation.md>), [observability](<https://devfeed.tech/tags/observability.md>), [opensource](<https://devfeed.tech/tags/opensource.md>), [opentelemetry](<https://devfeed.tech/tags/opentelemetry.md>), [product-news](<https://devfeed.tech/tags/product-news.md>), [root-cause-analysis](<https://devfeed.tech/tags/root-cause-analysis.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>)

### AI overview

Dynatrace combines telemetry controls, unified storage, topology context, and causal AI to help teams find root causes across OpenTelemetry-instrumented services. Its telemetry pipeline filters, masks, enriches, and routes data, with centralized collector management for consistent policies. Grail brings logs, metrics, and traces together, while Smartscape maps service dependencies. Causal AI follows those relationships to merge anomalies across affected services into one Problem and identify a root cause, including an OpenTelemetry service or a monitored, alerting database. This can reduce noise and triage time. Root-cause information can also provide AI workflows with impact, origin, and causal-path context; agent actions depend on their implementation, permissions, and available data.

### Source excerpt

OpenTelemetry instrumentation gives organizations flexibility in how they collect telemetry. Getting value from that investment depends on what happens next: how effectively teams can manage growing volumes of data and turn that telemetry into reliable answers. As AI becomes part of software delivery and operations, teams and AI agents need context around the telemetry they [...] The post Get more from OpenTelemetry: From connected data to faster root cause analysis appeared first on Dynatrace news.

## Dagster: Open-source cloud-native data pipeline orchestrator

DevFeed: [Dagster: Open-source cloud-native data pipeline orchestrator](<https://devfeed.tech/articles/dagster-open-source-cloud-native-data-pipeline-orchestrator-129958.md>)

Original publisher: [Read original article](<https://www.helpnetsecurity.com/2026/11/12/dagster-open-source-cloud-native-data-pipeline-orchestrator/>)

Author: Mirko Zorz

Published: 2026-11-12T06:30:08Z

Content type: article

Language: en

Sources: [Help Net Security](<https://devfeed.tech/sources/help-net-security.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [asset](<https://devfeed.tech/tags/asset.md>), [assets](<https://devfeed.tech/tags/assets.md>), [dagster](<https://devfeed.tech/tags/dagster.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [days](<https://devfeed.tech/tags/days.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [don-t-miss](<https://devfeed.tech/tags/don-t-miss.md>), [github](<https://devfeed.tech/tags/github.md>), [graph](<https://devfeed.tech/tags/graph.md>), [integrations](<https://devfeed.tech/tags/integrations.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [models](<https://devfeed.tech/tags/models.md>), [news](<https://devfeed.tech/tags/news.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [orchestrator](<https://devfeed.tech/tags/orchestrator.md>), [reports](<https://devfeed.tech/tags/reports.md>), [software](<https://devfeed.tech/tags/software.md>)

### AI overview

Dagster models each data asset as a Python function decorated with `@dg.asset`; function parameters declare dependencies, which Dagster uses to build a lineage graph in its web UI. Asset checks can test data quality, including imported dbt tests, and can block downstream materialization when they fail. Schedules and sensors can run assets and checks automatically. Partitioned backfills launch one run per partition by default, while a backfill policy can combine a range into one run and still record each partition as filled. That policy applies to backfills launched from the asset graph or an asset page, but not those started from a job page. The excerpt states support for Python 3.9-3.14 and lists dbt among its integrations.

### Source excerpt

Dagster is an open source orchestrator for data pipelines. It decides what runs, in what order, and when. It also tracks how the tables, datasets, machine learning models, and reports it manages connect to the data they came from. When a source table goes bad, the tables built from it carry the problem forward. Dagster's lineage graph shows which ones sit downstream, and that list is the starting point for fixing a broken dashboard or ... More -> The post Dagster: Open-source cloud-native data pipeline orchestrator appeared first on Help Net Security.

## Edelbyte reels in $70M for its multipurpose data analytics platform

DevFeed: [Edelbyte reels in $70M for its multipurpose data analytics platform](<https://devfeed.tech/articles/edelbyte-reels-in-70m-for-its-multipurpose-data-analytics-platform-129740.md>)

Original publisher: [Read original article](<https://siliconangle.com/2026/11/11/edelbyte-reels-in-70m-for-its-multi-purpose-data-analytics-platform/>)

Author: Maria Deutscher

Published: 2026-11-11T23:19:47Z

Content type: news

Language: en

Sources: [SiliconANGLE](<https://devfeed.tech/sources/siliconangle.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [business-intelligence](<https://devfeed.tech/topics/business-intelligence.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [artificial-intelligence-agents](<https://devfeed.tech/tags/artificial-intelligence-agents.md>), [azure](<https://devfeed.tech/tags/azure.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [business-intelligence](<https://devfeed.tech/tags/business-intelligence.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analytics](<https://devfeed.tech/tags/data-analytics.md>), [data-platform](<https://devfeed.tech/tags/data-platform.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [edelbyte](<https://devfeed.tech/tags/edelbyte.md>), [funding](<https://devfeed.tech/tags/funding.md>), [funding-round](<https://devfeed.tech/tags/funding-round.md>), [graph-processing](<https://devfeed.tech/tags/graph-processing.md>), [news](<https://devfeed.tech/tags/news.md>), [platform](<https://devfeed.tech/tags/platform.md>), [query-optimization](<https://devfeed.tech/tags/query-optimization.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [startup](<https://devfeed.tech/tags/startup.md>), [the-latest](<https://devfeed.tech/tags/the-latest.md>)

### AI overview

Edelbyte says its platform can query multipetabyte datasets with under 100 milliseconds of latency while supporting records and AI-related vectors, search, business intelligence, and graph processing. It uses query federation to run queries where data is stored rather than requiring users to move it into the platform, and automates some index customization. The company says several Fortune 500 customers use it in petabyte-scale environments. Edelbyte launched with $70 million raised over two rounds, led by Sequoia and Quiet Capital, and plans to use the funding to expand its customer base and hiring.

### Source excerpt

Edelbyte Ltd., the developer of a data analytics platform optimized for artificial intelligence agents, launched today with $70 million in funding. The startup raised the capital over two rounds. The investments were led by Sequoia and Quiet Capital with contributions from Greenoaks and Swish Ventures. According to CTech, Edelbyte is valued at $500 million. Tel [...] The post Edelbyte reels in $70M for its multipurpose data analytics platform appeared first on SiliconANGLE.

## How to Build Customer-Facing Analytics Pipelines with CDC

DevFeed: [How to Build Customer-Facing Analytics Pipelines with CDC](<https://devfeed.tech/articles/how-to-build-customer-facing-analytics-pipelines-with-cdc-129458.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/customer-facing-analytics-cdc>)

Author: Brian Sheehan

Published: 2026-11-11T06:36:13Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [build](<https://devfeed.tech/tags/build.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [customer](<https://devfeed.tech/tags/customer.md>), [data](<https://devfeed.tech/tags/data.md>), [data-freshness](<https://devfeed.tech/tags/data-freshness.md>), [examples](<https://devfeed.tech/tags/examples.md>), [guide](<https://devfeed.tech/tags/guide.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [isolation](<https://devfeed.tech/tags/isolation.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [practical](<https://devfeed.tech/tags/practical.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [saas](<https://devfeed.tech/tags/saas.md>), [serving](<https://devfeed.tech/tags/serving.md>), [streaming-analytics](<https://devfeed.tech/tags/streaming-analytics.md>), [tenant-isolation](<https://devfeed.tech/tags/tenant-isolation.md>), [with](<https://devfeed.tech/tags/with.md>), [world](<https://devfeed.tech/tags/world.md>)

### AI overview

Customer facing analytics needs a CDC pipeline that keeps dashboard scans off production while meeting freshness, concurrency, tenant isolation, and correctness requirements. The described flow captures database log changes, maintains history and current state, transforms reporting tables, and serves them through a layer that enforces tenant access. Freshness depends on capture lag, destination apply schedules, transformations, serving refresh, and caching; a 30 minute warehouse apply interval can leave data up to about 30 minutes old before transformations. Read replicas move scans off the primary but long queries can be canceled, increase replica lag, or contribute to primary database bloat, depending on configuration. Tenant identifiers must follow rows through the pipeline, with access policies or isolated databases enforced at the serving layer. Monitoring should track processing progress as well as row freshness, and validate current state against history and capture completeness against the source. The excerpt reports Deel reduced reporting latency from 2-6 hours to under 1 hour with Estuary and Snowflake; Connect&GO typically loaded MySQL data to Snowflake within 15 seconds

### Source excerpt

A practical architecture guide to customer-facing analytics with CDC, covering data freshness, serving layers, tenant isolation, and real-world SaaS examples.

## Cloudflare Makes Basin Serverless Analytics Generally Available

DevFeed: [Cloudflare Makes Basin Serverless Analytics Generally Available](<https://devfeed.tech/articles/cloudflare-makes-basin-serverless-analytics-generally-available-127296.md>)

Original publisher: [Read original article](<https://www.infoq.com/news/2026/11/cloudflare-basin-analytics/>)

Author: Renato Losio

Published: 2026-11-10T19:14:00Z

Content type: release

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [Cloudflare](<https://devfeed.tech/topics/cloudflare.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Amazon Aurora](<https://devfeed.tech/topics/amazon-aurora.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [Cloudflare Workers](<https://devfeed.tech/topics/cloudflare-workers.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-ml-data-engineering](<https://devfeed.tech/tags/ai-ml-data-engineering.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [architecture-design](<https://devfeed.tech/tags/architecture-design.md>), [availability](<https://devfeed.tech/tags/availability.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloudflare](<https://devfeed.tech/tags/cloudflare.md>), [cloudflare-basin-analytics](<https://devfeed.tech/tags/cloudflare-basin-analytics.md>), [data-analytics](<https://devfeed.tech/tags/data-analytics.md>), [data-portability](<https://devfeed.tech/tags/data-portability.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [egress](<https://devfeed.tech/tags/egress.md>), [fees](<https://devfeed.tech/tags/fees.md>), [formats](<https://devfeed.tech/tags/formats.md>), [general-availability](<https://devfeed.tech/tags/general-availability.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [managed](<https://devfeed.tech/tags/managed.md>), [ml-data-engineering](<https://devfeed.tech/tags/ml-data-engineering.md>), [news](<https://devfeed.tech/tags/news.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [open-data](<https://devfeed.tech/tags/open-data.md>), [platform](<https://devfeed.tech/tags/platform.md>), [r2](<https://devfeed.tech/tags/r2.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [serverless-analytics](<https://devfeed.tech/tags/serverless-analytics.md>), [service](<https://devfeed.tech/tags/service.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

Cloudflare's Basin combines serverless data ingestion, Iceberg table management, and distributed SQL queries on R2, aiming to give smaller teams analytics without managing infrastructure. Basin Pipelines accepts events from Workers, HTTP endpoints, and Logpush, applies SQL transformations, and writes to Iceberg tables or R2; Basin Catalog manages Iceberg metadata and table maintenance; Basin SQL queries Iceberg tables. The platform supports Iceberg-compatible tools including PyIceberg, DuckDB, Snowflake, and Apache Spark, with free egress intended to enable portability across tools, regions, and clouds. Cloudflare says Basin is not a replacement for every warehouse workload; a data analyst describes it as an option for teams whose data is too large for spreadsheets but does not justify dedicated infrastructure. Since preview, Cloudflare reports higher ingestion capacity, expanded SQL capabilities, improved maintenance, Logpush integration, and Terraform support. Real-time processing, newer Iceberg specification support, adaptive maintenance, and more sovereignty and access controls are planned without committed release dates.

### Source excerpt

Cloudflare has announced the general availability of Basin, a serverless analytics platform built on Apache Iceberg and R2 Object Storage. The managed service supports open data formats and integrates with tools such as DuckDB, Snowflake, and Apache Spark, with no data egress fees. By Renato Losio

## How to Validate LLM Output in a DuckDB and dbt Pipeline

DevFeed: [How to Validate LLM Output in a DuckDB and dbt Pipeline](<https://devfeed.tech/articles/how-to-validate-llm-output-in-a-duckdb-and-dbt-pipeline-127370.md>)

Original publisher: [Read original article](<https://towardsdatascience.com/how-to-validate-llm-output-in-a-duckdb-and-dbt-pipeline/>)

Author: Ibrahim Salami

Published: 2026-11-10T13:30:00Z

Content type: tutorial

Language: en

Sources: [Towards Data Science](<https://devfeed.tech/sources/towards-data-science.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [Encoding](<https://devfeed.tech/topics/encoding.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [AI Prompt Templates](<https://devfeed.tech/topics/ai-prompt-templates.md>)

Tags: [dbt](<https://devfeed.tech/tags/dbt.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [gemini-flash-lite](<https://devfeed.tech/tags/gemini-flash-lite.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-evaluation](<https://devfeed.tech/tags/llm-evaluation.md>), [llmops](<https://devfeed.tech/tags/llmops.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [python](<https://devfeed.tech/tags/python.md>), [tests](<https://devfeed.tech/tags/tests.md>)

### AI overview

The pipeline uses Gemini Flash-Lite to extract skills, seniority, experience requirements, Nigeria eligibility, and a role summary from RemoteOK postings, then validates results in Python and dbt. The author added job location after an initial Nigeria eligibility answer lacked key context, and required supporting quotes for yes/no answers; code checks each quote against the posting or location before saving. Python rejects invalid responses, while dbt checks allowed categories, evidence, skill names, encoding problems, and unread-job coverage, with failures for bad data and warnings for suspicious conditions. A strict quote check exposed pre-existing mojibake in one location value; an early repair let that job pass, though the author could not establish where the corruption began. In this 18-job sample, 17 records initially passed, and the final results showed one Nigeria-eligible job and SQL as the most requested skill. The author cautions that the small, analyst-heavy sample does not represent broader data-engineering demand.

### Source excerpt

A Gemini Flash-Lite step pulls skills, seniority, and Nigeria eligibility from 18 RemoteOK postings, with Python checks and dbt tests guarding every answer. The post How to Validate LLM Output in a DuckDB and dbt Pipeline appeared first on Towards Data Science.

## Speed Insights and Web Analytics now support custom environments

DevFeed: [Speed Insights and Web Analytics now support custom environments](<https://devfeed.tech/articles/speed-insights-and-web-analytics-now-support-custom-environments-126762.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/speed-insights-and-web-analytics-now-support-custom-environments>)

Author: Damien Simonin Feugas

Published: 2026-11-10T00:00:00Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [api](<https://devfeed.tech/tags/api.md>), [cli](<https://devfeed.tech/tags/cli.md>), [command](<https://devfeed.tech/tags/command.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [custom](<https://devfeed.tech/tags/custom.md>), [dashboard](<https://devfeed.tech/tags/dashboard.md>), [data](<https://devfeed.tech/tags/data.md>), [deployments](<https://devfeed.tech/tags/deployments.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [environment](<https://devfeed.tech/tags/environment.md>), [environments](<https://devfeed.tech/tags/environments.md>), [filter](<https://devfeed.tech/tags/filter.md>), [insights](<https://devfeed.tech/tags/insights.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [performance](<https://devfeed.tech/tags/performance.md>), [speed](<https://devfeed.tech/tags/speed.md>), [support](<https://devfeed.tech/tags/support.md>), [web](<https://devfeed.tech/tags/web.md>)

### AI overview

Speed Insights and Web Analytics now attribute deployment data to its custom environment instead of Preview, allowing staging, QA, or canary activity to be viewed separately from Production and Preview. Select an environment in the Dashboard or filter and break down by environment in the Query builder; the `vc metrics` CLI can query the data, and the Aggregation API supports programmatic queries for Web Analytics only. This changes attribution only: no new data points are collected and no configuration changes are needed. Custom environments are available on Pro and Enterprise plans.

### Source excerpt

Speed Insights and Web Analytics now fully support custom environments. Previously, data captured on deployments to a custom environment was attributed to the Preview environment. That data is now attributed to the custom environment it came from, so you can track performance and traffic for staging, QA, or canary environments separately from Production and Preview. You can view this data in several places: Dashboard: use the Environment dropdown in Speed Insights and Web Analytics to pick a custom environment CLI: query it with the vc metrics command Query builder: filter and break down by environment Aggregation API: query it programmatically (Web Analytics only) This change only affects how data is attributed. No new data points are collected, and no configuration changes are needed. Custom environments are available on Pro and Enterprise plans. Learn more in the environments documentation. Read more

## Large Language Model Consensus and Corpus-Adaptive Topic Modeling for Classifying and Mapping Patent Innovation

DevFeed: [Large Language Model Consensus and Corpus-Adaptive Topic Modeling for Classifying and Mapping Patent Innovation](<https://devfeed.tech/articles/large-language-model-consensus-and-corpus-adaptive-topic-modeling-for-classifying-and-mapping-patent-innovation-124633.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09194>)

Author: Sercan Ozcan, Enes Ozgozler, C. Okan Sakar

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [clustering](<https://devfeed.tech/topics/clustering.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analysts](<https://devfeed.tech/tags/analysts.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-pipeline](<https://devfeed.tech/tags/analytics-pipeline.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [audit](<https://devfeed.tech/tags/audit.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [consensus](<https://devfeed.tech/tags/consensus.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [innovation](<https://devfeed.tech/tags/innovation.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large](<https://devfeed.tech/tags/large.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [mapping](<https://devfeed.tech/tags/mapping.md>), [model](<https://devfeed.tech/tags/model.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [patent](<https://devfeed.tech/tags/patent.md>), [topic](<https://devfeed.tech/tags/topic.md>), [topic-modeling](<https://devfeed.tech/tags/topic-modeling.md>)

### AI overview

The proposed pipeline combines auditable LLM classification with corpus-adaptive topic modeling to map patent innovation. Three LLMs classify each patent's novelty and innovation type; radical and innovation-type labels need two supporting votes, while incremental is the fallback novelty label. PatentSBERTa embeddings are clustered with parameter ranges scaled to corpus size. A patent-specific stopword list is used only for c-TF-IDF topic representation; a second embedding model supports topic similarity, hierarchical refinement, and visualization. An LLM labels topics using ranked keywords and representative patents. In a demonstration on 30,964 public-sector-related patents, five category corpora yielded 74 topic instances, 98.1% macro-average document coverage, topic diversity of 0.900-1.000, and fewer than 1% UNKNOWN classification outputs. Votes and topic-label evidence are retained for review. The study establishes feasibility and traceability, but expert validation and comparisons with alternative pipelines remain future work.

### Source excerpt

arXiv:2611.09194v1 Announce Type: new Abstract: Patent analytics increasingly relies on transformer representations and large language models (LLMs), yet linking conceptual classification, topic formation and interpretation requires explicit decisions about model roles and the evidence retained at each stage. In this design-oriented study, we propose and demonstrate a patent analytics pipeline that links these tasks and keeps each decision open to audit. Three LLMs separately classify each patent by novelty (radical or incremental) and innovation type (product, process, service). Radical and innovation-type labels each require two supporting votes, with Incremental used as the fallback novelty label. Topics are discovered by clustering PatentSBERTa embeddings, with parameter ranges scaled to corpus size. A patent-specific stopword list is applied only when topics are represented with class-based term frequency--inverse document frequency (c-TF-IDF), and a second embedding model supports inter-topic similarity, hierarchical refinement and visualization. An LLM then labels each topic from its ranked keywords and representative patents. We demonstrate the pipeline on 30,964 public-sector-related patents retrieved from Lens.org. Across five category corpora ranging from 591 to 29,215 patents, the pipeline produced 74 category-specific topic instances with a macro-average document coverage of 98.1% and topic diversity of 0.900--1.000, and fewer than 1% of classification outputs returned UNKNOWN. Because every model vote and the evidence behind each topic label are retained, analysts can audit and revise the outputs. The demonstration establishes the feasibility and traceability of the pipeline; testing its labels against expert coding and benchmarking it against alternative pipelines are identified as the next steps. The pipeline offers a reusable basis for LLM-assisted patent analysis in technology intelligence.

[Next page](<https://devfeed.tech/topics/analytics-pipeline.md?cursor=WyIyMDI2LTExLTA5VDA1OjAwOjAwKzAwOjAwIiwgImIyNmUzM2QzLWFiODUtNGNhOC1iN2Y5LTFhY2U2ZTZlMGYzNiJd>)