# BigQuery

BigQuery is a Google Cloud data analytics platform for loading, querying, transforming, and analyzing data.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Fivetran wins three major Partner of the Year awards | Blog | Fivetran

DevFeed: [Fivetran wins three major Partner of the Year awards | Blog | Fivetran](<https://devfeed.tech/articles/fivetran-wins-three-major-partner-of-the-year-awards-blog-fivetran-141318.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/fivetran-wins-three-major-partner-of-the-year-awards>)

Author: Logan Welley

Published: 2026-11-18T06:13:15Z

Content type: news

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [elt](<https://devfeed.tech/topics/elt.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>)

Tags: [awards](<https://devfeed.tech/tags/awards.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [data](<https://devfeed.tech/tags/data.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [google](<https://devfeed.tech/tags/google.md>), [partner](<https://devfeed.tech/tags/partner.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [technology](<https://devfeed.tech/tags/technology.md>)

### AI overview

Fivetran says Google Cloud, Snowflake and Databricks each named it Partner of the Year, citing integration work and joint customer outcomes. For Google Cloud, customers can buy and launch connectors through its Marketplace, and Fivetran describes SAP replication and data movement into BigQuery. Snowflake recognized Fivetran for integration innovation; the article reports more than 2,000 joint customers and says over 60% sync databases, up more than 80% year over year from 2021. Databricks recognized expanded availability and enhancements for unifying data on its lakehouse. Fivetran also says a managed high-volume replication service for Databricks on Azure, AWS and Google Cloud was planned for that summer, supporting real-time replication from Oracle, SQL Server and SAP. These are company-reported claims in the supplied article excerpt.

### Source excerpt

Google, Snowflake and Databricks recognize us for exceptional cloud data integration technology.

## Analysis: Fivetran Is the Fastest-Growing Cloud ELT Provider | Blog | Fivetran

DevFeed: [Analysis: Fivetran Is the Fastest-Growing Cloud ELT Provider | Blog | Fivetran](<https://devfeed.tech/articles/analysis-fivetran-is-the-fastest-growing-cloud-elt-provider-blog-fivetran-141312.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/fivetran-is-fastest-growing-cloud-elt-provider>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [elt](<https://devfeed.tech/topics/elt.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data transformation](<https://devfeed.tech/topics/data-transformation.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [automation](<https://devfeed.tech/tags/automation.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-teams](<https://devfeed.tech/tags/data-teams.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [elt](<https://devfeed.tech/tags/elt.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [gartner](<https://devfeed.tech/tags/gartner.md>)

### AI overview

The text attributes Fivetran's growth to managed cloud data ingestion and ease of use. It quotes Gartner's 2020 market-share report, which says Fivetran nearly doubled revenue from 2019 to 2020 and identifies it as a vendor of choice for moving data from popular SaaS applications and databases into cloud warehouses. The described service uses prebuilt connectors, self-healing pipelines, minimal infrastructure management, and automatic normalization. It also offers in-warehouse SQL transformations powered by dbt, with version control, testing, and documentation. Gartner is quoted as calling this a more complete offering that could support future enterprise adoption and revenue growth.

### Source excerpt

Gartner recognizes Fivetran as the "vendor of choice" for enterprises looking to modernize their data infrastructure.

## Automating credit card fraud detection with Google BigQuery ML and Fivetran

DevFeed: [Automating credit card fraud detection with Google BigQuery ML and Fivetran](<https://devfeed.tech/articles/automating-credit-card-fraud-detection-with-google-bigquery-ml-and-fivetran-140197.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/automating-credit-card-fraud-detection-with-google-bigquery-ml-and-fivetran>)

Author: Kelly Kohlleffel

Published: 2026-11-18T04:42:41Z

Content type: tutorial

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [online-fraud-prevention](<https://devfeed.tech/topics/online-fraud-prevention.md>), [etl](<https://devfeed.tech/topics/etl.md>)

Tags: [automated](<https://devfeed.tech/tags/automated.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [change-detection](<https://devfeed.tech/tags/change-detection.md>), [class](<https://devfeed.tech/tags/class.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cloud-data-warehouse](<https://devfeed.tech/tags/cloud-data-warehouse.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [credit-card](<https://devfeed.tech/tags/credit-card.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [data-privacy](<https://devfeed.tech/tags/data-privacy.md>), [database](<https://devfeed.tech/tags/database.md>), [databases](<https://devfeed.tech/tags/databases.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [fraud](<https://devfeed.tech/tags/fraud.md>), [fraud-detection](<https://devfeed.tech/tags/fraud-detection.md>), [ml](<https://devfeed.tech/tags/ml.md>)

### AI overview

The article demonstrates moving a SQL Server credit card transactions table into BigQuery with Fivetran, then training and querying a fraud classifier with BigQuery ML. It selects a single schema and table, configures log-free change detection, and allows schema changes to sync automatically. Fivetran supports incremental updates and defaults to six-hour polling, configurable from one minute to 24 hours; the described setup uses TLS in transit and AES256 for data temporarily stored by the service. The example dataset has 284,807 transactions and 492 fraud cases. Its logistic regression model applies a log transform to Amount, uses Class as the label, enables automatic class weights for imbalance, and splits data sequentially by Time. ML.PREDICT returns predicted class probabilities alongside transaction amount and actual Class for a selected timestamp.

### Source excerpt

Exploring fraud detection with automated data integration in a cloud data warehouse: part two of two.

## Use Fivetran, BigQuery, and Looker to Get a Handle on NetSuite Data | Blog | Fivetran

DevFeed: [Use Fivetran, BigQuery, and Looker to Get a Handle on NetSuite Data | Blog | Fivetran](<https://devfeed.tech/articles/use-fivetran-bigquery-and-looker-to-get-a-handle-on-netsuite-data-blog-fivetran-140245.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/netsuite-looker>)

Author: Charles Wang

Published: 2026-11-18T02:46:08Z

Content type: tutorial

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [business-intelligence](<https://devfeed.tech/topics/business-intelligence.md>), [Data visualization](<https://devfeed.tech/topics/data-visualization.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [looker](<https://devfeed.tech/topics/looker.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [bi-tools](<https://devfeed.tech/tags/bi-tools.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [business-intelligence](<https://devfeed.tech/tags/business-intelligence.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-platform](<https://devfeed.tech/tags/cloud-platform.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [data](<https://devfeed.tech/tags/data.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [example](<https://devfeed.tech/tags/example.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [github](<https://devfeed.tech/tags/github.md>), [google](<https://devfeed.tech/tags/google.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [google-cloud-platform](<https://devfeed.tech/tags/google-cloud-platform.md>), [looker](<https://devfeed.tech/tags/looker.md>), [netsuite](<https://devfeed.tech/tags/netsuite.md>)

### AI overview

Fivetran extracts NetSuite data, applies normalization and light cleaning, and loads it into a data warehouse, where BI tools can access, transform, and visualize it. The example template combines Fivetran, BigQuery, and Looker. Setup includes connecting NetSuite through the Google Cloud Platform marketplace connector, syncing at least the listed accounting, customer, transaction, and vendor tables, then using the warehouse-specific LookML from the analytics-templates GitHub repository in Looker and converting its dashboards to user-defined dashboards. The guide presents this as a starting point for custom reports; it notes that SQL dialects differ by warehouse.

### Source excerpt

NetSuite is essential but complicated. Use this guide to set up dashboards for your NetSuite data using Fivetran, BigQuery, and Looker.

## Don't let uncertainty cloud your vision - data clears the way | Blog | Fivetran

DevFeed: [Don't let uncertainty cloud your vision - data clears the way | Blog | Fivetran](<https://devfeed.tech/articles/don-t-let-uncertainty-cloud-your-vision-data-clears-the-way-blog-fivetran-140210.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/data-uncertain-times>)

Author: Fivetran

Published: 2026-11-18T02:45:50Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [data analyst](<https://devfeed.tech/topics/data-analyst.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [analyst](<https://devfeed.tech/tags/analyst.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [blog](<https://devfeed.tech/tags/blog.md>), [business](<https://devfeed.tech/tags/business.md>), [correlation](<https://devfeed.tech/tags/correlation.md>), [created](<https://devfeed.tech/tags/created.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [data-driven](<https://devfeed.tech/tags/data-driven.md>), [data-stack](<https://devfeed.tech/tags/data-stack.md>), [database](<https://devfeed.tech/tags/database.md>), [delta](<https://devfeed.tech/tags/delta.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [gap](<https://devfeed.tech/tags/gap.md>), [go-to-market](<https://devfeed.tech/tags/go-to-market.md>), [gtm](<https://devfeed.tech/tags/gtm.md>), [sales](<https://devfeed.tech/tags/sales.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>)

### AI overview

A comparison of production Postgres account creation data with Salesforce showed Salesforce recorded only about half the monthly accounts. Investigation found duplicates, partner and test accounts, but also real customers starting trials and connecting data without sales notification. In March, the team identified many such trials and shared the potential revenue gap with go-to-market partners; Fivetran says it is changing processes to address missed sales follow-up. The investigation used Postgres, Salesforce, Fivetran, BigQuery and Looker. The text describes a potential revenue opportunity, not a quantified realized loss or recovered revenue.

### Source excerpt

How we uncovered an unexpected sales opportunity through data sleuthing.

## Advancing Catalog Interoperability with Unity Catalog and Google Cloud's Borderless Lakehouse

DevFeed: [Advancing Catalog Interoperability with Unity Catalog and Google Cloud's Borderless Lakehouse](<https://devfeed.tech/articles/advancing-catalog-interoperability-with-unity-catalog-and-google-cloud-s-borderless-lakehouse-138347.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/advancing-catalog-interoperability-unity-catalog-and-google-clouds-borderless-lakehouse>)

Author: Alex Jiang; Parth Desai; John Spencer

Published: 2026-11-17T16:30:00Z

Content type: release

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Databricks Unity Catalog integration](<https://devfeed.tech/topics/databricks-unity-catalog-integration.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Local AI](<https://devfeed.tech/topics/local-ai.md>)

Tags: [accessible](<https://devfeed.tech/tags/accessible.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [business-context](<https://devfeed.tech/tags/business-context.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [column](<https://devfeed.tech/tags/column.md>), [community](<https://devfeed.tech/tags/community.md>), [context](<https://devfeed.tech/tags/context.md>), [copy](<https://devfeed.tech/tags/copy.md>), [google](<https://devfeed.tech/tags/google.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [governance](<https://devfeed.tech/tags/governance.md>), [interoperability](<https://devfeed.tech/tags/interoperability.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [open](<https://devfeed.tech/tags/open.md>), [partners](<https://devfeed.tech/tags/partners.md>), [platform](<https://devfeed.tech/tags/platform.md>), [unity-catalog](<https://devfeed.tech/tags/unity-catalog.md>)

### AI overview

The described Unity Catalog and Google Cloud Lakehouse integration enables access to the same data across platforms without copying it. Databricks can mount Iceberg tables managed by Google Cloud's Lakehouse runtime catalog through Lakehouse Federation for analysis in Databricks, its serverless data warehouse, or Genie, which answers natural-language questions without code or SQL. In the reverse direction, engines such as BigQuery can read Unity Catalog managed tables. The text also describes Iceberg catalog labels: key-value metadata on tables and columns can travel between catalogs, preserving governance context such as finance or certified tags and sensitive-data markers. The excerpt presents this shared metadata as useful for governance and agent context, but does not specify setup requirements or limitations.

### Source excerpt

The Lakehouse architecture is becoming more open than ever. What started with open...

## Euler meets Codd

DevFeed: [Euler meets Codd](<https://devfeed.tech/articles/euler-meets-codd-135998.md>)

Original publisher: [Read original article](<https://neo4j.com/blog/developer/euler-meets-codd/>)

Author: christoffer bergman

Published: 2026-11-16T11:43:00Z

Content type: article

Language: en

Sources: [Neo4j](<https://devfeed.tech/sources/graph-database-technology-neo4j-blog.md>)

Topics: [Neo4j](<https://devfeed.tech/topics/neo4j.md>), [graph-database](<https://devfeed.tech/topics/graph-database.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [graph theory](<https://devfeed.tech/topics/graph-theory.md>), [SQL](<https://devfeed.tech/topics/sql.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [aura](<https://devfeed.tech/tags/aura.md>), [aura-graph-analytics](<https://devfeed.tech/tags/aura-graph-analytics.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [cypher](<https://devfeed.tech/tags/cypher.md>), [cypher-gql](<https://devfeed.tech/tags/cypher-gql.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [developer](<https://devfeed.tech/tags/developer.md>), [fraud-detection](<https://devfeed.tech/tags/fraud-detection.md>), [graph](<https://devfeed.tech/tags/graph.md>), [knowledge-graph](<https://devfeed.tech/tags/knowledge-graph.md>), [neo4j](<https://devfeed.tech/tags/neo4j.md>), [query](<https://devfeed.tech/tags/query.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [sql](<https://devfeed.tech/tags/sql.md>), [virtual-graph](<https://devfeed.tech/tags/virtual-graph.md>)

### AI overview

Neo4j Virtual Graphs let users query relational data in Databricks with Cypher and explore it through Neo4j graph tools without copying the data. In a fraud-ring example, the author maps account and transaction tables to graph nodes and relationships, then finds chronologically ordered, decreasing transfers that return to their starting account. A bounded query returned the same rings as a native Neo4j graph, taking about 25 seconds versus 1.5 seconds; the author cautions this was not a scientific benchmark. Virtual Graphs provide graph query expression and visualization, but not native graph traversal speed, and the example required an upper bound because unbounded variable-length queries are unsupported. The described setup uses Aura, a paid feature requiring a credit card. The article says users can later import the data into a native graph using the same data source and model.

### Source excerpt

Neo4j Virtual Graph lets you query relational data in Databricks, Snowflake, BigQuery and similar platforms with Cypher/GQL, using Neo4j's full suite of graph tools.

## What's New in Q4'26 Confluent Data Streaming Platform: The Most Complete, Performant, and Open Foundation for Streamhouse

DevFeed: [What's New in Q4'26 Confluent Data Streaming Platform: The Most Complete, Performant, and Open Foundation for Streamhouse](<https://devfeed.tech/articles/what-s-new-in-q4-26-confluent-data-streaming-platform-the-most-complete-performant-and-open-foundation-for-streamhouse-117672.md>)

Original publisher: [Read original article](<https://www.confluent.io/blog/2026-q4-confluent-bundle-launch/>)

Author: Jason Lauritzen

Published: 2026-11-04T15:55:22Z

Content type: release

Language: en

Sources: [Confluent: Data in motion](<https://devfeed.tech/sources/confluent-data-in-motion.md>)

Topics: [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Confluent Platform](<https://devfeed.tech/topics/confluent-platform.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [private cloud](<https://devfeed.tech/topics/private-cloud.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [cloud](<https://devfeed.tech/tags/cloud.md>), [complete](<https://devfeed.tech/tags/complete.md>), [confluent](<https://devfeed.tech/tags/confluent.md>), [confluent-cloud](<https://devfeed.tech/tags/confluent-cloud.md>), [confluent-platform](<https://devfeed.tech/tags/confluent-platform.md>), [cpc](<https://devfeed.tech/tags/cpc.md>), [data](<https://devfeed.tech/tags/data.md>), [data-streaming](<https://devfeed.tech/tags/data-streaming.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [google](<https://devfeed.tech/tags/google.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [inference](<https://devfeed.tech/tags/inference.md>), [lightning](<https://devfeed.tech/tags/lightning.md>), [managed](<https://devfeed.tech/tags/managed.md>), [open](<https://devfeed.tech/tags/open.md>), [performant](<https://devfeed.tech/tags/performant.md>), [platform](<https://devfeed.tech/tags/platform.md>), [q4](<https://devfeed.tech/tags/q4.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [what-s-new](<https://devfeed.tech/tags/what-s-new.md>)

### AI overview

Confluent's Q4'26 updates position its Data Streaming Platform as a Streamhouse foundation that connects live Kafka data to applications, AI agents and analytics. Lightning Tables, generally available on AWS, provide read-only SQL and API access to continuously updated tables, with stated freshness within hundreds of milliseconds and query latency of 10-100 ms at 1,000+ queries per second. Managed Model Inference is generally available for supported models in Amazon Bedrock, Vertex AI and Azure OpenAI, callable from Flink SQL without configuring credentials or serving infrastructure. Other updates include Tableflow support for Google Cloud and BigQuery with BigLake Metastore synchronization; isolated logical Kafka clusters on shared Confluent Private Cloud infrastructure; and Schema Registry checks intended to catch incompatible changes across Kafka, Flink and lakehouse consumers. The excerpt also describes Cloud disaster recovery and topic-based migration, private-cloud monitoring and security controls, and WarpStream improvements. Availability varies: some features are generally available, while others are early access or open preview; the source cautions that product direction,.

### Source excerpt

Learn about Lightning Tables, Managed Model Inference, multi-tenancy for CPC, Tableflow for Google Cloud, and other Confluent Cloud, Confluent Platform, and WarpStream updates.

## Google fine-tunes its Agentic Data Cloud to support highly scalable autonomous AI workers

DevFeed: [Google fine-tunes its Agentic Data Cloud to support highly scalable autonomous AI workers](<https://devfeed.tech/articles/google-fine-tunes-its-agentic-data-cloud-to-support-highly-scalable-autonomous-ai-workers-116439.md>)

Original publisher: [Read original article](<https://siliconangle.com/2026/11/04/google-fine-tunes-its-agentic-data-cloud-to-support-highly-scalable-autonomous-ai-workers/>)

Author: Mike Wheatley

Published: 2026-11-04T14:00:09Z

Content type: news

Language: en

Sources: [SiliconANGLE](<https://devfeed.tech/sources/siliconangle.md>)

Topics: [Agentic AI](<https://devfeed.tech/topics/what-is-agentic-ai.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>)

Tags: [access-controls](<https://devfeed.tech/tags/access-controls.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-data-cloud](<https://devfeed.tech/tags/agentic-data-cloud.md>), [agentic-infrastructure](<https://devfeed.tech/tags/agentic-infrastructure.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [alloydb](<https://devfeed.tech/tags/alloydb.md>), [amazon-web-services](<https://devfeed.tech/tags/amazon-web-services.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [antigravity](<https://devfeed.tech/tags/antigravity.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [automation](<https://devfeed.tech/tags/automation.md>), [autonomous](<https://devfeed.tech/tags/autonomous.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [enterprise-ai](<https://devfeed.tech/tags/enterprise-ai.md>), [gemini](<https://devfeed.tech/tags/gemini.md>), [google](<https://devfeed.tech/tags/google.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [knowledge-catalog](<https://devfeed.tech/tags/knowledge-catalog.md>), [news](<https://devfeed.tech/tags/news.md>), [the-latest](<https://devfeed.tech/tags/the-latest.md>), [top-story-4](<https://devfeed.tech/tags/top-story-4.md>), [top-story-6](<https://devfeed.tech/tags/top-story-6.md>), [work-automation](<https://devfeed.tech/tags/work-automation.md>)

### AI overview

Google's Agentic Data Cloud updates aim to let autonomous agents work across distributed business data while improving grounding and handling more concurrent queries. Borderless Lakehouse, built on Apache Iceberg, can securely query remote AWS and Azure data and cache retrieved blocks in Google Cloud; an AlloyDB integration queries external Iceberg tables with PostgreSQL without copying data. Knowledge Catalog's self-learning enrichment, in preview, scans unstructured data from Workspace, Jira, SharePoint and GitHub to create context sets; Google reports early adopters saw four-times higher average agent accuracy and up to 75% lower token use. Gemini can generate and repair PySpark pipelines from natural-language goals, and create reusable BigQuery reports that teams can rerun without token costs. The generally available Data Agent Kit adds MCP tooling to several IDEs. Infrastructure additions include preview AlloyDB Express, PostgreSQL for agents in AlloyDB (claimed capacity of three million queries per second), and generally available BigQuery Graph; TimesFM 3.0 and TabFM are in preview.

### Source excerpt

Artificial intelligence is no longer just about conversational assistants that tell people what to do. These days, it's all about work automation, and Google Cloud is making sure it's right at the forefront of that trend with its revamped Agentic Data Cloud platform, which received a host of new updates today. The Agentic Data Cloud [...] The post Google fine-tunes its Agentic Data Cloud to support highly scalable autonomous AI workers appeared first on SiliconANGLE.

## Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg

DevFeed: [Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg](<https://devfeed.tech/articles/migrate-google-bigquery-to-an-open-lakehouse-on-aws-with-apache-iceberg-114202.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/migrate-google-bigquery-to-an-open-lakehouse-on-aws-with-apache-iceberg/>)

Author: Saman Irfan

Published: 2026-11-02T17:21:17Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [AWS Lake Formation](<https://devfeed.tech/topics/aws-lake-formation.md>), [Amazon Bedrock Knowledge Bases](<https://devfeed.tech/topics/amazon-bedrock-knowledge-bases.md>), [AWS Step Functions](<https://devfeed.tech/topics/aws-step-functions.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [data](<https://devfeed.tech/tags/data.md>), [design](<https://devfeed.tech/tags/design.md>), [future](<https://devfeed.tech/tags/future.md>), [google](<https://devfeed.tech/tags/google.md>), [google-bigquery](<https://devfeed.tech/tags/google-bigquery.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [migrate](<https://devfeed.tech/tags/migrate.md>), [migration](<https://devfeed.tech/tags/migration.md>), [open](<https://devfeed.tech/tags/open.md>), [optimize](<https://devfeed.tech/tags/optimize.md>), [over](<https://devfeed.tech/tags/over.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

The article presents a phased path from BigQuery to an AWS lakehouse using Apache Iceberg tables on Amazon S3, with governance through AWS Glue Data Catalog and Lake Formation and engines selected by workload. It recommends assessing tables, query complexity, consumers, volumes, and access controls; designing storage, partitioning, ingestion, and governance; then converting a representative slice and validating results, performance, and cost. For bulk transfer, export BigQuery tables to Parquet in Google Cloud Storage, move them to S3 with AWS DataSync, create or register Iceberg tables, and synchronize incremental changes during transition. Cutover includes parallel validation and deciding whether each consumer will reconnect, be rebuilt, or relay data. The article distinguishes managed S3 Tables, which handle maintenance automatically, from self-managed Iceberg on general-purpose S3, which requires tuning. It also cautions that BigQuery scan-based and Redshift compute-based pricing are not directly comparable, recommending workload-based benchmarks and monthly TCO estimates. Post-migration steps include query and cost optimization.

### Source excerpt

Learn how to migrate Google BigQuery workloads to an open lakehouse on AWS built on Apache Iceberg and Amazon S3. This post explains why an open lakehouse future-proofs your data, presents the target architecture on AWS, and walks through a step-by-step migration: assess, design, convert, migrate, cut over, and optimize.

## Export RUM and Product Analytics data with Export Pipelines

DevFeed: [Export RUM and Product Analytics data with Export Pipelines](<https://devfeed.tech/articles/export-rum-and-product-analytics-data-with-export-pipelines-113321.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/export-pipelines/>)

Author: Stella Ma; Quentin Sellem

Published: 2026-11-02T00:00:00Z

Content type: article

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog-the-monitor-blog.md>)

Topics: [product analytics](<https://devfeed.tech/topics/product-analytics.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [anomaly-detection](<https://devfeed.tech/tags/anomaly-detection.md>), [aws](<https://devfeed.tech/tags/aws.md>), [azure](<https://devfeed.tech/tags/azure.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-storage](<https://devfeed.tech/tags/cloud-storage.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [data-analytics](<https://devfeed.tech/tags/data-analytics.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [digital-experience-monitoring](<https://devfeed.tech/tags/digital-experience-monitoring.md>), [events](<https://devfeed.tech/tags/events.md>), [export](<https://devfeed.tech/tags/export.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [ml](<https://devfeed.tech/tags/ml.md>), [model](<https://devfeed.tech/tags/model.md>), [model-training](<https://devfeed.tech/tags/model-training.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [product-analytics](<https://devfeed.tech/tags/product-analytics.md>), [real-user-monitoring](<https://devfeed.tech/tags/real-user-monitoring.md>), [rum](<https://devfeed.tech/tags/rum.md>), [send](<https://devfeed.tech/tags/send.md>), [storage](<https://devfeed.tech/tags/storage.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

Datadog Export Pipelines continuously send selected RUM and Product Analytics events to Amazon S3, Azure Blob Storage, or Google Cloud Storage, configured in the Datadog UI. They support Parquet, which the source says loads directly into Snowflake, BigQuery, and Databricks without preprocessing, and JSON for custom processing pipelines or a flexible event representation. Files are organized by date and hour; pipelines can be enabled, disabled, or updated, with states shown as Active, Error, Pending, or Paused. Exported data can support warehouse analysis, ML workflows, or long-term retention under an organization's own storage and lifecycle policies. The source states standard RUM retention is 30 days, with a six-month RUM Managed Archive option, while Product Analytics retention is 15 months.

### Source excerpt

Send Datadog RUM and Product Analytics events to your cloud storage for data analysis, ML model training, and long-term data access.

## How to put generative AI directly into a dbt pipeline with plain SQL

DevFeed: [How to put generative AI directly into a dbt pipeline with plain SQL](<https://devfeed.tech/articles/how-to-put-generative-ai-directly-into-a-dbt-pipeline-with-plain-sql-102638.md>)

Original publisher: [Read original article](<https://www.getdbt.com/blog/bigquery-generative-ai-dbt-sql-functions>)

Author: Daniel Poppy

Published: 2026-10-27T19:06:00Z

Content type: article

Language: en

Sources: [dbt Labs](<https://devfeed.tech/sources/dbt-labs.md>)

Topics: [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [Generative AI](<https://devfeed.tech/topics/generative-ai.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [SQL](<https://devfeed.tech/topics/sql.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [JOIN](<https://devfeed.tech/topics/join.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-experiments](<https://devfeed.tech/tags/ai-experiments.md>), [ai-logic](<https://devfeed.tech/tags/ai-logic.md>), [api](<https://devfeed.tech/tags/api.md>), [applied-ai](<https://devfeed.tech/tags/applied-ai.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [functions](<https://devfeed.tech/tags/functions.md>), [generative](<https://devfeed.tech/tags/generative.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [google](<https://devfeed.tech/tags/google.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [json](<https://devfeed.tech/tags/json.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [learn](<https://devfeed.tech/tags/learn.md>), [llm](<https://devfeed.tech/tags/llm.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

BigQuery's AI SQL functions let dbt models apply LLM-based analysis directly to table rows, without a separate Python service. The excerpt describes AI.GENERATE for custom prompts and typed JSON via output_schema, plus managed functions such as AI.IF, AI.CLASSIFY, AI.SCORE, and AI.AGG for common analytical tasks; these are generally cheaper than custom prompts. Optimized AI.IF and AI.CLASSIFY use a sample-labeled distilled model to process the remaining data. For production, the speaker recommends incremental models with AI logic inside is_incremental() to avoid resending all rows on full refresh, token estimates and daily caps to manage spend, and max_error_ratio to stop queries when failures exceed a threshold. Pairing output_schema with dbt tests such as not_null and accepted_values can catch invalid results before they reach marts. The described demo scores review sentiment and summarizes it by product. This overview reflects the supplied excerpt.

### Source excerpt

Google Cloud on wiring BigQuery's AI SQL functions into dbt pipelines, from sentiment scoring to LLM-generated product briefings.

## BigQuery to Databricks: A Strategic Framework

DevFeed: [BigQuery to Databricks: A Strategic Framework](<https://devfeed.tech/articles/bigquery-to-databricks-a-strategic-framework-102249.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/bigquery-databricks-strategic-framework>)

Author: Takero Ibuki

Published: 2026-10-27T11:07:53Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [data migrations](<https://devfeed.tech/topics/data-migrations.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [data mobility](<https://devfeed.tech/topics/data-mobility.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [business-intelligence](<https://devfeed.tech/topics/business-intelligence.md>), [Learning](<https://devfeed.tech/topics/learning.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [3](<https://devfeed.tech/tags/3.md>), [academy](<https://devfeed.tech/tags/academy.md>), [account](<https://devfeed.tech/tags/account.md>), [adopting](<https://devfeed.tech/tags/adopting.md>), [aggregate](<https://devfeed.tech/tags/aggregate.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-assist](<https://devfeed.tech/tags/ai-assist.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [ai-workloads](<https://devfeed.tech/tags/ai-workloads.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-workloads](<https://devfeed.tech/tags/analytics-workloads.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [data](<https://devfeed.tech/tags/data.md>), [data-and-analytics](<https://devfeed.tech/tags/data-and-analytics.md>), [data-warehousing](<https://devfeed.tech/tags/data-warehousing.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [framework](<https://devfeed.tech/tags/framework.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [migration](<https://devfeed.tech/tags/migration.md>), [organizations](<https://devfeed.tech/tags/organizations.md>), [platform](<https://devfeed.tech/tags/platform.md>), [range](<https://devfeed.tech/tags/range.md>), [support](<https://devfeed.tech/tags/support.md>)

### AI overview

The guide recommends a phased BigQuery migration in workload-sized waves, starting with an estate assessment and ending with validation and decommissioning. It describes BI-first teams using Lakehouse Federation to let Databricks read BigQuery during dashboard migration; ETL-first teams can move transformations to Databricks while BigQuery serves curated open-format tables, with external-table limitations such as manual schema refresh. Bulk history can be exported as Parquet to Google Cloud Storage, while frequently updated data can use the Storage API connector or federation. For logic, it proposes rule-based SQL transpilation for routine work, LLM assistance for dialect quirks, and engineers for complex cases. Validation spans row counts, schemas and types, and aggregate and row-level comparisons; the text cautions that SQL function differences can affect hashes. It also recommends migrating permissions before data and setting parity criteria to guide cutover. The supplied excerpt frames governance, interoperability, and workforce training as supporting benefits.

### Source excerpt

Migration as a strategic evolutionMany organizations support a range of data and analytics workloads on BigQuery today...

## Architecting Governed, Self-Evolving AI Agents on Gemini Agent Platform

DevFeed: [Architecting Governed, Self-Evolving AI Agents on Gemini Agent Platform](<https://devfeed.tech/articles/architecting-governed-self-evolving-ai-agents-on-gemini-agent-platform-95939.md>)

Original publisher: [Read original article](<https://developers.googleblog.com/architecting-governed-self-evolving-ai-agents-on-gemini-agent-platform/>)

Author: Anant Nawalgaria

Published: 2026-10-22T16:44:19.101536Z

Content type: article

Language: en

Sources: [Google Developers Blog](<https://devfeed.tech/sources/google-developers-blog.md>)

Topics: [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [google-adk](<https://devfeed.tech/topics/google-adk.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [human in the loop agent](<https://devfeed.tech/topics/human-in-the-loop-agent.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [supervisor agent pattern](<https://devfeed.tech/topics/supervisor-agent-pattern.md>), [agent orchestration](<https://devfeed.tech/topics/agent-orchestration.md>), [hierarchical agent systems](<https://devfeed.tech/topics/hierarchical-agent-systems.md>)

Tags: [adk](<https://devfeed.tech/tags/adk.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-development-kit](<https://devfeed.tech/tags/agent-development-kit.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [authorization](<https://devfeed.tech/tags/authorization.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [defense](<https://devfeed.tech/tags/defense.md>), [defense-in-depth](<https://devfeed.tech/tags/defense-in-depth.md>), [development](<https://devfeed.tech/tags/development.md>), [development-kit](<https://devfeed.tech/tags/development-kit.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [gate](<https://devfeed.tech/tags/gate.md>), [gemini](<https://devfeed.tech/tags/gemini.md>), [google](<https://devfeed.tech/tags/google.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [kms](<https://devfeed.tech/tags/kms.md>), [platform](<https://devfeed.tech/tags/platform.md>)

### AI overview

Project Chrysalis describes a simulated Google Cloud architecture for agents that diagnose cache-stampede outages, synthesize rate-limiting middleware, assess interventions, and retain operational lessons. In the walkthrough, a token-bucket patch passed AST sandbox, OPA policy, and Cloud KMS/Binary Authorization checks before mounting; a proposed permanent agent-graph change paused for dual-custody SRE approval. BigQuery replay compared 400 traffic policies, while step-level attribution penalized a database restart that raised latency and credited the patch. The simulation reports recovery from 72.7-80.0% HTTP 500 errors to 0%, 18.2 ms P99 latency, and a 0.5 ms replay. These are claims about the described demo, not evidence of production deployment or independently verified performance.

### Source excerpt

Learn how Project Chrysalis combines Google Cloud's Gemini Enterprise Agent Platform, Agent Development Kit (ADK), BigQuery, and a four-gate defense-in-depth security stack (Model Armor, Cloud KMS, Binary Authorization, and OPA) to build governed, self-evolving AI agents that remediate live production outages in 1.8 seconds.

## Native Support for Debezium in Apache Beam

DevFeed: [Native Support for Debezium in Apache Beam](<https://devfeed.tech/articles/native-support-for-debezium-in-apache-beam-92451.md>)

Original publisher: [Read original article](<https://debezium.io/blog/2026/10/21/native-support-for-debezium-in-apache-beam/>)

Author: Ilyas Ahsan

Published: 2026-10-21T00:00:00Z

Content type: tutorial

Language: en

Sources: [Debezium](<https://devfeed.tech/sources/debezium-io.md>)

Topics: [Debezium Connector](<https://devfeed.tech/topics/debezium-connector.md>), [Cloud Dataflow](<https://devfeed.tech/topics/cloud-dataflow.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [beam](<https://devfeed.tech/tags/beam.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [debezium](<https://devfeed.tech/tags/debezium.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [mysql](<https://devfeed.tech/tags/mysql.md>)

### AI overview

Apache Beam's DebeziumIO runs Debezium change capture inside a Beam pipeline, without requiring Kafka or Kafka Connect. The excerpt demonstrates a Java pipeline that reads MySQL changes, converts each event to a BigQuery row, and appends it using either the Direct runner or Dataflow. It requires Java 17, Gradle, Docker, and a Google Cloud project with BigQuery enabled; the destination table must exist beforehand. DebeziumIO is available in Beam 2.76.0, supports Debezium 3.5, and the excerpt lists MySQL, PostgreSQL, SQL Server, and Db2 as supported databases. The feature is described as experimental. A noted limitation is that its metadata omits binlog position and row number, which the example says are needed for reliable deduplication, as well as the database commit timestamp useful for debugging.

### Source excerpt

Apache Beam has a built-in connector for Debezium called DebeziumIO. It lets you capture changes from your database and process them directly inside a Beam pipeline. You don't need to run Apache Kafka or Kafka Connect. In this post, we'll build an example pipeline that streams changes from MySQL into Google BigQuery. The complete example is available in the debezium-examples repository. Read more...

## Google aims to make BigQuery spending on AI functions more predictable with token quotas

DevFeed: [Google aims to make BigQuery spending on AI functions more predictable with token quotas](<https://devfeed.tech/articles/google-aims-to-make-bigquery-spending-on-ai-functions-more-predictable-with-token-quotas-87691.md>)

Original publisher: [Read original article](<https://www.infoworld.com/article/4235774/google-aims-to-make-bigquery-spending-on-ai-functions-more-predictable-with-token-quotas.html>)

Author: Anirban Ghoshal

Published: 2026-10-15T11:25:26Z

Content type: news

Language: en

Sources: [InfoWorld](<https://devfeed.tech/sources/infoworld.md>)

Topics: [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-spending](<https://devfeed.tech/tags/ai-spending.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-artificial-intelligence-data-warehousing](<https://devfeed.tech/tags/analytics-artificial-intelligence-data-warehousing.md>), [api](<https://devfeed.tech/tags/api.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [data-warehousing](<https://devfeed.tech/tags/data-warehousing.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [language-en](<https://devfeed.tech/tags/language-en.md>), [limit](<https://devfeed.tech/tags/limit.md>), [quotas](<https://devfeed.tech/tags/quotas.md>), [storytype-news](<https://devfeed.tech/tags/storytype-news.md>), [terraform](<https://devfeed.tech/tags/terraform.md>)

### AI overview

Google has made daily token quotas generally available for BigQuery's generative AI functions, allowing enterprises to cap input and output token use by project and user. The article explains quota defaults, management options, enforcement behavior, and limits of the AI.COUNT_TOKENS estimate. Analysts say quotas improve cost predictability, but teams still need financial oversight and safeguards against interrupted or incomplete queries.

### Source excerpt

Google has made daily token quotas generally available for BigQuery's generative AI functions to give enterprises a way to limit AI consumption by project and individual user. These quotas, which can be managed via the IAM and Admin tab in the Google Cloud Console, cover both input and output tokens at the project and individual-user levels per day and apply to the AI.GENERATE, AI.GENERATE_TABLE, AI.GENERATE_TEXT, AI.CLASSIFY, AI.IF, AI.AGG, and AI.SCORE functions. Last year in November, the hyperscaler added AI-based SQL functions to its managed data warehouse and analytics platform to help enterprises reduce the complexity of running large-scale analytics, especially on unstructured data. These functions, which use LLMs to prepare unstructured data for analysis, reduce the traditionally time-consuming process of preparing such data because SQL cannot understand its nuances and meaning. However, embedding functions such as AI.GENERATE_EMBEDDING, also available as ML.GENERATE_EMBEDDING and AI.EMBED are not subject to these quotas because they run on different, substantially cheaper models, the company wrote in a blog post. For the other functions, by default, input tokens are capped at 200 billion per project and 150 billion per user per day, while output tokens are capped at 20 billion per project and 15 billion per user per day. The output token quotas also include the reasoning tokens used by the LLM, the company wrote, adding that enterprises also have the option to raise a request to increase these default quotas. There is also the option to manage these quotas programmatically through the Cloud Quotas API, the gcloud CLI, and Terraform. How BigQuery enforces token quotas Beyond that, however, there are nuances to how the data warehouse platform enforces token quotas. BigQuery rejects a query with a "QuotaExceeded" error if the token quota it needs has already been exhausted. If the quota runs out while the query is running, BigQuery stops sending new requests

## October 2026 ClickHouse Newsletter: 26.9 Release, Analytics Benchmarks, and UDFs

DevFeed: [October 2026 ClickHouse Newsletter: 26.9 Release, Analytics Benchmarks, and UDFs](<https://devfeed.tech/articles/october-2026-newsletter-87672.md>)

Original publisher: [Read original article](<https://clickhouse.com/blog/202610-newsletter>)

Author: Mark Needham

Published: 2026-10-15T10:07:35Z

Content type: news

Language: en

Sources: [ClickHouse Blog](<https://devfeed.tech/sources/clickhouse-blog.md>)

Topics: [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [Tinybird news](<https://devfeed.tech/topics/tinybird-news.md>)

Tags: [bigquery](<https://devfeed.tech/tags/bigquery.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [events](<https://devfeed.tech/tags/events.md>), [news](<https://devfeed.tech/tags/news.md>), [october-2026](<https://devfeed.tech/tags/october-2026.md>)

### AI overview

The October 2026 ClickHouse newsletter covers ClickHouse 26.9, including LIMIT boundary conditions, incremental materialized view refresh, time-limited credentials, and executable user-defined functions on ClickHouse Cloud. It also summarizes CostBench comparisons with BigQuery and Databricks for real-time analytics, Postgres developments, an Airflow provider, community projects, and upcoming events.

### Source excerpt

Welcome to the October 2026 ClickHouse newsletter, featuring CostBench results against BigQuery and Databricks, ClickHouse 26.9, executable UDFs on ClickHouse Cloud, and the latest community news and events.

## Google Cloud Launches Gemini Agent, One Universal Agent for Enterprise Work

DevFeed: [Google Cloud Launches Gemini Agent, One Universal Agent for Enterprise Work](<https://devfeed.tech/articles/google-cloud-launches-gemini-agent-one-universal-agent-for-enterprise-work-84018.md>)

Original publisher: [Read original article](<https://www.marktechpost.com/2026/10/08/google-cloud-launches-gemini-agent-one-universal-agent-for-enterprise-work/>)

Author: Asif Razzaq

Published: 2026-10-09T06:47:56Z

Content type: news

Language: en

Sources: [MarkTechPost](<https://devfeed.tech/sources/marktechpost.md>)

Topics: [agent orchestration](<https://devfeed.tech/topics/agent-orchestration.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [Cloud Architecture](<https://devfeed.tech/topics/cloud-architecture.md>)

Tags: [agent-gateway](<https://devfeed.tech/tags/agent-gateway.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-shorts](<https://devfeed.tech/tags/ai-shorts.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [applications](<https://devfeed.tech/tags/applications.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [data-agents](<https://devfeed.tech/tags/data-agents.md>), [editors-pick](<https://devfeed.tech/tags/editors-pick.md>), [for-devs](<https://devfeed.tech/tags/for-devs.md>), [gemini](<https://devfeed.tech/tags/gemini.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [introduced](<https://devfeed.tech/tags/introduced.md>), [new-releases](<https://devfeed.tech/tags/new-releases.md>), [tech-news](<https://devfeed.tech/tags/tech-news.md>), [technology](<https://devfeed.tech/tags/technology.md>)

### AI overview

Google Cloud introduced a cloud hosted Gemini agent for enterprise work that handles knowledge tasks, media creation and code through one prompt box and API. The article describes its model routing, persistent memory, subagent orchestration, Workspace integrations and governance controls, including identity, authorization, auditing and project spend caps. It reports a Bloomberg Media example but says reproducible benchmarks, pricing and a general availability date are not provided.

### Source excerpt

Google Cloud has introduced the Google Cloud Gemini agent, a single agent for enterprise work. The Gemini agent is a cloud-hosted agent from Google Cloud that answers questions, does knowledge work, creates media, and writes and runs code. It does all of this from 1 prompt box and 1 API. For developers, the agent is [...] The post Google Cloud Launches Gemini Agent, One Universal Agent for Enterprise Work appeared first on MarkTechPost.

## New public datasets available in Google Cloud Lakehouse

DevFeed: [New public datasets available in Google Cloud Lakehouse](<https://devfeed.tech/articles/new-public-datasets-available-in-google-cloud-lakehouse-62963.md>)

Original publisher: [Read original article](<http://opensource.googleblog.com/2026/09/new-public-datasets-available-in-google-cloud-lakehouse.html>)

Author: Google Open Source (noreply@blogger.com)

Published: 2026-09-30T18:30:00Z

Content type: release

Language: en

Sources: [Google Open Source Blog](<https://devfeed.tech/sources/google-open-source-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Open Data](<https://devfeed.tech/topics/open-data.md>)

Tags: [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [biglake](<https://devfeed.tech/tags/biglake.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [commits](<https://devfeed.tech/tags/commits.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [github](<https://devfeed.tech/tags/github.md>), [google](<https://devfeed.tech/tags/google.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [open-data](<https://devfeed.tech/tags/open-data.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [public](<https://devfeed.tech/tags/public.md>), [pyiceberg](<https://devfeed.tech/tags/pyiceberg.md>), [python](<https://devfeed.tech/tags/python.md>)

### AI overview

Google Cloud Lakehouse offers public Wikipedia pageview and GitHub commit datasets as Apache Iceberg tables stored in Parquet, for exploring data or benchmarking query engines such as BigQuery, Spark, and Trino. The excerpt shows how to inspect a table's schema with PyIceberg and query the datasets with serverless Managed Service for Apache Spark. Access requires a Google Cloud project ID for quota authentication and Application Default Credentials; the examples use the BigLake REST catalog and vended credentials. The excerpt provides setup and sample queries, but does not specify dataset-wide coverage or performance benchmarks.

### Source excerpt

by Daniel Rodrigues & Alex Stephen, Google Cloud Did you know the Google Chrome Wikipedia page received over 36 million page views in 2025, or that Chromium logged over 3,700 commits in April 2010? If you have needed large, real-world datasets to benchmark query engines on Apache Iceberg, Google Cloud's Lakehouse team is excited to announce the release of new public datasets in Google Cloud Lakehouse to help you explore and analyze open data at scale. Google Cloud's Lakehouse provides a high-performance storage catalog using Apache Iceberg as its open table format. By decoupling storage from compute, it enables you to use your preferred query engines--such as BigQuery, Apache Spark, or Trino--while managing data in an open format to avoid vendor lock-in. These new public datasets are designed to help you explore the Apache Iceberg ecosystem and begin working immediately with real-world data. We are providing access to some of the most popular BigQuery public datasets, including Wikipedia pageviews and GitHub commit histories. Let's dive into how you can start querying them. Prerequisites A Google Cloud project (for authentication). Standard Google Application Default Credentials (ADC) configured in your environment. Explore metadata with PyIceberg To explore table metadata using PyIceberg, install the required Python packages in a virtual environment: sudo apt-get install python3-venv python3 -m venv public_data_lakehouse source public_data_lakehouse/bin/activate pip install google-auth pip install pyiceberg pip install pyarrow After installing the required packages, you can inspect a table's schema using the Python script below. Here is how to inspect the table containing Wikipedia page view data from 2016: from pyiceberg import catalog as pyiceberg_catalog PROJECT = "<YOUR_PROJECT_ID>" NAMESPACE = "wikipedia" TABLE = "pageviews_2016" def load_catalog() -> pyiceberg_catalog.Catalog: properties = { "type": "rest", "uri": "https://biglake.googleapis.com/iceberg/v1/rest

## Google Cloud Storage Intelligence Advisor Is GA, With 24-Hour Anomaly Findings and Batch Jobs Across 1,000 Buckets

DevFeed: [Google Cloud Storage Intelligence Advisor Is GA, With 24-Hour Anomaly Findings and Batch Jobs Across 1,000 Buckets](<https://devfeed.tech/articles/google-cloud-storage-intelligence-advisor-is-ga-with-24-hour-anomaly-findings-and-batch-jobs-across-1-000-buckets-61931.md>)

Original publisher: [Read original article](<https://www.storagereview.com/news/google-cloud-storage-intelligence-advisor-ga-24-hour-anomaly-findings-batch-jobs-1000-buckets>)

Author: Harold Fritts

Published: 2026-09-29T12:11:02Z

Content type: news

Language: en

Sources: [StorageReview.com](<https://devfeed.tech/sources/storagereview-com.md>)

Topics: [Cloud Monitoring](<https://devfeed.tech/topics/cloud-monitoring.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [hosting](<https://devfeed.tech/topics/hosting.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [anomaly-detection](<https://devfeed.tech/tags/anomaly-detection.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-storage](<https://devfeed.tech/tags/cloud-storage.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [devops](<https://devfeed.tech/tags/devops.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>)

### AI overview

Google Cloud has made its Storage Intelligence advisor generally available and expanded managed batch operations for Cloud Storage. The advisor analyzes daily usage and activity baselines to flag anomalies such as request spikes, 429 errors, cross-region egress increases, and consumption above long-term trends. Google reports that more than 6,000 findings were generated across hundreds of customer environments in the last 30 days. The batch engine can apply policy changes across up to 1,000 buckets per project in a single serverless job.

### Source excerpt

Google Cloud has made its Storage Intelligence advisor generally available and expanded batch operations for Google Cloud Storage. The advisor monitors storage environments for operational anomalies, while the enhanced batch engine applies policy changes to up to 1,000 buckets per project in a single managed job. Both features are now available to all Storage Intelligence The post Google Cloud Storage Intelligence Advisor Is GA, With 24-Hour Anomaly Findings and Batch Jobs Across 1,000 Buckets appeared first on StorageReview.com.

## Neo4j previews Virtual Graph for querying governed warehouse data with AI applications

DevFeed: [Neo4j previews Virtual Graph for querying governed warehouse data with AI applications](<https://devfeed.tech/articles/scale-ai-into-production-50064.md>)

Original publisher: [Read original article](<https://neo4j.com/blog/auradb/scale-ai-into-production/>)

Author: terri schlosser

Published: 2026-09-17T12:45:00Z

Content type: release

Language: en

Sources: [Neo4j](<https://devfeed.tech/sources/graph-database-technology-neo4j-blog.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Neo4j](<https://devfeed.tech/topics/neo4j.md>), [Graphs](<https://devfeed.tech/topics/graphs.md>), [data](<https://devfeed.tech/topics/data.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [Google](<https://devfeed.tech/topics/google.md>), [clickhouse](<https://devfeed.tech/topics/clickhouse.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [aura-graph-analytics](<https://devfeed.tech/tags/aura-graph-analytics.md>), [auradb](<https://devfeed.tech/tags/auradb.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [graph-data-science](<https://devfeed.tech/tags/graph-data-science.md>), [graph-database](<https://devfeed.tech/tags/graph-database.md>), [graphrag](<https://devfeed.tech/tags/graphrag.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [knowledge-graph](<https://devfeed.tech/tags/knowledge-graph.md>), [neo4j](<https://devfeed.tech/tags/neo4j.md>), [preview](<https://devfeed.tech/tags/preview.md>), [sql](<https://devfeed.tech/tags/sql.md>), [virtual-graph](<https://devfeed.tech/tags/virtual-graph.md>)

### AI overview

Neo4j describes Virtual Graph, a public-preview capability for Aura customers that creates and queries knowledge graphs directly over data in Snowflake, Databricks, and Google BigQuery. It uses zero-copy access, preserves existing governance, and supports natural-language or Cypher queries translated into SQL.

### Source excerpt

Getting an AI application into production is only part of the journey. As organizations move beyond their first applications, the challenges shift. It's no longer one graph behind one application. It's dozens of workloads across teams, tenants, and environments; availability... Read more ->

## How a Neo4j semantic layer reduces Text2SQL token costs across BigQuery and Databricks

DevFeed: [How a Neo4j semantic layer reduces Text2SQL token costs across BigQuery and Databricks](<https://devfeed.tech/articles/cut-text2sql-token-costs-up-to-81-and-stop-paying-for-wrong-answers-a-neo4j-semantic-layer-across-bigquery-and-databricks-50083.md>)

Original publisher: [Read original article](<https://neo4j.com/blog/genai/cut-text2sql-token-costs-up-to-81-and-stop-paying-for-wrong-answers-a-neo4j-semantic-layer/>)

Author: Jocelyn Hoppa

Published: 2026-09-07T08:00:00Z

Content type: article

Language: en

Sources: [Neo4j](<https://devfeed.tech/sources/graph-database-technology-neo4j-blog.md>)

Topics: [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>), [text2sql](<https://devfeed.tech/topics/text2sql.md>), [Neo4j](<https://devfeed.tech/topics/neo4j.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Graphs](<https://devfeed.tech/topics/graphs.md>), [metric-standardization](<https://devfeed.tech/topics/metric-standardization.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [databases](<https://devfeed.tech/tags/databases.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [developer](<https://devfeed.tech/tags/developer.md>), [genai](<https://devfeed.tech/tags/genai.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [knowledge-graph](<https://devfeed.tech/tags/knowledge-graph.md>), [looker](<https://devfeed.tech/tags/looker.md>), [neo4j](<https://devfeed.tech/tags/neo4j.md>), [semantic-layer](<https://devfeed.tech/tags/semantic-layer.md>), [text2sql](<https://devfeed.tech/tags/text2sql.md>)

### AI overview

The article describes a Neo4j-based semantic layer that helps Text2SQL agents discover relevant tables and business context across BigQuery and Databricks. It reports lower token usage, lower projected costs, and improved answer accuracy in tests involving large, difficult catalogs.

### Source excerpt

Across two deliberately hostile catalogs (278 Census tables on BigQuery and 264 legacy tables on Databricks), a Neocarta semantic layer in Neo4j makes Text2SQL agents faster, cheaper, and more accurate. On Census it cut tokens up to 81% while lifting... Read more ->

## We Cut Cloud Waste Before Touching Cluster Sizes: Lessons from Running a Data Platform

DevFeed: [We Cut Cloud Waste Before Touching Cluster Sizes: Lessons from Running a Data Platform](<https://devfeed.tech/articles/we-cut-cloud-waste-before-touching-cluster-sizes-lessons-from-running-a-data-platform-26516.md>)

Original publisher: [Read original article](<https://medium.com/engineering-housing/we-cut-cloud-waste-before-touching-cluster-sizes-lessons-from-running-a-data-platform-9ea96a1f9fbe?source=rss----3a69e32e2594---4>)

Author: Deepika Saini

Published: 2026-09-07T06:33:31Z

Content type: article

Language: en

Sources: [Housing.com](<https://devfeed.tech/sources/housing-com.md>)

Topics: [finops](<https://devfeed.tech/topics/finops.md>), [Platform Engineering](<https://devfeed.tech/topics/platform-engineering.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [databricks](<https://devfeed.tech/topics/databricks.md>)

Tags: [bigquery](<https://devfeed.tech/tags/bigquery.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-computing](<https://devfeed.tech/tags/cloud-computing.md>), [cloud-cost](<https://devfeed.tech/tags/cloud-cost.md>), [cloud-cost-optimization](<https://devfeed.tech/tags/cloud-cost-optimization.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cost-optimization](<https://devfeed.tech/tags/cost-optimization.md>), [costs](<https://devfeed.tech/tags/costs.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-platform](<https://devfeed.tech/tags/data-platform.md>), [data-platforms](<https://devfeed.tech/tags/data-platforms.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [delta](<https://devfeed.tech/tags/delta.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [finops](<https://devfeed.tech/tags/finops.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [jobs](<https://devfeed.tech/tags/jobs.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [platform](<https://devfeed.tech/tags/platform.md>), [waste](<https://devfeed.tech/tags/waste.md>)

### AI overview

The article's main finding is that forgotten storage and unmonitored workloads offered larger cost savings than starting with cluster resizing. After migrating workloads from BigQuery, the team removed datasets still incurring storage charges and reduced Delta retention from 90 to 7 days for workloads whose recovery needs allowed it, after validation. A metadata-driven `config.metadata` framework standardized DMS-to-Delta CDC ingestion and merges. Consolidating DMS instances, switching eligible workloads from Multi-AZ to Single-AZ, and retiring old migration infrastructure cut DMS costs by over 50% without affecting delivery. Databricks System Tables revealed long-running jobs; the team stopped obsolete jobs after checking with owners and optimized useful ones, prioritizing the ten longest-running. Weekly Slack reports surface runtime offenders, while automation pauses jobs after three consecutive failures and alerts their creators. The article also recommends tagging workloads by team, product, environment, pipeline, and cost center to make spending attributable.

### Source excerpt

How orphaned BigQuery storage, Delta retention, DMS right-sizing, and Databricks System Tables became our biggest cloud cost wins. The biggest cloud cost optimization we made wasn't shrinking clusters.It was deleting data we'd forgotten we were paying for.Like most teams, our first instinct was to tune infrastructure first. Instead, we discovered a treasure trove of hidden costs: orphaned BigQuery datasets, 90-day Delta retention, 24-hour jobs no one monitored, and DMS infrastructure that no longer matched business needs.We stopped treating cloud bills as a finance problem and started treating them as a platform engineering problem.30-second takeaway Why deleting forgotten data saved more than shrinking clusters. How we cut DMS costs by over 50%. How Databricks System Tables exposed hidden 24-hour jobs. How config.metadata standardized pipeline onboarding. How weekly Slack alerts turned cost optimization into a habit. Section 1: Storage Was Our Biggest Leak -- We Were Paying to Store Data Nobody Used This is the most overlooked cost on many data platforms. Storage duplication across platforms We had already migrated several workloads from BigQuery to Databricks. Large datasets were still sitting in BigQuery long after they had stopped serving production workloads - quietly generating storage costs month after month. Nothing failed. No alerts fired. Every month, we paid for storage that no longer served production workloads.A migration isn't complete until the old storage is decommissioned.The hidden cost of long retention The next surprise came from Delta Lake retention settings. Our workspace was configured to retain deleted table data and transaction history for 90 days to support time travel. Time travel is incredibly useful. But did every table need three months of historical recovery? Not really. We reduced retention to 7 days for appropriate workloads after validating operational needs. What changed immediately: Less storage tied up in deleted data. Faster clea

## From Raw Tables to Conversational Analytics in One Afternoon

DevFeed: [From Raw Tables to Conversational Analytics in One Afternoon](<https://devfeed.tech/articles/from-raw-tables-to-conversational-analytics-in-one-afternoon-80254.md>)

Original publisher: [Read original article](<https://dlthub.com/blog/bigquery-property-graph-dlt>)

Author: Roshni Melwani

Published: 2026-09-01T00:00:00Z

Content type: tutorial

Language: en

Sources: [DLT Hub](<https://devfeed.tech/sources/dlt-hub.md>)

Topics: [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [JOIN](<https://devfeed.tech/topics/join.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>)

Tags: [bigquery](<https://devfeed.tech/tags/bigquery.md>), [database](<https://devfeed.tech/tags/database.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [gemini](<https://devfeed.tech/tags/gemini.md>), [gql](<https://devfeed.tech/tags/gql.md>), [graph](<https://devfeed.tech/tags/graph.md>), [models](<https://devfeed.tech/tags/models.md>), [ontology](<https://devfeed.tech/tags/ontology.md>), [tutorials](<https://devfeed.tech/tags/tutorials.md>)

### AI overview

The article demonstrates a workflow that loads the Chinook relational dataset into BigQuery with dlt, creates canonical views, declares a property graph, and queries it with GQL. It also tests BigQuery Conversational Analytics on a natural-language question, reporting a correct answer and generated query. The article argues that resolving business meaning in the canonical model helps the graph and conversational interface produce useful results.

### Source excerpt

We previously explained in detail how to build canonical models. This blog takes that further: declare a BigQuery property graph over those models, query it with GQL, then ask it questions in plain English. The ontology resolved in the first step is what makes the answers right at the end.

[Next page](<https://devfeed.tech/topics/bigquery.md?cursor=WyIyMDI2LTA5LTAxVDAwOjAwOjAwKzAwOjAwIiwgImUwMGIxYmM4LWZkYmEtNDhjOS04ZWQ3LWQ0MTczMzExZmRjZiJd>)