# Lakehouse Published articles for Lakehouse. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Why Keep a Data Warehouse If You Are Adopting a Data Lakehouse? DevFeed: [Why Keep a Data Warehouse If You Are Adopting a Data Lakehouse?]() Original publisher: [Read original article]() Published: 2026-11-05T22:52:02Z Content type: article Language: en Sources: [Starburst Data]() Topics: [data lake](), [Amazon Redshift](), [data migrations](), [Data pipelines](), [Open Data](), [etl]() Tags: [agents](), [ai](), [analytics](), [apache](), [apache-iceberg](), [apache-iceberg-tables](), [batch](), [bi](), [bigquery](), [business-intelligence](), [cloud-data-warehouse](), [concurrency](), [concurrent](), [cost](), [costs](), [data](), [data-lakehouse](), [data-warehouse](), [lakehouse](), [warehouse]() ### AI overview The article recommends keeping a cloud data warehouse and an Iceberg-based lakehouse side by side, matching each workload to its needs and moving workloads gradually. Warehouses such as Snowflake, Redshift, and BigQuery remain suited to small, stable, governed data and high-concurrency BI, where managed performance may justify proprietary storage and indexing. Their costs can grow faster than useful query work due to idle or oversized clusters, egress, and duplicated data and pipelines. Open Iceberg tables add schema enforcement, ACID transactions, and time travel on object storage; they may better fit large or semi-structured data and shared access across teams or platforms. Iceberg's optimistic concurrency suits batch and merge-style writes, so frequent small-row writes need evaluation. The suggested low-risk approach is to query both platforms in place through a federated SQL layer, compare cost and performance on candidate workloads, then migrate incrementally based on evidence. ### Source excerpt Adopting a data lakehouse doesn't mean retiring the cloud data warehouse on day one. Most ... ## Oracle rethinks the lakehouse as AI agents become data consumers DevFeed: [Oracle rethinks the lakehouse as AI agents become data consumers]() Original publisher: [Read original article]() Author: Ryan Stevens Published: 2026-11-04T19:39:13Z Content type: news Language: en Sources: [SiliconANGLE]() Topics: [semantic-layer](), [AI-Ready Data](), [Artificial Intelligence](), [Data Management](), [Oracle Database]() Tags: [2026](), [agentic-ai](), [agents](), [ai](), [ai-agents](), [ai-data-platform](), [ai-inference](), [ai-infrastructure](), [ai-security](), [analysts](), [analytics](), [apache](), [apache-iceberg](), [artificial-intelligence](), [autonomous-ai-lakehouse](), [cloud-infrastructure](), [cube-event-coverage](), [data](), [data-governance](), [data-lake-accelerator](), [dave-vellante](), [enterprise-ai](), [enterprise-automation](), [enterprises](), [fast](), [formats](), [generative-ai](), [hybrid-cloud](), [lakehouse](), [multicloud](), [news](), [nipun-agarwal](), [nyse-orcl](), [oracle](), [oracle-ai-database-26ai](), [oracle-ai-world-2026](), [oracle-fusion-cloud-applications](), [oracleaiworld](), [semantic](), [semantic-layer](), [siliconangle](), [sovereign-cloud](), [systems](), [thecube]() ### AI overview Oracle describes its Autonomous AI Lakehouse as an open semantic layer across existing data sources, intended to support both analysts and AI agents without replacing or copying the underlying systems. Its catalog can mount sources including Databricks Unity Catalog, Postgres and MySQL so data is queried in place; generated table and column descriptions can cover non-Oracle sources. Oracle says Lake Cache accelerates queries by moving frequently used Iceberg tables into Exadata flash cache. The company argues that on-demand AI agents increase the need for real-time access, since scheduled refreshes of copied data may be inadequate. The excerpt reports Oracle's approach and claims, not independent performance results. ### Source excerpt A shared semantic layer is fast becoming the dividing line between lakehouses built for reporting and those built for artificial intelligence. As AI agents join human analysts in querying corporate data, enterprises need platforms that make sense of information scattered across clouds, formats and rival systems. Oracle Corp. has spent the past year rebuilding its [...] The post Oracle rethinks the lakehouse as AI agents become data consumers appeared first on SiliconANGLE. ## A Tale of Two Houses: A Comparison Between Streamhouse and Lakehouse DevFeed: [A Tale of Two Houses: A Comparison Between Streamhouse and Lakehouse]() Original publisher: [Read original article]() Author: Alex Stuart Published: 2026-11-04T15:53:00Z Content type: comparison Language: en Sources: [Confluent: Data in motion]() Topics: [streaming data analytics](), [data lake](), [Apache Kafka](), [deltalake](), [Confluent Cloud]() Tags: [about](), [access-control](), [account](), [acid](), [act](), [agent](), [agentic](), [agents](), [ai](), [ai-agents](), [ai-models](), [architectures](), [compare](), [comparison](), [confluent-cloud](), [data](), [houses](), [lakehouse](), [learn](), [one](), [other]() ### AI overview A Streamhouse captures, validates, enriches, and governs each change as it happens, then serves current data to operational applications and agents in the formats they need. A lakehouse stores historical data in object storage with open table formats and supports large batch queries, business intelligence, and model training. Their latency and reliability needs differ: Streamhouse outages can stop production workflows, while lakehouse delays typically make reports or models stale. For customer 360 use cases, the text assigns long-range analysis to the lakehouse and time-sensitive questions--such as fraud checks or live personalization--to the Streamhouse. The two can work together: the Streamhouse shares curated Iceberg or Delta tables for lakehouse use, avoiding a separate ingestion job. The text presents them as complementary architectures, with the Streamhouse for current operational context and the lakehouse for historical analysis. ### Source excerpt Compare streamhouse and lakehouse data architectures, learn which workloads each one fits and how they complement each other. ## Knowledge Graph Lakehouse: A Cloud-Native Architecture for Managing Enterprise Knowledge Graphs at Scale DevFeed: [Knowledge Graph Lakehouse: A Cloud-Native Architecture for Managing Enterprise Knowledge Graphs at Scale]() Original publisher: [Read original article]() Author: Bashar Ahmad, Christoph G. Schuetz, Loris Bozzato Published: 2026-11-04T05:00:00Z Content type: article Language: en Sources: [arXiv: Databases]() Topics: [Cloud Architecture](), [big-data](), [data lake](), [Microservices](), [Big data infrastructure]() Tags: [architecture](), [caching](), [cloud-native](), [cloud-native-architecture](), [cluster](), [configuration](), [context](), [cs-db](), [data](), [data-analysis](), [data-lake](), [data-lakehouse](), [data-warehouse](), [deployment](), [dimensions](), [distributed](), [enterprise](), [experiments](), [knowledge-graphs](), [lakehouse](), [scale]() ### AI overview The Knowledge Graph Lakehouse constructs application-specific contextualized knowledge graphs on demand, rather than maintaining one monolithic graph. It indexes incoming documents along configurable context dimensions and stores them in a data lake; independently deployable microservices use configuration for context dimensions and document formats, keeping the architecture domain-independent. The authors report a proof of concept evaluated in experiments on a distributed cluster, but the supplied abstract gives no performance results. ### Source excerpt arXiv:2611.04899v1 Announce Type: new Abstract: Modern data architectures for integrating heterogeneous data sources to discover actionable knowledge include the data warehouse, the data lake, and the data lakehouse, which combines the flexibility of the data lake with the structured data management capabilities of the data warehouse through metadata, indexes, and caching. Enterprise knowledge graphs (KGs) offer an alternative approach to such integration for the purposes of data analysis, but handling them at scale must address the Three Vs of big data: volume, velocity, and variety. Contextualized KGs (CKGs) meet part of that challenge by enriching facts with contextual information, yet many applications require only a context-specific subset of that knowledge. We propose the Knowledge Graph Lakehouse (KG Lakehouse), a cloud-native microservice architecture that adapts the data lakehouse paradigm for management of CKGs at scale. The KG Lakehouse indexes incoming source documents along configurable context dimensions and stores these documents in a data lake, from which application-specific CKGs are constructed on demand instead of maintaining a single, monolithic KG. We realize the proposed architecture as a proof of concept, with independently deployable services built from established cloud technology; context dimensions and format-specific document handling enter as configuration, leaving the architecture independent of the application domain. We evaluate the implemented system in experiments on a distributed cluster deployment. ## SpBench: An Agent-Native Framework for Cross-Engine Lakehouse Benchmarking DevFeed: [SpBench: An Agent-Native Framework for Cross-Engine Lakehouse Benchmarking]() Original publisher: [Read original article]() Author: Lyndon Shi, Nic Crouch, Yi Hu, Zihao Ye, Choden Konigsmark, Eric Vandenberg, Dionysios Logothetis, Kuba Niemiec, Krzysztof Gadomski, Yuxiong He Published: 2026-11-03T05:00:00Z Content type: article Language: en Sources: [arXiv: Databases](), [arXiv: Machine Learning]() Topics: [Phoronix Test Suite](), [Apache Spark]() Tags: [agent](), [agents](), [apache](), [apache-spark](), [arxiv](), [benchmark](), [benchmarking](), [benchmarks](), [building](), [cadence](), [code](), [collaboration](), [connect](), [continuous](), [cross](), [cs-db](), [cs-lg](), [cs-pf](), [custom](), [engine](), [framework](), [lakehouse](), [native]() ### AI overview SpBench uses an agent-native framework to make cross-engine lakehouse benchmarking repeatable and actionable. It compiles platform-agnostic benchmark definitions into comparable, platform-specific implementations, records runs as structured, queryable telemetry, and provides three composable agent skills for building benchmarks and analyzing results. In production, it benchmarks Snowpark, Snowpark Connect for Apache Spark, and other execution environments across standard suites such as TPCx-BB and TPC-DI, as well as custom workloads. The authors report that it enabled more than 400 regularly run benchmark queries, helping detect regressions whose identification directly led to production improvements. ### Source excerpt arXiv:2611.00449v1 Announce Type: new Abstract: SpBench is an agent-native framework for cross-engine lakehouse benchmarking, in production use for benchmarking Snowpark, Snowpark Connect for Apache Spark, and other execution environments across standardized suites -- such as TPCx-BB and TPC-DI -- and custom benchmarks distilled from production workload patterns. It runs continuously across a broad portfolio of workloads, guiding the performance roadmap and detecting regressions for production-scale data engines. Benchmarking has historically been an expensive investment requiring continuous cross-functional collaboration and domain expertise, making it difficult to obtain actionable performance insights. SpBench provides (i) a typed, code-generated benchmark substrate that compiles platform-agnostic benchmark definitions into fair, comparable platform-specific implementations; (ii) a structured-telemetry contract recording every run as queryable fact and dimension data, letting LLM agents query results deterministically; and (iii) an agent-native harness of three composable skills to expedite building benchmarks and analyzing results. This agent-native architecture enabled the rapid development of more than 400 benchmark queries that run on a regular cadence to detect performance regressions; catching these has directly led to improvements in production. ## Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg DevFeed: [Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg]() Original publisher: [Read original article]() Author: Saman Irfan Published: 2026-11-02T17:21:17Z Content type: article Language: en Sources: [AWS Big Data Blog]() Topics: [Apache Iceberg](), [BigQuery](), [Amazon Web Services](), [Amazon S3 Tables](), [AWS Glue](), [AWS Lake Formation](), [Amazon Bedrock Knowledge Bases](), [AWS Step Functions](), [Amazon SageMaker](), [parquet](), [Amazon MSK](), [Apache Kafka]() Tags: [advanced-300](), [amazon](), [amazon-athena](), [amazon-redshift](), [amazon-s3](), [apache](), [apache-iceberg](), [architecture](), [aws](), [bigquery](), [data](), [design](), [future](), [google](), [google-bigquery](), [iceberg](), [lakehouse](), [migrate](), [migration](), [open](), [optimize](), [over](), [technical-how-to](), [with]() ### AI overview The article presents a phased path from BigQuery to an AWS lakehouse using Apache Iceberg tables on Amazon S3, with governance through AWS Glue Data Catalog and Lake Formation and engines selected by workload. It recommends assessing tables, query complexity, consumers, volumes, and access controls; designing storage, partitioning, ingestion, and governance; then converting a representative slice and validating results, performance, and cost. For bulk transfer, export BigQuery tables to Parquet in Google Cloud Storage, move them to S3 with AWS DataSync, create or register Iceberg tables, and synchronize incremental changes during transition. Cutover includes parallel validation and deciding whether each consumer will reconnect, be rebuilt, or relay data. The article distinguishes managed S3 Tables, which handle maintenance automatically, from self-managed Iceberg on general-purpose S3, which requires tuning. It also cautions that BigQuery scan-based and Redshift compute-based pricing are not directly comparable, recommending workload-based benchmarks and monthly TCO estimates. Post-migration steps include query and cost optimization. ### Source excerpt Learn how to migrate Google BigQuery workloads to an open lakehouse on AWS built on Apache Iceberg and Amazon S3. This post explains why an open lakehouse future-proofs your data, presents the target architecture on AWS, and walks through a step-by-step migration: assess, design, convert, migrate, cut over, and optimize. ## The Lakehouse Isn't Enough. Agents Need an Intelligence Layer of Their Own. DevFeed: [The Lakehouse Isn't Enough. Agents Need an Intelligence Layer of Their Own.]() Original publisher: [Read original article]() Published: 2026-10-30T18:27:05Z Content type: opinion Language: en Sources: [Starburst Data]() Topics: [ai-governance](), [semantic-layer](), [ai observability](), [Data Management](), [AI for data automation]() Tags: [2](), [3](), [agent](), [agents](), [ai](), [ai-agents](), [all](), [allowed](), [analyst](), [autonomous](), [autonomous-agents](), [autonomy](), [cto](), [engineering](), [intelligence](), [isn-t](), [lakehouse](), [layer](), [martin](), [need]() ### AI overview The excerpt argues that autonomous agents need an intelligence layer above individual data platforms because a lakehouse alone cannot govern work that crosses systems. It frames defensibility around three needs: shared business meaning, access checks across the task, and a trace linking what the agent read and did. In its fraud example spanning PostgreSQL, Snowflake, and Salesforce, separate platform rules and logs leave gaps; the proposed layer would use a certified definition of "flagged," check access before a Salesforce read, and provide one trace across all three systems. The excerpt presents this as a way for analysts to stand behind agent answers, while acknowledging that accuracy alone is insufficient. ### Source excerpt Our CTO, Martin Traverso, and SVP of Engineering, Jitender Aswani, recently argued that AI agents ... ## The Missing AI Layer in Your Iceberg Lakehouse DevFeed: [The Missing AI Layer in Your Iceberg Lakehouse]() Original publisher: [Read original article]() Author: Vinod S Published: 2026-10-30T17:01:04Z Content type: tutorial Language: en Sources: [Snowflake Builders]() Topics: [AI Integration Strategies](), [Apache Iceberg](), [elt](), [Synthetic Data Generation]() Tags: [ai](), [apache-iceberg](), [build](), [building](), [builds](), [business](), [call](), [can](), [catalog](), [change](), [changes](), [complete](), [custom](), [customers](), [dashboard](), [data](), [data-engineering](), [data-products](), [data-team](), [dynamic-table](), [iceberg](), [lakehouse](), [layer](), [llm](), [snowflake]() ### AI overview The post demonstrates a Snowflake pipeline that runs AI functions inside incremental dynamic Iceberg tables, calling models for new or changed rows and exposing results to Iceberg readers. A frozen region preserves completed outputs through prompt changes; the demo reprocessed 16 active rows while leaving 10 frozen summaries unchanged. It recommends placing AI_FILTER in an upstream SELECT and filtering its computed flag in a downstream table: using the function in WHERE either fails in incremental mode or can silently select a full refresh. The example also covers access requirements, NULL checks for failed model calls, and the limitation that outside engines can read these tables but cannot write to them. Results are from a demo with synthetic tickets and may differ in other workloads. ### Source excerpt AI Functions in dynamic Iceberg tables: pay the model only for changed rows On Monday, someone improves the summary prompt. It's a good change: the output is shorter, with no chatty preamble. On Tuesday, the backfill starts. It re-summarizes every ticket since spring, at full token price. On Wednesday, it's still running. Last quarter's report no longer matches the dashboard. The backfill reworded some old summaries, even at temperature 0. Nobody touched those tickets. The pipeline rewrote them anyway. On Thursday, two questions arrive from outside the data team: "Can we improve a prompt without paying for every ticket again?" No. With this pipeline, every backfill pays for your whole history again, and history only grows. "Can we improve a prompt without changing last quarter's numbers?" No. Every prompt change reopens a report that you already shipped. This post builds a pipeline where both answers are yes. TL;DR: Put AI_CLASSIFY, AI_FILTER and AI_COMPLETE in the SELECT of a Snowflake dynamic Iceberg table. The AI step then becomes a layer of your lakehouse that any Iceberg engine can read, not a job beside it. Snowflake calls the model only for new and changed rows. A frozen region keeps the old outputs when the prompt changes. You'll build a ticket-enrichment pipeline from two CREATE statements, with no scheduler, history table or backfill script. Customers already use Dynamic Iceberg Tables to turn changing source data into fresh, business-ready datasets in open formats, like current-state network inventories, clinical data products, and regional delivery feeds, without building custom merge pipelines. With externally managed dynamic Iceberg tables, now in Private Preview, Snowflake runs those same declarative, target-lag refreshes and publishes the results straight into your existing external Iceberg catalog, so you keep your catalog and still get Snowflake-managed pipelinesAI Functions inside a dynamic Iceberg table call the model only for changed rows. You'l ## AI-Assisted Reconciliation of Enterprise Data: Detecting and Explaining Cross-Platform Discrepancies DevFeed: [AI-Assisted Reconciliation of Enterprise Data: Detecting and Explaining Cross-Platform Discrepancies]() Original publisher: [Read original article]() Author: ajaysikharam Published: 2026-10-30T06:00:10Z Content type: article Language: en Sources: [HackerNoon]() Topics: [Machine learning](), [Data Quality]() Tags: [2](), [2026](), [3](), [about](), [account](), [ai](), [ai-assisted-reconciliation](), [all](), [arithmetic](), [audit](), [audit-trail](), [away](), [background](), [batch](), [battery](), [better](), [data](), [data-engineering](), [data-quality](), [data-reconciliation](), [data-regulations](), [data-science](), [explain](), [lakehouse](), [learn](), [reconciliation](), [work]() ### AI overview The article recommends establishing reconciliation facts with repeatable checks before using AI to explain differences. Run row counts, measure sums, join-key coverage, and sampled row hashes on each load; keep a structured log of each discrepancy, its cause, resolution, and reviewer. Historical resolutions can inform per-check variance baselines and let an LLM draft evidence-linked explanations, label known patterns, and rank unresolved issues by impact, novelty, and trend. In the example, checks identify missing orders between the source and lakehouse; prior incidents suggest an early extract, while the dashboard's remaining gap is attributed to cache lag. Reviewers should verify exceptions, spot-check automatically labeled normal differences, and retain audit evidence because explanations can be wrong and poor closure notes can teach the model mistakes. The added compute and maintenance may not pay off for small pipelines or low alert volumes; the AI layer can wait until the resolution log has history. ### Source excerpt Learn how deterministic checks and AI work together to detect data discrepancies, explain their causes, and simplify financial reconciliation. ## Unlocking Investment Insights Without Migrating Your Data: Lakehouse Federation DevFeed: [Unlocking Investment Insights Without Migrating Your Data: Lakehouse Federation]() Original publisher: [Read original article]() Author: Meghana Padmanabhan; Isaac Gritz; Max Nienu Published: 2026-10-27T18:00:00Z Content type: tutorial Language: en Sources: [Databricks]() Topics: [databricks](), [AI for SQL query generation](), [Data pipelines](), [etl](), [AI Chat](), [data-governance](), [enterprise data protection](), [Grafana]() Tags: [about](), [accessible](), [actions](), [agent](), [all](), [analysis](), [analyst](), [analytics](), [analyze](), [asset](), [based](), [blog](), [build](), [building](), [customers](), [data](), [enterprise](), [financial-services](), [insights](), [investment](), [lakehouse](), [platform](), [query](), [solutions](), [tutorials]() ### AI overview Databricks Lakehouse Federation lets teams query on-premises data alongside cloud-shared data through Unity Catalog without copying the source data. The excerpt's investment example connects a local database such as SQL Server as a foreign catalog, then joins portfolio holdings with FactSet datasets subscribed to through Databricks Marketplace. Queries run against the on-premises source in real time; dashboards re-run the combined query on refresh, and a Genie agent can translate natural-language questions into queries using the same tables and configured join and data-quality instructions. The described setup avoids pipelines, materialized views, and persisted tables, but the source must be connected and FactSet access approved. The excerpt does not detail performance, cost, or operational tradeoffs. ### Source excerpt Lakehouse Federation: Query Anywhere, Move NothingIn today's enterprise landscape,... ## BigQuery to Databricks: A Strategic Framework DevFeed: [BigQuery to Databricks: A Strategic Framework]() Original publisher: [Read original article]() Author: Takero Ibuki Published: 2026-10-27T11:07:53Z Content type: article Language: en Sources: [Databricks]() Topics: [data migrations](), [BigQuery](), [databricks](), [data-architecture](), [data mobility](), [deltalake](), [business-intelligence](), [Learning](), [CI/CD]() Tags: [3](), [academy](), [account](), [adopting](), [aggregate](), [ai](), [ai-assist](), [ai-models](), [ai-workloads](), [analytics](), [analytics-workloads](), [bigquery](), [data](), [data-and-analytics](), [data-warehousing](), [databricks](), [framework](), [lakehouse](), [migration](), [organizations](), [platform](), [range](), [support]() ### AI overview The guide recommends a phased BigQuery migration in workload-sized waves, starting with an estate assessment and ending with validation and decommissioning. It describes BI-first teams using Lakehouse Federation to let Databricks read BigQuery during dashboard migration; ETL-first teams can move transformations to Databricks while BigQuery serves curated open-format tables, with external-table limitations such as manual schema refresh. Bulk history can be exported as Parquet to Google Cloud Storage, while frequently updated data can use the Storage API connector or federation. For logic, it proposes rule-based SQL transpilation for routine work, LLM assistance for dialect quirks, and engineers for complex cases. Validation spans row counts, schemas and types, and aggregate and row-level comparisons; the text cautions that SQL function differences can affect hashes. It also recommends migrating permissions before data and setting parity criteria to guide cutover. The supplied excerpt frames governance, interoperability, and workforce training as supporting benefits. ### Source excerpt Migration as a strategic evolutionMany organizations support a range of data and analytics workloads on BigQuery today... ## Article: AI Changes the Data Interface, Not the Query Engine DevFeed: [Article: AI Changes the Data Interface, Not the Query Engine]() Original publisher: [Read original article]() Author: Peihao Yuan Published: 2026-10-22T09:00:00Z Content type: article Language: en Sources: [InfoQ]() Topics: [DuckDB](), [reproducibility](), [Disaster Recovery](), [parquet](), [llm-reasoning](), [Data pipelines](), [Query (disambiguation)](), [Apache Spark]() Tags: [agent](), [agentic-ai-architecture](), [ai](), [ai-agent](), [ai-architecture](), [ai-changes-data-interface](), [ai-ml-data-engineering](), [analysis](), [analytics](), [architecture](), [architecture-design](), [article](), [changes](), [data](), [data-lake](), [database](), [disaster](), [disaster-recovery](), [duckdb](), [embedded](), [engine](), [interface](), [lakehouse](), [managed](), [ml-data-engineering](), [parquet](), [presents](), [query](), [query-engine]() ### AI overview In the disaster-recovery workflow, DuckDB read Parquet and deterministically calculated traffic, availability, and latency facts; an AI agent generated constrained SQL or Python, interpreted those facts in service context, and explained the results. The workflow retained inputs, configuration, code, execution metadata, facts, and model outputs so analyses could be replayed and checked. The author argues an embedded engine can suffice for moderate, focused workloads with controlled writes. A shared catalog, open table format, or distributed engine becomes useful when independent consumers, large shared joins, concurrent writes, schema evolution, or organization-wide governance create concrete needs; these capabilities add integration and operational costs. ### Source excerpt This article presents a disaster-recovery analytics workflow in which an AI agent generates constrained SQL or Python, DuckDB computes reproducible facts from Parquet, and the agent interprets the results. It explains the determinism split, replayable analysis, and four signals for deciding when an embedded engine is sufficient or a managed lakehouse is justified. By Peihao Yuan ## What makes a catalog open? Unity Catalog Advances the Open Lakehouse DevFeed: [What makes a catalog open? Unity Catalog Advances the Open Lakehouse]() Original publisher: [Read original article]() Author: Elizabeth Bowman Published: 2026-10-19T21:20:00Z Content type: article Language: en Sources: [Databricks]() Topics: [Databricks Unity Catalog integration](), [Apache Iceberg](), [deltalake]() Tags: [cloud-storage](), [costs](), [customer](), [customers](), [data-movement](), [databricks](), [delta-lake](), [iceberg](), [iceberg-rest-catalog](), [lakehouse](), [open-source](), [platform-products-announcements]() ### AI overview The article defines an open data catalog by three properties: data stays in storage the customer owns, the catalog provides its full feature set there, and tables can move to another catalog without copying data. It describes how Unity Catalog uses Apache Iceberg REGISTER and UNREGISTER for portability and contrasts this approach with provider-controlled storage. ### Source excerpt While almost every platform now supports open formats, what makes a catalog truly... ## How to get the most out of running dbt on Databricks DevFeed: [How to get the most out of running dbt on Databricks]() Original publisher: [Read original article]() Author: Srilekha Dornadula; Luís Cabral Published: 2026-10-19T19:45:59Z Content type: tutorial Language: en Sources: [Databricks]() Topics: [databricks](), [database-optimization](), [API Governance](), [CI/CD]() Tags: [access-control](), [agent](), [automate](), [automation](), [code](), [cost](), [data-transformation](), [databricks](), [dbt](), [engineering](), [features](), [lakehouse](), [partners](), [platform-partners]() ### AI overview The article presents practices for improving dbt pipelines on Databricks, including incremental processing, materialized views, automated table maintenance, query cost attribution, and integrated ingestion, orchestration, and governance. It describes these as ways to reduce compute and operational work while improving pipeline performance. ### Source excerpt Tens of thousands of organizations rely on dbt for data transformation. Where you... ## Transactions are Generally Available, advancing warehousing on the Lakehouse DevFeed: [Transactions are Generally Available, advancing warehousing on the Lakehouse]() Original publisher: [Read original article]() Author: Benjamin Mathew; Prakhar Jain; Sumeet Varma Published: 2026-10-16T21:00:00Z Content type: release Language: en Sources: [Databricks]() Topics: [Transactions](), [deltalake](), [database-consistency](), [Streaming](), [scalable-architecture]() Tags: [access-controls](), [analytics](), [apache-iceberg](), [apache-iceberg-tables](), [cloud](), [concurrency](), [cost](), [data-ingestion](), [data-pipelines](), [data-warehousing](), [documentation](), [engineering-manager](), [lakehouse](), [platform](), [transactions]() ### AI overview Databricks announces general availability of multi-statement transactions for Unity Catalog managed Delta Lake and Apache Iceberg tables. The feature provides atomic guarantees across multiple statements and tables, supports row-level concurrent writes, and integrates with governance controls, warehouse development tools, data pipelines, and observability. The article describes use cases such as ETL updates, stored procedures, streaming workflows, and write-audit-publish processes. ### Source excerpt Multi-Statement Transactions are Generally Available, enabling organizations to run... ## Komprise Survey of Unstructured Data and AI Reveals Wide Gap Between Perceived AI Maturity and AI Results DevFeed: [Komprise Survey of Unstructured Data and AI Reveals Wide Gap Between Perceived AI Maturity and AI Results]() Original publisher: [Read original article]() Author: SD Times Newswire Published: 2026-10-16T18:45:34Z Content type: news Language: en Sources: [SD Times]() Topics: [AI-Ready Data](), [data-governance](), [data lake](), [databricks]() Tags: [ai](), [ai-maturity](), [ai-roi](), [data-readiness](), [databricks](), [enterprise-ai](), [enterprise-data](), [infrastructure](), [lakehouse](), [newswire](), [schema](), [unstructured-data]() ### AI overview Komprise reports survey results showing a gap between enterprises' perceived AI maturity and measurable outcomes: 68% say their AI programs are maturing or mature, while only 28% have quantified AI ROI on average. Respondents also report substantial unstructured-data readiness challenges, including data quality, classification, metadata enrichment, and preparation for lakehouse analysis. ### Source excerpt Campbell, California -- Komprise, the leader in analytics-driven unstructured data management, today announced the results of a new survey on AI data management and data readiness. Enterprise organizations say they are largely maturing or mature with AI (68%), yet the ROI numbers tell a different story, as only 28% of organizations on average have quantified... ... continue reading The post Komprise Survey of Unstructured Data and AI Reveals Wide Gap Between Perceived AI Maturity and AI Results appeared first on SD Times. ## Reducing the noise: How Atlassian built an OCSF security lakehouse on Databricks DevFeed: [Reducing the noise: How Atlassian built an OCSF security lakehouse on Databricks]() Original publisher: [Read original article]() Author: Taylor Kain; Zachary Schmerber (Atlassian); Niels Heijmans (Atlassian) Published: 2026-10-15T20:50:26Z Content type: article Language: en Sources: [Databricks]() Topics: [databricks](), [Security](), [cloud cost management](), [Logging](), [DevSecOps](), [observability](), [shadow AI]() Tags: [ai](), [customers](), [cybersecurity](), [databricks](), [lakehouse](), [logging](), [noise](), [ocsf](), [scale](), [security](), [the-challenge]() ### AI overview Atlassian standardized fragmented security logs with the Open Cybersecurity Schema Framework (OCSF) and built a Databricks lakehouse to process its high-volume pipeline. The article describes how typed schemas, ETL, and automated workflows improved query speed, reduced costs, and enabled standardized alerting and analysis. ### Source excerpt The challenge: Two decades of logging noise at global scale Atlassian is a global... ## Geospatial queries are up to 38X faster with Lakehouse//RT DevFeed: [Geospatial queries are up to 38X faster with Lakehouse//RT]() Original publisher: [Read original article]() Author: Kent Marten; Matt Slack; Kaitlin Baumgardner Published: 2026-10-15T17:31:02Z Content type: release Language: en Sources: [Databricks]() Topics: [geospatial](), [olap](), [clickhouse]() Tags: [benchmark](), [concurrency](), [geospatial](), [iceberg](), [lakehouse](), [sql]() ### AI overview Databricks says Lakehouse//RT now supports Spatial SQL and spatial joins in beta, with benchmarked queries up to 38 times faster than a traditional cloud data warehouse under heavy concurrency. It reports an average 3 times faster performance across tested queries, and describes using native GEOMETRY data types to enable query optimizations and file skipping. ### Source excerpt Spatial SQL and spatial joins now run on Lakehouse//RT and they're up to 38X faster... ## How to Architect an On-Premises Data Lakehouse DevFeed: [How to Architect an On-Premises Data Lakehouse]() Original publisher: [Read original article]() Published: 2026-10-15T08:00:34Z Content type: article Language: en Sources: [Starburst Data]() Topics: [data lake](), [database-optimization](), [acid](), [Transactions](), [Cache](), [control-plane](), [parquet](), [AI Infrastructure](), [AI-Ready Data]() Tags: [access-control](), [ai](), [ai-workloads](), [air-gapped](), [amazon-s3](), [analytics](), [apache-iceberg](), [apache-iceberg-tables](), [architect](), [audit](), [availability](), [bandwidth](), [batch](), [benchmark](), [bi](), [big-bang](), [build](), [built-in](), [data-lakehouse](), [lakehouse]() ### AI overview An on-premises lakehouse can serve sovereignty, regulatory, latency, or egress-cost needs using S3-compatible object storage, Apache Iceberg, and Trino, while separating storage from compute. Plan capacity, durability, network bandwidth, and the Iceberg catalog early; compatibility varies by provider: open-source Trino is tested with Amazon S3 and MinIO, while Starburst Enterprise documents its supported stores. The article recommends starting federation-first, then materializing frequently used paths into Iceberg, and building access controls, audit, and governed data products in from day one. Migrate from Hadoop in stages: replace Hive or Impala with Trino while HDFS remains, write new data to Iceberg, selectively convert Hive tables, then retire legacy components. Starburst Enterprise multi-cluster availability features require Elite and were in public preview in 480-e; running queries may fail during coordinator failover, so the article advises testing recovery plans that do not depend on those features. ### Source excerpt An on-premises data lakehouse brings lakehouse capabilities into data centers you own or lease. It ... ## A practical guide to real-time analytics in 2026 DevFeed: [A practical guide to real-time analytics in 2026]() Original publisher: [Read original article]() Author: Jeremy Lewallen; Tad Rosenberg; Kaitlin Baumgardner Published: 2026-10-13T19:17:26Z Content type: article Language: en Sources: [Databricks]() Topics: [real-time](), [clickhouse](), [data lake](), [olap](), [product analytics]() Tags: [access-controls](), [analytics](), [architecture](), [batch-processing](), [concurrency](), [dashboards](), [data](), [data-analytics](), [decision-making](), [lakehouse](), [real-time](), [real-time-analytics]() ### AI overview Real-time analytics helps teams query fresh data with low latency and high concurrency. This guide explains how to assess workloads across those dimensions, compares real-time and batch analytics, and describes the operational overhead of separate serving layers. It presents a lakehouse approach for running real-time and traditional analytics on shared data and governance, with Databricks Lakehouse//RT as an example. ### Source excerpt Data has always been central to business decision-making, and most analytical workloads... ## From Anomaly Detection to Agentic AI: Engineering Intelligent Data Reliability on Databricks DevFeed: [From Anomaly Detection to Agentic AI: Engineering Intelligent Data Reliability on Databricks]() Original publisher: [Read original article]() Author: Sanya Kapoor Published: 2026-10-13T10:49:51Z Content type: article Language: en Sources: [HackerNoon]() Topics: [databricks](), [deltalake](), [data-engineering](), [Analytics Pipeline](), [streaming data analytics](), [LLM observability]() Tags: [agentic-ai](), [agentic-ai-for-data-engineers](), [ai-engineering](), [anomaly-detection](), [data](), [data-reliability](), [databricks](), [databricks-data-reliability](), [delta-lake-architecture](), [good-company](), [governed-enterprise-ai](), [intelligent-data-operations](), [lakehouse](), [lokeshkumar-madabathula](), [real-time](), [real-time-anomaly-detection]() ### AI overview The article describes a proposed Databricks architecture for improving data reliability by combining streaming ingestion, Delta Lake, anomaly detection and a governed agentic AI layer. After an anomaly is detected, the AI layer can gather operational context, assist with root-cause investigation and downstream impact assessment, and recommend remediation. Production actions remain subject to permissions and human oversight. ### Source excerpt Explore how Lokeshkumar Madabathula combines Databricks, real-time anomaly detection, lakehouse architecture, and governed Agentic AI for data reliability. ## Article: When Should You Use Liquid Clustering? Benchmarking Lakehouse Layout Strategies DevFeed: [Article: When Should You Use Liquid Clustering? Benchmarking Lakehouse Layout Strategies]() Original publisher: [Read original article]() Author: Kushal Kumar Vishwakarma Published: 2026-10-12T11:00:00Z Content type: article Language: en Sources: [InfoQ]() Topics: [deltalake](), [table-partitioning](), [clustering]() Tags: [ai](), [ai-ml-data-engineering](), [application-performance-management](), [architecture-design](), [article](), [benchmark](), [benchmarking](), [clustering-caching](), [data-lake](), [data-storage](), [delta-lake](), [engineering](), [infrastructure](), [lakehouse](), [liquid-clustering-lakehouse-layout](), [ml-data-engineering](), [partitioning](), [performance](), [performance-evaluation](), [query]() ### AI overview The article benchmarks Liquid Clustering, ZOrdering, and partitioning on Delta Lake across four analytical workloads. Liquid Clustering has the lowest latency for point lookups, time-range aggregations, and large aggregations, while ZOrder performs best for selective multicolumn queries whose filters match its clustering keys. The author recommends validating layout choices against representative query patterns and notes that the benchmark uses synthetic data, so results may vary in production. ### Source excerpt Physical data layout is one of the most consequential decisions in lakehouse engineering. The author compares Liquid Clustering, ZOrdering, and partitioning across four workload types on Delta Lake, uncovering two undocumented failure modes that likely affect production systems today and offering a decision framework for choosing the right layout strategy. By Kushal Kumar Vishwakarma ## How to Repoint dbt ETL Pipelines to Databricks DevFeed: [How to Repoint dbt ETL Pipelines to Databricks]() Original publisher: [Read original article]() Author: Khagay Nagdimov; Ismail Makhlouf; Shweta Verma Published: 2026-10-06T18:53:48Z Content type: tutorial Language: en Sources: [Databricks]() Topics: [databricks](), [etl](), [Data pipelines](), [CI/CD]() Tags: [data-warehousing](), [databricks](), [dbt](), [etl](), [etl-pipelines](), [how-to](), [lakehouse](), [pipelines]() ### AI overview This tutorial explains how to move a dbt transformation project to Databricks by configuring the adapter and connection profile, mapping namespaces, and addressing SQL dialect differences. It recommends validating models against the existing warehouse, comparing outputs, then migrating in batches and cutting over after validation. ### Source excerpt More teams are running their dbt transformations on Databricks Lakehouse: an open... ## The lakehouse is a better data warehouse: 2026 benchmarks and proof DevFeed: [The lakehouse is a better data warehouse: 2026 benchmarks and proof]() Original publisher: [Read original article]() Author: Joanna Zouhour Published: 2026-10-06T16:35:12Z Content type: comparison Language: en Sources: [Databricks]() Topics: [deltalake](), [Amazon Redshift](), [streaming data analytics](), [data observability](), [clean energy data centers]() Tags: [2026](), [benchmarks](), [data](), [data-warehouse](), [lakehouse]() ### AI overview The article argues that lakehouse platforms can now match or exceed data warehouses for many SQL workloads while also serving BI, streaming, machine learning, and AI workloads from shared governed data. It cites Databricks performance benchmarks and customer migration results, describes the lakehouse's support for open table formats and multiple data types, and acknowledges that warehouses retain advantages for some workloads. ### Source excerpt The warehouse was the right answer for 20 yearsThe data warehouse earned its place. For two decades... [Next page]()