# deltalake

Delta Lake is an open-source storage layer and framework for building lakehouse architectures on data lakes, providing ACID transactions and unified streaming and batch processing.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## A modern dilemma: Consume data from a stream or a lake

DevFeed: [A modern dilemma: Consume data from a stream or a lake](<https://devfeed.tech/articles/a-modern-dilemma-consume-data-from-a-stream-or-a-lake-141161.md>)

Original publisher: [Read original article](<https://medium.com/capital-one-tech/a-modern-dilemma-consume-data-from-a-stream-or-a-lake-296bbb354094?source=rss----3db3a67cb648---4>)

Author: Capital One Tech

Published: 2026-11-18T17:35:14Z

Content type: article

Language: en

Sources: [Capital One Tech](<https://devfeed.tech/sources/capital-one-tech.md>)

Topics: [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [etl](<https://devfeed.tech/topics/etl.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [apis](<https://devfeed.tech/tags/apis.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [batch](<https://devfeed.tech/tags/batch.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [business-processes](<https://devfeed.tech/tags/business-processes.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-computing](<https://devfeed.tech/tags/cloud-computing.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-lakes](<https://devfeed.tech/tags/data-lakes.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>), [enrichment](<https://devfeed.tech/tags/enrichment.md>), [etl](<https://devfeed.tech/tags/etl.md>), [event](<https://devfeed.tech/tags/event.md>), [flink](<https://devfeed.tech/tags/flink.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [latency](<https://devfeed.tech/tags/latency.md>), [legacy](<https://devfeed.tech/tags/legacy.md>), [log](<https://devfeed.tech/tags/log.md>), [mapping](<https://devfeed.tech/tags/mapping.md>), [model-context-protocol](<https://devfeed.tech/tags/model-context-protocol.md>), [software-architecture](<https://devfeed.tech/tags/software-architecture.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [stream](<https://devfeed.tech/tags/stream.md>)

### AI overview

Choosing a data consumption pattern early can avoid costly architectural changes later. APIs support synchronous, record-by-record requests and paginated multirecord results, making them suitable for low-latency needs; Model Context Protocol can provide similar access for agents and is often an API wrapper. Event streams, commonly built with Kafka, Flink or Spark Streaming, suit sub-second latency and enable per-event calculations, transformations and enrichment. The excerpt also identifies Delta Change Data Feed (CDF) as a record-by-record pattern and traditional batch ETL as a multirecord pattern, but ends before explaining CDF's mechanics or comparing these approaches further.

### Source excerpt

Data is a critical component of software architecture, providing the foundation for business processes, including those powered by artificial intelligence (AI). Quality, volume, latency, consistency, ordering and duplicates are essential considerations in software design. Modern cloud architectures consume data in various ways, such as through application programming interfaces (APIs), streams and data lakes. You can consume data at the individual event or record level as it arrives or in larger batches. Legacy architectures often process data in scheduled batches, forcing surrounding architecture and business processes to depend on that approach. Getting the data consumption approach right up front is vital, because rearchitecting later is costly. General patterns to consume data There are several general data consumption patterns. These include APIs (record-by-record or multirecord results), event stream consumption (record-by-record), Delta Change Data Feed (CDF) consumption (record-by-record) or traditional batch ETL-extract, transform, load (multirecord results). APIs are ideal for low-latency, record-by-record requirements. They also support multirecord results, typically through pagination. APIs usually follow a synchronous approach, where the client makes a call and waits for a response before moving to the next step. Model Context Protocol is an industry specification for agents to consume data and can support record-by-record or batch results via pagination, similar to APIs. They are often a wrapper on top of APIs. The event stream pattern works well for sub-second latency requirements and typically runs on a tech stack like Kafka, Flink or Spark Streaming. These technologies excel at record-level calculations (like on-the-fly calculation), transformations (like column mapping) or enrichment (like adding data to an existing event). Delta Lake is often used in data lakes and serves as a transaction log to standard Apache Parquet data files (source). CDF is

## Fivetran Launches Support for New Databricks + GCP Offering | Blog | Fivetran

DevFeed: [Fivetran Launches Support for New Databricks + GCP Offering | Blog | Fivetran](<https://devfeed.tech/articles/fivetran-launches-support-for-new-databricks-gcp-offering-blog-fivetran-141313.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/fivetran-launches-support-for-new-databricks-gcp-offering>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: release

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [billing](<https://devfeed.tech/tags/billing.md>), [choice](<https://devfeed.tech/tags/choice.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-marketplace](<https://devfeed.tech/tags/cloud-marketplace.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [data-science-and-analytics](<https://devfeed.tech/tags/data-science-and-analytics.md>), [data-scientists](<https://devfeed.tech/tags/data-scientists.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [gcp](<https://devfeed.tech/tags/gcp.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [launches](<https://devfeed.tech/tags/launches.md>), [platform](<https://devfeed.tech/tags/platform.md>), [support](<https://devfeed.tech/tags/support.md>)

### AI overview

Fivetran now supports Databricks on Google Cloud. Customers can purchase Fivetran through Google Cloud Marketplace, use Google Cloud credits, and receive unified billing for Google Cloud products and Fivetran purchases made there. The article also says all Fivetran connectors support Databricks Delta Lake and that Marketplace purchasing simplifies procurement. The offering is presented as expanding customer choice across Google Cloud and Databricks; no technical setup details are provided.

### Source excerpt

Businesses can now use Fivetran with the Lakehouse Platform on Google Cloud.

## Fivetran Named Databricks Innovation Partner | Blog | Fivetran

DevFeed: [Fivetran Named Databricks Innovation Partner | Blog | Fivetran](<https://devfeed.tech/articles/fivetran-named-databricks-innovation-partner-blog-fivetran-141310.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/fivetran-databricks-innovation-partner-award-winner-announces-sql-analytics-plans>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: news

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [award](<https://devfeed.tech/tags/award.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [innovation](<https://devfeed.tech/tags/innovation.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [lakehouse-architecture](<https://devfeed.tech/tags/lakehouse-architecture.md>), [partner](<https://devfeed.tech/tags/partner.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

Fivetran received Databricks' 2020 ISV Partners Innovation Award for its collaboration around Delta Lake and the Lakehouse architecture. The article says Fivetran automates ingestion, including handling changing SaaS APIs and schemas, and can create normalized source tables, capture historical changes through history mode, and provide prebuilt dbt transformations. It says setup requires minimal configuration and does not require customers to create a staging area. Planned work included dbt packages and job orchestration for Databricks SQL, which was presented as a way for analysts to run SQL-based pipelines and access data through BI tools; those capabilities were described as upcoming. The article also reports customer claims of near-real-time syncing and savings equivalent to two to three data engineers per year.

### Source excerpt

We're honored to win this prestigious award, and we're doubling down on the Lakehouse architecture with Databricks SQL plans.

## Databricks is a Leader and positioned highest in execution in the 2026 Gartner® Magic Quadrant™ for Data Lakehouse Platforms

DevFeed: [Databricks is a Leader and positioned highest in execution in the 2026 Gartner® Magic Quadrant™ for Data Lakehouse Platforms](<https://devfeed.tech/articles/databricks-is-a-leader-and-positioned-highest-in-execution-in-the-2026-gartner-magic-quadranttm-for-data-lakehouse-platforms-136417.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/databricks-leader-and-positioned-highest-execution-2026-gartnerr-magic-quadranttm-data>)

Author: Kaitlin Baumgardner; Kevin Balaji; Brianna Helm; Jonathan Keller; Susan Pierce

Published: 2026-11-14T18:15:02Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2026](<https://devfeed.tech/tags/2026.md>), [3](<https://devfeed.tech/tags/3.md>), [about](<https://devfeed.tech/tags/about.md>), [acid](<https://devfeed.tech/tags/acid.md>), [affiliates](<https://devfeed.tech/tags/affiliates.md>), [ai](<https://devfeed.tech/tags/ai.md>), [company](<https://devfeed.tech/tags/company.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [decades](<https://devfeed.tech/tags/decades.md>), [enterprises](<https://devfeed.tech/tags/enterprises.md>), [execution](<https://devfeed.tech/tags/execution.md>), [gartner](<https://devfeed.tech/tags/gartner.md>), [gartner-magic-quadranttm](<https://devfeed.tech/tags/gartner-magic-quadranttm.md>), [lakebase](<https://devfeed.tech/tags/lakebase.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [lakehouse-architecture](<https://devfeed.tech/tags/lakehouse-architecture.md>), [leader](<https://devfeed.tech/tags/leader.md>), [ltap](<https://devfeed.tech/tags/ltap.md>), [magic](<https://devfeed.tech/tags/magic.md>), [operational](<https://devfeed.tech/tags/operational.md>), [platforms](<https://devfeed.tech/tags/platforms.md>)

### AI overview

Databricks argues that AI workloads need operational and analytical data on one governed foundation. Its Lake Transactional/Analytical Processing (LTAP) architecture pairs Lakebase Postgres for ACID transactions with lakehouse engines for analytics, both reading one durable copy in open formats such as Delta Lake and Apache Iceberg under Unity Catalog. The company says this avoids pipelines and mirrored tables while giving analytics fresh application data without slowing transactions. Lakebase Postgres uses separated, serverless compute and storage on cloud object storage, scales compute to zero, and supports instant database branching. Databricks cites PRADA Group's reported KPI delivery at about 130 times faster and infrastructure costs about 50% lower. It also says Gartner named Databricks a Leader with the highest execution scores in its 2026 Data Lakehouse Platforms Magic Quadrant and the only Visionary in its Cloud Operational Database Management Systems Magic Quadrant. These are vendor-reported claims; the text includes Gartner's disclaimer that its research is not an endorsement.

### Source excerpt

AI is pulling analytical and operational data togetherFor decades, enterprises have...

## Towards Unified File Management for Data Lakehouse Systems

DevFeed: [Towards Unified File Management for Data Lakehouse Systems](<https://devfeed.tech/articles/towards-unified-file-management-for-data-lakehouse-systems-132696.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15975>)

Author: Keonwoo Oh, Akshay Mehra, Joshua Kimball, Amol Deshpande

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [Data Management](<https://devfeed.tech/topics/data-management.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [scalable-architecture](<https://devfeed.tech/topics/scalable-architecture.md>), [rdbms](<https://devfeed.tech/topics/rdbms.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [database](<https://devfeed.tech/tags/database.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>), [garbage-collection](<https://devfeed.tech/tags/garbage-collection.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [latency](<https://devfeed.tech/tags/latency.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [metadata-management](<https://devfeed.tech/tags/metadata-management.md>), [raven](<https://devfeed.tech/tags/raven.md>), [snapshot-isolation](<https://devfeed.tech/tags/snapshot-isolation.md>), [tracking](<https://devfeed.tech/tags/tracking.md>), [transactions](<https://devfeed.tech/tags/transactions.md>), [versioning](<https://devfeed.tech/tags/versioning.md>)

### AI overview

Raven combines a transactional database catalog for transactional metadata with underlying Open Table Formats (OTFs) for analytical metadata, aiming to balance database scalability with file-based metadata management. Its extended Merge-on-Read mechanism is designed to reduce commit latency and metadata bloat, while snapshot isolation supports multi-table transactions. An extensible entity-relationship model unifies versioning and reference tracking to enable global garbage collection across asset types. The abstract reports lower commit latency and metadata bloat with Apache Iceberg and Delta Lake, catalog storage orders of magnitude smaller than database-only DuckLake, and scalable garbage collection with minimal disruption; it provides no evaluation details or quantitative measurements.

### Source excerpt

arXiv:2611.15975v1 Announce Type: new Abstract: Data lakehouse systems have garnered widespread adoption, yet their metadata management systems face a fundamental architectural dilemma. File-based OTFs (Open Table Formats) scale well, but suffer from metadata bloat, high commit latencies, the absence of multi-table transactions, and unsafe garbage collection. On the other hand, catalogs relying entirely on a transactional database incur severe scalability bottlenecks due to analytic file-level metadata. To reconcile this tension, we present Raven, an operational catalog that stores transactional metadata within a database while offloading analytical metadata to underlying OTF structures. Based on this hybrid architecture, Raven introduces an extended Merge-on-Read (MoR) mechanism to reduce commit latency and mitigate metadata bloat, alongside multi-table transactions under snapshot isolation. Furthermore, Raven unifies core data lakehouse semantics including versioning and reference tracking under an extensible ER (Entity-Relationship) metadata model, enabling safe global garbage collection across diverse types of assets. Empirical evaluations demonstrate that Raven substantially reduces commit latency and metadata bloat when integrated with Apache Iceberg and Delta Lake, maintains a catalog storage footprint orders of magnitude smaller than database-only systems like DuckLake, and achieves scalable garbage collection with minimal disruption.

## Unity Catalog Metric Views, now in Power BI: one unified, governed semantic layer for all your reporting and AI

DevFeed: [Unity Catalog Metric Views, now in Power BI: one unified, governed semantic layer for all your reporting and AI](<https://devfeed.tech/articles/unity-catalog-metric-views-now-in-power-bi-one-unified-governed-semantic-layer-for-all-your-reporting-and-ai-127421.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/unity-catalog-metric-views-now-power-bi-one-unified-governed-semantic-layer-all-your-reporting>)

Author: Toussaint Webb; Joanna Zouhour; Chris Eubank; Jade Wang

Published: 2026-11-10T18:45:00Z

Content type: release

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>), [metric-standardization](<https://devfeed.tech/topics/metric-standardization.md>), [bi-tools](<https://devfeed.tech/topics/bi-tools.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [all](<https://devfeed.tech/tags/all.md>), [bi](<https://devfeed.tech/tags/bi.md>), [business](<https://devfeed.tech/tags/business.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [definition](<https://devfeed.tech/tags/definition.md>), [layer](<https://devfeed.tech/tags/layer.md>), [metric](<https://devfeed.tech/tags/metric.md>), [one](<https://devfeed.tech/tags/one.md>), [organizations](<https://devfeed.tech/tags/organizations.md>), [platform](<https://devfeed.tech/tags/platform.md>), [power](<https://devfeed.tech/tags/power.md>), [power-bi](<https://devfeed.tech/tags/power-bi.md>), [reporting](<https://devfeed.tech/tags/reporting.md>), [revenue](<https://devfeed.tech/tags/revenue.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [semantic-layer](<https://devfeed.tech/tags/semantic-layer.md>), [semantics](<https://devfeed.tech/tags/semantics.md>), [unity](<https://devfeed.tech/tags/unity.md>), [unity-catalog](<https://devfeed.tech/tags/unity-catalog.md>), [want](<https://devfeed.tech/tags/want.md>)

### AI overview

Publish to Power BI connects Unity Catalog metric views to Power BI semantic models so teams can maintain metric definitions in Databricks while using Power BI for reporting. Each metric view becomes a table, with dimensions as columns and measures as DAX measures; queries run against the metric view, and its joins remain defined in Databricks. Lakeflow Jobs can republish changes on a schedule or after upstream data refreshes. In DirectQuery mode, queries honor Unity Catalog access controls and security policies; in Import mode, access to the copied data follows Power BI permissions. Publishing is one-way: edits made in Power BI do not update Unity Catalog. The feature is described as being in Private Preview.

### Source excerpt

Organizations want a single, trusted definition of their business semantics. Revenue,...

## Transactional Data Change for Real-Time Analytics in Lakehouses

DevFeed: [Transactional Data Change for Real-Time Analytics in Lakehouses](<https://devfeed.tech/articles/transactional-data-change-for-real-time-analytics-in-lakehouses-125674.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.12169>)

Author: Dongyang Geng, Haoqiong Bian, Zinuo Li, Haoyue Li

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>)

Tags: [acid](<https://devfeed.tech/tags/acid.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [data-analytics](<https://devfeed.tech/tags/data-analytics.md>), [data-freshness](<https://devfeed.tech/tags/data-freshness.md>), [data-synchronization](<https://devfeed.tech/tags/data-synchronization.md>), [databases](<https://devfeed.tech/tags/databases.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [file-storage](<https://devfeed.tech/tags/file-storage.md>), [framework](<https://devfeed.tech/tags/framework.md>), [git](<https://devfeed.tech/tags/git.md>), [htap](<https://devfeed.tech/tags/htap.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [real-time-analytics](<https://devfeed.tech/tags/real-time-analytics.md>), [real-time-data](<https://devfeed.tech/tags/real-time-data.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [version-control](<https://devfeed.tech/tags/version-control.md>)

### AI overview

Retina is a data synchronization framework designed to propagate changes into lakehouses with ACID guarantees while limiting resource use and analytical-query overhead. The abstract says existing lakehouses' single-table, git-style catalog versioning trade freshness, query performance, or efficiency for limited change throughput. Retina reportedly achieves freshness at the 10-millisecond level and more than 3.2 million row changes per second with linear scalability, low resource consumption, and low query-performance overhead; evaluations are reported to outperform state-of-the-art lakehouse systems and lake-compatible HTAP databases. The supplied text is an abstract and gives no implementation details or evaluation methodology.

### Source excerpt

arXiv:2611.12169v1 Announce Type: new Abstract: Real-time data analytics has become a key demand for lakehouses. It requires propagating data changes from the data source to the lakehouse in real time, while ensuring ACID guarantees and minimizing resource consumption and impact on analytical query performance. This is highly challenging for a data system according to the RUM conjecture, especially given the read-optimized immutable file storage layer in lakehouses. Existing lakehouses offer only git-style catalog version control on a single table, and they trade data freshness, query performance, and/or resource efficiency to achieve a barely acceptable data-change throughput. In this paper, we propose a novel data synchronization framework, Retina, that enables real-time, cost-efficient, query-performant, and fully transactional data changes in lakehouses. We implement and open-source Retina. Evaluations show that Retina delivers 10-millisecond-level data freshness and over 3.2 million row/s linear-scalable data-change throughput with low resource consumption and query-performance overhead, significantly outperforming state-of-the-art lakehouse systems and lake-compatible HTAP databases.

## Read, don't copy: replacing file feeds with OpenSharing - Part 2 (How to Implement)

DevFeed: [Read, don't copy: replacing file feeds with OpenSharing - Part 2 (How to Implement)](<https://devfeed.tech/articles/read-don-t-copy-replacing-file-feeds-with-opensharing-part-2-how-to-implement-121796.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/read-dont-copy-replacing-file-feeds-opensharing-part-2-how-implement>)

Author: Jordan Soldo; Nikhil Tallam

Published: 2026-11-06T22:13:00Z

Content type: tutorial

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [IO](<https://devfeed.tech/topics/io.md>), [copy](<https://devfeed.tech/topics/copy.md>), [CRUD](<https://devfeed.tech/topics/crud.md>), [CSV](<https://devfeed.tech/topics/csv.md>)

Tags: [access-control](<https://devfeed.tech/tags/access-control.md>), [added](<https://devfeed.tech/tags/added.md>), [admin](<https://devfeed.tech/tags/admin.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [api](<https://devfeed.tech/tags/api.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [automatic](<https://devfeed.tech/tags/automatic.md>), [better](<https://devfeed.tech/tags/better.md>), [bi](<https://devfeed.tech/tags/bi.md>), [bi-tools](<https://devfeed.tech/tags/bi-tools.md>), [blog](<https://devfeed.tech/tags/blog.md>), [copy](<https://devfeed.tech/tags/copy.md>), [data](<https://devfeed.tech/tags/data.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [file](<https://devfeed.tech/tags/file.md>), [implement](<https://devfeed.tech/tags/implement.md>), [pattern](<https://devfeed.tech/tags/pattern.md>), [sharing](<https://devfeed.tech/tags/sharing.md>), [tutorials](<https://devfeed.tech/tags/tutorials.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

For Databricks-to-Databricks sharing, publish a Delta table with history in a share; the recipient mounts it as a catalog and reads it incrementally with Structured Streaming. A checkpoint lets scheduled Trigger.AvailableNow runs process only changes since the previous run, and a table update trigger can run jobs when new data is committed. Durable checkpoint storage is recommended. History is required for incremental reads; Change Data Feed is optional for the basic append-oriented path but needed for the described upserts-and-deletes path. The excerpt cautions that ignoreChanges may re-emit rows and does not propagate deletes. CREATE SHARE and CREATE RECIPIENT require metastore-level privilege. Other recipients use a credentials file; OSS Spark can use the streaming connector, while non-Spark readers pull changes by version range. The excerpt proposes validating a minimal flow with an initial snapshot and subsequent insert, update, and delete.

### Source excerpt

Many data-sharing architectures begin with a familiar pattern: A provider exports...

## What's New in Q4'26 Confluent Data Streaming Platform: The Most Complete, Performant, and Open Foundation for Streamhouse

DevFeed: [What's New in Q4'26 Confluent Data Streaming Platform: The Most Complete, Performant, and Open Foundation for Streamhouse](<https://devfeed.tech/articles/what-s-new-in-q4-26-confluent-data-streaming-platform-the-most-complete-performant-and-open-foundation-for-streamhouse-117672.md>)

Original publisher: [Read original article](<https://www.confluent.io/blog/2026-q4-confluent-bundle-launch/>)

Author: Jason Lauritzen

Published: 2026-11-04T15:55:22Z

Content type: release

Language: en

Sources: [Confluent: Data in motion](<https://devfeed.tech/sources/confluent-data-in-motion.md>)

Topics: [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Confluent Platform](<https://devfeed.tech/topics/confluent-platform.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [private cloud](<https://devfeed.tech/topics/private-cloud.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [cloud](<https://devfeed.tech/tags/cloud.md>), [complete](<https://devfeed.tech/tags/complete.md>), [confluent](<https://devfeed.tech/tags/confluent.md>), [confluent-cloud](<https://devfeed.tech/tags/confluent-cloud.md>), [confluent-platform](<https://devfeed.tech/tags/confluent-platform.md>), [cpc](<https://devfeed.tech/tags/cpc.md>), [data](<https://devfeed.tech/tags/data.md>), [data-streaming](<https://devfeed.tech/tags/data-streaming.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [google](<https://devfeed.tech/tags/google.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [inference](<https://devfeed.tech/tags/inference.md>), [lightning](<https://devfeed.tech/tags/lightning.md>), [managed](<https://devfeed.tech/tags/managed.md>), [open](<https://devfeed.tech/tags/open.md>), [performant](<https://devfeed.tech/tags/performant.md>), [platform](<https://devfeed.tech/tags/platform.md>), [q4](<https://devfeed.tech/tags/q4.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [what-s-new](<https://devfeed.tech/tags/what-s-new.md>)

### AI overview

Confluent's Q4'26 updates position its Data Streaming Platform as a Streamhouse foundation that connects live Kafka data to applications, AI agents and analytics. Lightning Tables, generally available on AWS, provide read-only SQL and API access to continuously updated tables, with stated freshness within hundreds of milliseconds and query latency of 10-100 ms at 1,000+ queries per second. Managed Model Inference is generally available for supported models in Amazon Bedrock, Vertex AI and Azure OpenAI, callable from Flink SQL without configuring credentials or serving infrastructure. Other updates include Tableflow support for Google Cloud and BigQuery with BigLake Metastore synchronization; isolated logical Kafka clusters on shared Confluent Private Cloud infrastructure; and Schema Registry checks intended to catch incompatible changes across Kafka, Flink and lakehouse consumers. The excerpt also describes Cloud disaster recovery and topic-based migration, private-cloud monitoring and security controls, and WarpStream improvements. Availability varies: some features are generally available, while others are early access or open preview; the source cautions that product direction,.

### Source excerpt

Learn about Lightning Tables, Managed Model Inference, multi-tenancy for CPC, Tableflow for Google Cloud, and other Confluent Cloud, Confluent Platform, and WarpStream updates.

## A Tale of Two Houses: A Comparison Between Streamhouse and Lakehouse

DevFeed: [A Tale of Two Houses: A Comparison Between Streamhouse and Lakehouse](<https://devfeed.tech/articles/a-tale-of-two-houses-a-comparison-between-streamhouse-and-lakehouse-117673.md>)

Original publisher: [Read original article](<https://www.confluent.io/blog/a-tale-of-two-houses-streamhouse-lakehouse/>)

Author: Alex Stuart

Published: 2026-11-04T15:53:00Z

Content type: comparison

Language: en

Sources: [Confluent: Data in motion](<https://devfeed.tech/sources/confluent-data-in-motion.md>)

Topics: [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [access-control](<https://devfeed.tech/tags/access-control.md>), [account](<https://devfeed.tech/tags/account.md>), [acid](<https://devfeed.tech/tags/acid.md>), [act](<https://devfeed.tech/tags/act.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [compare](<https://devfeed.tech/tags/compare.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [confluent-cloud](<https://devfeed.tech/tags/confluent-cloud.md>), [data](<https://devfeed.tech/tags/data.md>), [houses](<https://devfeed.tech/tags/houses.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [learn](<https://devfeed.tech/tags/learn.md>), [one](<https://devfeed.tech/tags/one.md>), [other](<https://devfeed.tech/tags/other.md>)

### AI overview

A Streamhouse captures, validates, enriches, and governs each change as it happens, then serves current data to operational applications and agents in the formats they need. A lakehouse stores historical data in object storage with open table formats and supports large batch queries, business intelligence, and model training. Their latency and reliability needs differ: Streamhouse outages can stop production workflows, while lakehouse delays typically make reports or models stale. For customer 360 use cases, the text assigns long-range analysis to the lakehouse and time-sensitive questions--such as fraud checks or live personalization--to the Streamhouse. The two can work together: the Streamhouse shares curated Iceberg or Delta tables for lakehouse use, avoiding a separate ingestion job. The text presents them as complementary architectures, with the Streamhouse for current operational context and the lakehouse for historical analysis.

### Source excerpt

Compare streamhouse and lakehouse data architectures, learn which workloads each one fits and how they complement each other.

## BigQuery to Databricks: A Strategic Framework

DevFeed: [BigQuery to Databricks: A Strategic Framework](<https://devfeed.tech/articles/bigquery-to-databricks-a-strategic-framework-102249.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/bigquery-databricks-strategic-framework>)

Author: Takero Ibuki

Published: 2026-10-27T11:07:53Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [data migrations](<https://devfeed.tech/topics/data-migrations.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [data mobility](<https://devfeed.tech/topics/data-mobility.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [business-intelligence](<https://devfeed.tech/topics/business-intelligence.md>), [Learning](<https://devfeed.tech/topics/learning.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [3](<https://devfeed.tech/tags/3.md>), [academy](<https://devfeed.tech/tags/academy.md>), [account](<https://devfeed.tech/tags/account.md>), [adopting](<https://devfeed.tech/tags/adopting.md>), [aggregate](<https://devfeed.tech/tags/aggregate.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-assist](<https://devfeed.tech/tags/ai-assist.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [ai-workloads](<https://devfeed.tech/tags/ai-workloads.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-workloads](<https://devfeed.tech/tags/analytics-workloads.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [data](<https://devfeed.tech/tags/data.md>), [data-and-analytics](<https://devfeed.tech/tags/data-and-analytics.md>), [data-warehousing](<https://devfeed.tech/tags/data-warehousing.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [framework](<https://devfeed.tech/tags/framework.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [migration](<https://devfeed.tech/tags/migration.md>), [organizations](<https://devfeed.tech/tags/organizations.md>), [platform](<https://devfeed.tech/tags/platform.md>), [range](<https://devfeed.tech/tags/range.md>), [support](<https://devfeed.tech/tags/support.md>)

### AI overview

The guide recommends a phased BigQuery migration in workload-sized waves, starting with an estate assessment and ending with validation and decommissioning. It describes BI-first teams using Lakehouse Federation to let Databricks read BigQuery during dashboard migration; ETL-first teams can move transformations to Databricks while BigQuery serves curated open-format tables, with external-table limitations such as manual schema refresh. Bulk history can be exported as Parquet to Google Cloud Storage, while frequently updated data can use the Storage API connector or federation. For logic, it proposes rule-based SQL transpilation for routine work, LLM assistance for dialect quirks, and engineers for complex cases. Validation spans row counts, schemas and types, and aggregate and row-level comparisons; the text cautions that SQL function differences can affect hashes. It also recommends migrating permissions before data and setting parity criteria to guide cutover. The supplied excerpt frames governance, interoperability, and workforce training as supporting benefits.

### Source excerpt

Migration as a strategic evolutionMany organizations support a range of data and analytics workloads on BigQuery today...

## Why Modern Data Lakehouses Fail at Scale: The Metadata Problem Nobody Talks About

DevFeed: [Why Modern Data Lakehouses Fail at Scale: The Metadata Problem Nobody Talks About](<https://devfeed.tech/articles/why-modern-data-lakehouses-fail-at-scale-the-metadata-problem-nobody-talks-about-98982.md>)

Original publisher: [Read original article](<https://hackernoon.com/why-modern-data-lakehouses-fail-at-scale-the-metadata-problem-nobody-talks-about?source=rss>)

Author: FNU Aditya Ranjan

Published: 2026-10-24T17:00:02Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [database-optimization](<https://devfeed.tech/topics/database-optimization.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-applications](<https://devfeed.tech/tags/ai-applications.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-hudi](<https://devfeed.tech/tags/apache-hudi.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [applications](<https://devfeed.tech/tags/applications.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [delta](<https://devfeed.tech/tags/delta.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>), [files](<https://devfeed.tech/tags/files.md>), [growth](<https://devfeed.tech/tags/growth.md>), [hackernoon-top-story](<https://devfeed.tech/tags/hackernoon-top-story.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [maintenance](<https://devfeed.tech/tags/maintenance.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [metadata-management](<https://devfeed.tech/tags/metadata-management.md>), [modern](<https://devfeed.tech/tags/modern.md>), [modern-data](<https://devfeed.tech/tags/modern-data.md>), [modern-data-lakehouses](<https://devfeed.tech/tags/modern-data-lakehouses.md>), [partitioning](<https://devfeed.tech/tags/partitioning.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [small](<https://devfeed.tech/tags/small.md>)

### AI overview

Lakehouse performance can degrade as file counts, partition fragmentation, and metadata grow, even when object storage scales. Small files increase metadata handling, object operations, scheduling, and scan overhead; compaction can improve layout but costs compute and I/O. Iceberg describes table state through metadata and manifests, while Delta Lake uses a transaction log and checkpoints, so each requires lifecycle maintenance. Partitioning helps pruning only when it fits real workloads; overly granular partitions can add more file and metadata overhead than benefit. Snapshot expiration and orphan cleanup must preserve reader, rollback, and concurrency requirements. The excerpt proposes comparing no proactive optimization, fixed-schedule maintenance, and workload-aware triggers using matched workloads and configurations, measuring latency, planning time, operations, rewrite volume, storage, and cost. This framework is proposed, not experimentally validated.

### Source excerpt

Explore why modern data lakehouses slow down at scale and how metadata growth, small files, partitioning, and maintenance affect Iceberg and Delta Lake.

## Neoclouds: Your Storage Layer Just Got a Public Grade

DevFeed: [Neoclouds: Your Storage Layer Just Got a Public Grade](<https://devfeed.tech/articles/neoclouds-your-storage-layer-just-got-a-public-grade-91064.md>)

Original publisher: [Read original article](<https://www.backblaze.com/blog/neoclouds-your-storage-layer-just-got-a-public-grade/>)

Author: David Johnson

Published: 2026-10-20T14:21:23Z

Content type: article

Language: en

Sources: [Backblaze](<https://devfeed.tech/sources/backblaze-blog-cloud-storage-cloud-backup.md>)

Topics: [Infrastructure as a Service (IaaS)](<https://devfeed.tech/topics/infrastructure-as-a-service-iaas.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [BlueField DPU](<https://devfeed.tech/topics/bluefield-dpu.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-labs](<https://devfeed.tech/tags/ai-labs.md>), [ai-ml](<https://devfeed.tech/tags/ai-ml.md>), [aws](<https://devfeed.tech/tags/aws.md>), [azure](<https://devfeed.tech/tags/azure.md>), [b2cloud](<https://devfeed.tech/tags/b2cloud.md>), [backend](<https://devfeed.tech/tags/backend.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [ceph](<https://devfeed.tech/tags/ceph.md>), [cloud-storage](<https://devfeed.tech/tags/cloud-storage.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [compute](<https://devfeed.tech/tags/compute.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [control-plane](<https://devfeed.tech/tags/control-plane.md>), [coreweave](<https://devfeed.tech/tags/coreweave.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [featured](<https://devfeed.tech/tags/featured.md>), [featured-cloud-storage](<https://devfeed.tech/tags/featured-cloud-storage.md>), [isolation](<https://devfeed.tech/tags/isolation.md>), [neoclouds](<https://devfeed.tech/tags/neoclouds.md>), [performance](<https://devfeed.tech/tags/performance.md>), [providers](<https://devfeed.tech/tags/providers.md>), [storage](<https://devfeed.tech/tags/storage.md>), [strategy](<https://devfeed.tech/tags/strategy.md>)

### AI overview

SemiAnalysis' ClusterMAX 3.0 requires neoclouds to provide both high-performance file and S3-compatible object storage, then tests speed and performance under 15 minutes of mixed traffic; some providers lost nearly 40% of bandwidth. The excerpt says only 19 of 77 providers tested in depth earned a Medallion rating, and describes recurring problems such as ineffective volume RBAC, exposed backend networks, and default DPU settings that can let a GPU-cloud customer reach tenant-isolation controls. It distinguishes deliberate storage design from defaults that went unchecked, citing examples ranging from missing shared filesystems to a fast Lustre tier undermined by a broken default StorageClass. Providers can build on Ceph, build their own storage hardware, or partner: these trade control and staffing or capital commitments differently. The article urges providers to decide storage responsibilities and isolation before customers leave data on the platform.

### Source excerpt

SemiAnalysis' ClusterMAX 3.0 puts neocloud storage under scrutiny. From performance to tenant isolation, default configurations can undermine customer confidence. Explore what the findings reveal and why providers need a deliberate storage strategy before customers leave their data on the platform. The post Neoclouds: Your Storage Layer Just Got a Public Grade appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

## What makes a catalog open? Unity Catalog Advances the Open Lakehouse

DevFeed: [What makes a catalog open? Unity Catalog Advances the Open Lakehouse](<https://devfeed.tech/articles/what-makes-a-catalog-open-unity-catalog-advances-the-open-lakehouse-90219.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/what-makes-catalog-open-unity-catalog-advances-open-lakehouse>)

Author: Elizabeth Bowman

Published: 2026-10-19T21:20:00Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Databricks Unity Catalog integration](<https://devfeed.tech/topics/databricks-unity-catalog-integration.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>)

Tags: [cloud-storage](<https://devfeed.tech/tags/cloud-storage.md>), [costs](<https://devfeed.tech/tags/costs.md>), [customer](<https://devfeed.tech/tags/customer.md>), [customers](<https://devfeed.tech/tags/customers.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [iceberg-rest-catalog](<https://devfeed.tech/tags/iceberg-rest-catalog.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [platform-products-announcements](<https://devfeed.tech/tags/platform-products-announcements.md>)

### AI overview

The article defines an open data catalog by three properties: data stays in storage the customer owns, the catalog provides its full feature set there, and tables can move to another catalog without copying data. It describes how Unity Catalog uses Apache Iceberg REGISTER and UNREGISTER for portability and contrasts this approach with provider-controlled storage.

### Source excerpt

While almost every platform now supports open formats, what makes a catalog truly...

## Pinterest Uses Event Time Watermarks to Finalize Petabyte Scale Data Partitions

DevFeed: [Pinterest Uses Event Time Watermarks to Finalize Petabyte Scale Data Partitions](<https://devfeed.tech/articles/pinterest-uses-event-time-watermarks-to-finalize-petabyte-scale-data-partitions-89988.md>)

Original publisher: [Read original article](<https://www.infoq.com/news/2026/10/pinterest-event-time-partition/>)

Author: Leela Kumili

Published: 2026-10-19T14:37:00Z

Content type: news

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Temporal data](<https://devfeed.tech/topics/temporal-data.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>)

Tags: [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [architecture-design](<https://devfeed.tech/tags/architecture-design.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-partitioning](<https://devfeed.tech/tags/data-partitioning.md>), [development](<https://devfeed.tech/tags/development.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [event-driven-architecture](<https://devfeed.tech/tags/event-driven-architecture.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [news](<https://devfeed.tech/tags/news.md>), [partitioning](<https://devfeed.tech/tags/partitioning.md>), [pinterest-event-time-partition](<https://devfeed.tech/tags/pinterest-event-time-partition.md>), [published](<https://devfeed.tech/tags/published.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

Pinterest's ingestion framework combines Apache Flink event-time progress with Apache Iceberg metadata to publish a monotonic partition-finalization watermark for continuously ingested change data. The signal helps downstream jobs determine when data is sufficiently complete, while percentile-based policies can trade completeness for freshness.

### Source excerpt

Pinterest has introduced event time-based partition finalization for its database ingestion framework. The mechanism uses Apache Flink to track event time and Apache Iceberg metadata to publish a non-regressing watermark, helping downstream jobs determine when continuously ingested CDC data is sufficiently complete for processing. By Leela Kumili

## Building a Rust tool with Delta Kernel to import and export Delta Lake tables

DevFeed: [Building a Rust tool with Delta Kernel to import and export Delta Lake tables](<https://devfeed.tech/articles/delta-kernel-with-rust-doing-stuff-90337.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/delta-kernel-with-rust-doing-stuff>)

Author: Daniel Beach

Published: 2026-10-19T13:17:32Z

Content type: tutorial

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Rust](<https://devfeed.tech/topics/rust.md>)

Tags: [build-tools](<https://devfeed.tech/tags/build-tools.md>), [cli](<https://devfeed.tech/tags/cli.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [rust](<https://devfeed.tech/tags/rust.md>)

### AI overview

The article introduces Delta Kernel as a reusable library that helps query engines read and write Delta Lake tables without implementing the protocol themselves. It walks through building a Rust command-line tool that exports an S3-hosted Delta table to Parquet and imports CSV data into a new Delta table, while noting that the Rust implementation is experimental and that production use may need additional support for multiple writers and Unity Catalog.

### Source excerpt

look at me go ma!

## Transactions are Generally Available, advancing warehousing on the Lakehouse

DevFeed: [Transactions are Generally Available, advancing warehousing on the Lakehouse](<https://devfeed.tech/articles/transactions-are-generally-available-advancing-warehousing-on-the-lakehouse-88969.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/transactions-are-generally-available-advancing-warehousing-lakehouse>)

Author: Benjamin Mathew; Prakhar Jain; Sumeet Varma

Published: 2026-10-16T21:00:00Z

Content type: release

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Transactions](<https://devfeed.tech/topics/transactions.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [database-consistency](<https://devfeed.tech/topics/database-consistency.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [scalable-architecture](<https://devfeed.tech/topics/scalable-architecture.md>)

Tags: [access-controls](<https://devfeed.tech/tags/access-controls.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-warehousing](<https://devfeed.tech/tags/data-warehousing.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [engineering-manager](<https://devfeed.tech/tags/engineering-manager.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [platform](<https://devfeed.tech/tags/platform.md>), [transactions](<https://devfeed.tech/tags/transactions.md>)

### AI overview

Databricks announces general availability of multi-statement transactions for Unity Catalog managed Delta Lake and Apache Iceberg tables. The feature provides atomic guarantees across multiple statements and tables, supports row-level concurrent writes, and integrates with governance controls, warehouse development tools, data pipelines, and observability. The article describes use cases such as ETL updates, stored procedures, streaming workflows, and write-audit-publish processes.

### Source excerpt

Multi-Statement Transactions are Generally Available, enabling organizations to run...

## Announcing Fivetran Managed Data Lake Service | Blog | Fivetran

DevFeed: [Announcing Fivetran Managed Data Lake Service | Blog | Fivetran](<https://devfeed.tech/articles/announcing-fivetran-managed-data-lake-service-blog-fivetran-136757.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/announcing-fivetran-managed-data-lake-service>)

Author: Taylor Brown

Published: 2026-10-13T16:41:32Z

Content type: release

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [data lake](<https://devfeed.tech/topics/data-lake.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>)

Tags: [accessible](<https://devfeed.tech/tags/accessible.md>), [acid](<https://devfeed.tech/tags/acid.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-ml](<https://devfeed.tech/tags/ai-ml.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-polaris](<https://devfeed.tech/tags/apache-polaris.md>), [apache-software-foundation](<https://devfeed.tech/tags/apache-software-foundation.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [maintenance](<https://devfeed.tech/tags/maintenance.md>), [managed](<https://devfeed.tech/tags/managed.md>), [optimized](<https://devfeed.tech/tags/optimized.md>), [query](<https://devfeed.tech/tags/query.md>), [service](<https://devfeed.tech/tags/service.md>), [sync](<https://devfeed.tech/tags/sync.md>), [table](<https://devfeed.tech/tags/table.md>)

### AI overview

Fivetran Managed Data Lake automates ingestion from over 700 pre-built or custom sources into Amazon S3, ADLS, Microsoft OneLake, or Google Cloud Storage. It normalizes, compacts, and deduplicates data into Delta Lake or Apache Iceberg tables, then maintains updates, merges, deletes, schema evolution, and performance so data stays current and query-ready. Catalog integrations include AWS Glue, Databricks Unity Catalog, and Apache Polaris; compatible tools named include Databricks, Snowflake, Starburst, and Redshift. Fivetran says it covers ingestion costs. The service may not suit workloads requiring sub-second streaming latency or organizations that do not want open table formats.

### Source excerpt

Fivetran's new offering simplifies data lake table maintenance, delivering optimized, query-ready data that is always in sync.

## Governing your lakehouse with Fivetran Managed Data Lake Service | Blog | Fivetran

DevFeed: [Governing your lakehouse with Fivetran Managed Data Lake Service | Blog | Fivetran](<https://devfeed.tech/articles/governing-your-lakehouse-with-fivetran-managed-data-lake-service-blog-fivetran-136781.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/governing-your-lakehouse-with-fivetran-managed-data-lake-service>)

Author: Andrew Madson

Published: 2026-10-13T16:41:32Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Data Management](<https://devfeed.tech/topics/data-management.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [data ownership](<https://devfeed.tech/topics/data-ownership.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Data Infrastructure](<https://devfeed.tech/topics/data-infrastructure.md>), [Security, Identity, & Compliance](<https://devfeed.tech/topics/security-identity-compliance.md>), [AWS Lake Formation](<https://devfeed.tech/topics/aws-lake-formation.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>)

Tags: [access-control](<https://devfeed.tech/tags/access-control.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-polaris](<https://devfeed.tech/tags/apache-polaris.md>), [athena](<https://devfeed.tech/tags/athena.md>), [authentication](<https://devfeed.tech/tags/authentication.md>), [automated](<https://devfeed.tech/tags/automated.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-privatelink](<https://devfeed.tech/tags/aws-privatelink.md>), [azure](<https://devfeed.tech/tags/azure.md>), [biglake](<https://devfeed.tech/tags/biglake.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [governance](<https://devfeed.tech/tags/governance.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [lifecycle](<https://devfeed.tech/tags/lifecycle.md>)

### AI overview

The excerpt presents Fivetran Managed Data Lake governance as a single-source-of-truth model: Fivetran alone writes Iceberg metadata, while query engines read it, and Fivetran republishes the correct version if AWS Glue drifts. Engines can use the Iceberg REST Catalog with OAuth2; Athena and Redshift require Glue integration enabled at destination setup, a permanent choice with regional and naming constraints. Access controls include short-lived, table-scoped S3 credentials, cross-account IAM, and optional KMS, Lake Formation, and Business Critical PrivateLink configuration. Automated retention, metadata version cleanup, biweekly orphan-file removal, compaction, deduplication, and schema evolution address lifecycle work. The excerpt warns against manual catalog changes and raw Parquet queries, and notes cloud-specific catalog requirements, including BigQuery Admin access for GCS BigLake.

### Source excerpt

Lakehouse governance requires observability, access control, data lifecycle management, and more. Fivetran MDLS makes it easy.

## From Anomaly Detection to Agentic AI: Engineering Intelligent Data Reliability on Databricks

DevFeed: [From Anomaly Detection to Agentic AI: Engineering Intelligent Data Reliability on Databricks](<https://devfeed.tech/articles/from-anomaly-detection-to-agentic-ai-engineering-intelligent-data-reliability-on-databricks-86078.md>)

Original publisher: [Read original article](<https://hackernoon.com/from-anomaly-detection-to-agentic-ai-engineering-intelligent-data-reliability-on-databricks?source=rss>)

Author: Sanya Kapoor

Published: 2026-10-13T10:49:51Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>)

Tags: [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [agentic-ai-for-data-engineers](<https://devfeed.tech/tags/agentic-ai-for-data-engineers.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [anomaly-detection](<https://devfeed.tech/tags/anomaly-detection.md>), [data](<https://devfeed.tech/tags/data.md>), [data-reliability](<https://devfeed.tech/tags/data-reliability.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [databricks-data-reliability](<https://devfeed.tech/tags/databricks-data-reliability.md>), [delta-lake-architecture](<https://devfeed.tech/tags/delta-lake-architecture.md>), [good-company](<https://devfeed.tech/tags/good-company.md>), [governed-enterprise-ai](<https://devfeed.tech/tags/governed-enterprise-ai.md>), [intelligent-data-operations](<https://devfeed.tech/tags/intelligent-data-operations.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [lokeshkumar-madabathula](<https://devfeed.tech/tags/lokeshkumar-madabathula.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [real-time-anomaly-detection](<https://devfeed.tech/tags/real-time-anomaly-detection.md>)

### AI overview

The article describes a proposed Databricks architecture for improving data reliability by combining streaming ingestion, Delta Lake, anomaly detection and a governed agentic AI layer. After an anomaly is detected, the AI layer can gather operational context, assist with root-cause investigation and downstream impact assessment, and recommend remediation. Production actions remain subject to permissions and human oversight.

### Source excerpt

Explore how Lokeshkumar Madabathula combines Databricks, real-time anomaly detection, lakehouse architecture, and governed Agentic AI for data reliability.

## Announcing Fivetran Managed Data Lake Service | Blog | Fivetran

DevFeed: [Announcing Fivetran Managed Data Lake Service | Blog | Fivetran](<https://devfeed.tech/articles/announcing-fivetran-managed-data-lake-service-blog-fivetran-85458.md>)

Original publisher: [Read original article](<https://www.fivetran.com/blog/announcing-fivetran-managed-data-lake-service>)

Author: Taylor Brown

Published: 2026-10-12T19:05:09Z

Content type: release

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [data lake](<https://devfeed.tech/topics/data-lake.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [data-governance](<https://devfeed.tech/tags/data-governance.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>)

### AI overview

Fivetran introduces a managed data lake service that ingests data from more than 700 sources, standardizes it into Delta Lake or Apache Iceberg formats, and automates ongoing maintenance and governance. The service works with major cloud storage destinations and data catalogs; the article notes it may not suit organizations needing sub-second streaming latency.

### Source excerpt

Fivetran's new offering simplifies data lake table maintenance, delivering optimized, query-ready data that is always in sync.

## Article: When Should You Use Liquid Clustering? Benchmarking Lakehouse Layout Strategies

DevFeed: [Article: When Should You Use Liquid Clustering? Benchmarking Lakehouse Layout Strategies](<https://devfeed.tech/articles/article-when-should-you-use-liquid-clustering-benchmarking-lakehouse-layout-strategies-85212.md>)

Original publisher: [Read original article](<https://www.infoq.com/articles/liquid-clustering-lakehouse-layout/>)

Author: Kushal Kumar Vishwakarma

Published: 2026-10-12T11:00:00Z

Content type: article

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [deltalake](<https://devfeed.tech/topics/deltalake.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [clustering](<https://devfeed.tech/topics/clustering.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-ml-data-engineering](<https://devfeed.tech/tags/ai-ml-data-engineering.md>), [application-performance-management](<https://devfeed.tech/tags/application-performance-management.md>), [architecture-design](<https://devfeed.tech/tags/architecture-design.md>), [article](<https://devfeed.tech/tags/article.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [clustering-caching](<https://devfeed.tech/tags/clustering-caching.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-storage](<https://devfeed.tech/tags/data-storage.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [liquid-clustering-lakehouse-layout](<https://devfeed.tech/tags/liquid-clustering-lakehouse-layout.md>), [ml-data-engineering](<https://devfeed.tech/tags/ml-data-engineering.md>), [partitioning](<https://devfeed.tech/tags/partitioning.md>), [performance](<https://devfeed.tech/tags/performance.md>), [performance-evaluation](<https://devfeed.tech/tags/performance-evaluation.md>), [query](<https://devfeed.tech/tags/query.md>)

### AI overview

The article benchmarks Liquid Clustering, ZOrdering, and partitioning on Delta Lake across four analytical workloads. Liquid Clustering has the lowest latency for point lookups, time-range aggregations, and large aggregations, while ZOrder performs best for selective multicolumn queries whose filters match its clustering keys. The author recommends validating layout choices against representative query patterns and notes that the benchmark uses synthetic data, so results may vary in production.

### Source excerpt

Physical data layout is one of the most consequential decisions in lakehouse engineering. The author compares Liquid Clustering, ZOrdering, and partitioning across four workload types on Delta Lake, uncovering two undocumented failure modes that likely affect production systems today and offering a decision framework for choosing the right layout strategy. By Kushal Kumar Vishwakarma

## The lakehouse is a better data warehouse: 2026 benchmarks and proof

DevFeed: [The lakehouse is a better data warehouse: 2026 benchmarks and proof](<https://devfeed.tech/articles/the-lakehouse-is-a-better-data-warehouse-2026-benchmarks-and-proof-65639.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/lakehouse-better-data-warehouse-2026-benchmarks-and-proof>)

Author: Joanna Zouhour

Published: 2026-10-06T16:35:12Z

Content type: comparison

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [data observability](<https://devfeed.tech/topics/data-observability.md>), [clean energy data centers](<https://devfeed.tech/topics/clean-energy-data-centers.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [data](<https://devfeed.tech/tags/data.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>)

### AI overview

The article argues that lakehouse platforms can now match or exceed data warehouses for many SQL workloads while also serving BI, streaming, machine learning, and AI workloads from shared governed data. It cites Databricks performance benchmarks and customer migration results, describes the lakehouse's support for open table formats and multiple data types, and acknowledges that warehouses retain advantages for some workloads.

### Source excerpt

The warehouse was the right answer for 20 yearsThe data warehouse earned its place. For two decades...

## Your data, your storage, your rules: a 2026 guide to storing Unity Catalog managed tables

DevFeed: [Your data, your storage, your rules: a 2026 guide to storing Unity Catalog managed tables](<https://devfeed.tech/articles/your-data-your-storage-your-rules-a-2026-guide-to-storing-unity-catalog-managed-tables-61920.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/your-data-your-storage-your-rules-2026-guide-storing-unity-catalog-managed-tables>)

Author: Elizabeth Bowman; Isabel Dong

Published: 2026-09-29T16:40:16Z

Content type: tutorial

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Databricks Unity Catalog integration](<https://devfeed.tech/topics/databricks-unity-catalog-integration.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [cloud-storage](<https://devfeed.tech/tags/cloud-storage.md>), [data-platforms](<https://devfeed.tech/tags/data-platforms.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [s3](<https://devfeed.tech/tags/s3.md>), [storage](<https://devfeed.tech/tags/storage.md>), [unity-catalog](<https://devfeed.tech/tags/unity-catalog.md>)

### AI overview

The guide explains how Databricks Unity Catalog managed tables store data in a cloud account chosen by the organization, while Unity Catalog automates table management. It describes setting storage locations at the metastore, catalog, or schema level, and how external tools can access governed data through open APIs and the Iceberg REST Catalog.

### Source excerpt

Unity catalog managed tables allow you to control the placement of your data when...

[Next page](<https://devfeed.tech/topics/deltalake.md?cursor=WyIyMDI2LTA5LTI5VDE2OjQwOjE2KzAwOjAwIiwgIjFhZmUzMTM5LTI0NDktNDZiOC05MTMzLTIzZTA3OWNjZjNlMyJd>)