# Apache Iceberg

Open table format for large analytic datasets that enables multiple compute engines to safely use the same tables.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## QCon San Francisco 2026: The Future of Apache Iceberg

DevFeed: [QCon San Francisco 2026: The Future of Apache Iceberg](<https://devfeed.tech/articles/qcon-san-francisco-2026-the-future-of-apache-iceberg-145863.md>)

Original publisher: [Read original article](<https://www.infoq.com/news/2026/11/iceberg-v4-metadata/>)

Author: Renato Losio

Published: 2026-11-20T09:17:00Z

Content type: news

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-ml-data-engineering](<https://devfeed.tech/tags/ai-ml-data-engineering.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [architecture-design](<https://devfeed.tech/tags/architecture-design.md>), [constraint](<https://devfeed.tech/tags/constraint.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analytics](<https://devfeed.tech/tags/data-analytics.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-management](<https://devfeed.tech/tags/data-management.md>), [data-types](<https://devfeed.tech/tags/data-types.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [iceberg-v4-metadata](<https://devfeed.tech/tags/iceberg-v4-metadata.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [ml-data-engineering](<https://devfeed.tech/tags/ml-data-engineering.md>), [news](<https://devfeed.tech/tags/news.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [qcon-san-francisco-2026](<https://devfeed.tech/tags/qcon-san-francisco-2026.md>), [redesign](<https://devfeed.tech/tags/redesign.md>), [san-francisco](<https://devfeed.tech/tags/san-francisco.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [schema-evolution](<https://devfeed.tech/tags/schema-evolution.md>), [session](<https://devfeed.tech/tags/session.md>), [specification](<https://devfeed.tech/tags/specification.md>), [table-format](<https://devfeed.tech/tags/table-format.md>)

### AI overview

Apache Iceberg V4 proposes a major metadata redesign to help very large tables scale and enable more frequent writes. Its Adaptive Metadata Tree can point directly to data files, reducing the described commit path from at least four I/O operations to a single metadata write; the speaker said this could support second-level or subsecond commits, compared with the current recommendation of at least two minutes. Other proposed changes include typed column statistics, relative paths, column-level updates, expression-based defaults, constraints, and file and vector data types. The excerpt says V4 metadata will not be backward compatible with older readers, although existing V3 metadata can remain in place during an upgrade.

### Source excerpt

Apache Iceberg V4 proposes a redesign of table metadata to improve scalability and support more frequent writes, alongside new capabilities for data management and schema evolution. Daniel Weeks, co-creator of Apache Iceberg and member of technical staff at Databricks, presented a session at QCon San Francisco on the fourth version of the Apache Iceberg open table format specification. By Renato Losio

## Apache Iceberg Integrates FILE Type for Images and Audio Into Next Specification: Gap Remains Between Google's Announcement and Implementation Status

DevFeed: [Apache Iceberg Integrates FILE Type for Images and Audio Into Next Specification: Gap Remains Between Google's Announcement and Implementation Status](<https://devfeed.tech/articles/apache-iceberg-integrates-file-type-for-images-and-audio-into-next-specification-gap-remains-between-google-s-announcement-and-implementation-status-144635.md>)

Original publisher: [Read original article](<https://joho-todai.com/en/apache-iceberg-file-type-image-audio-integration/>)

Author: Toru Misaki

Published: 2026-11-19T23:53:13Z

Content type: article

Language: en

Sources: [Joho no Todai](<https://devfeed.tech/sources/joho-no-todai.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [implementation](<https://devfeed.tech/topics/implementation.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [Image](<https://devfeed.tech/topics/image.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [audio](<https://devfeed.tech/tags/audio.md>), [file](<https://devfeed.tech/tags/file.md>), [google](<https://devfeed.tech/tags/google.md>), [images](<https://devfeed.tech/tags/images.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [specification](<https://devfeed.tech/tags/specification.md>)

### AI overview

Apache Iceberg's FILE type standardizes references to images, audio, and documents as table values, with fields for uri, byte range, MIME type, checksum, and optional inline data. Subfield statistics may help engines filter data by content type. This could reduce reliance on engine-specific file handling and auxiliary metadata, but it does not guarantee that referenced files exist or manage their ownership, access, deletion, or relocation; readers must compare checksums to detect changes. The specification text was merged into Iceberg's main branch, but v4 remains under development and has not been formally adopted. An initial Java API implementation is merged; serialization, Avro/Parquet/ORC support, Spark and Flink support, and other work remain. The excerpt therefore describes a developing standard, not established cross-engine interoperability.

### Source excerpt

The Apache Iceberg FILE type, introduced by Google on November 19, describes images, audio, and documents as typed table data. The specification has been merged into the main branch, but v4 has not yet been formally adopted and processing-engine implementation remains at an early stage.

## Bringing the lake back to the lakehouse: Introducing the FILE type in Apache Iceberg

DevFeed: [Bringing the lake back to the lakehouse: Introducing the FILE type in Apache Iceberg](<https://devfeed.tech/articles/bringing-the-lake-back-to-the-lakehouse-introducing-the-file-type-in-apache-iceberg-146546.md>)

Original publisher: [Read original article](<http://opensource.googleblog.com/2026/11/iceberg-file-type-multimodal-lakehouse.html>)

Author: Google Open Source (noreply@blogger.com)

Published: 2026-11-19T19:30:00Z

Content type: article

Language: en

Sources: [Google Open Source Blog](<https://devfeed.tech/sources/google-open-source-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [collective](<https://devfeed.tech/tags/collective.md>), [columnar](<https://devfeed.tech/tags/columnar.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-type](<https://devfeed.tech/tags/data-type.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [external-storage](<https://devfeed.tech/tags/external-storage.md>), [file](<https://devfeed.tech/tags/file.md>), [files](<https://devfeed.tech/tags/files.md>), [formats](<https://devfeed.tech/tags/formats.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [isolation](<https://devfeed.tech/tags/isolation.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [organization](<https://devfeed.tech/tags/organization.md>), [type](<https://devfeed.tech/tags/type.md>)

### AI overview

Apache Iceberg's new `FILE` type models external assets as structured values containing fields such as `uri`, `offset`, `size`, `content_type`, `checksum`, and optional inline bytes, addressing limitations of bare string paths. The excerpt says this lets supporting engines reuse stored metadata instead of fetching it per file, use MIME types rather than guessing from extensions, detect possible changes through checksums, and read specified byte ranges. It also places structured records and media references in the same transactional table. The specification is described as finalized and targeted for Iceberg v4; native reader and writer support across engines including Spark, Trino, and DuckDB is presented as future work. The excerpt does not establish current implementation availability.

### Source excerpt

by Sung Yun & Gaurav Soni, Google Cloud The original promise of the data lake was simple: a single, universal home for all an organization's data. Whether structured database exports, raw text documents, audio recordings, or high-resolution imagery, everything landed in object storage. But as data lakes grew into sprawling swamps lacking transactional guarantees and consistency, the industry pioneered the lakehouse. Open table formats like Apache Iceberg brought order to the chaos, introducing data warehouse rigor--ACID transactions, snapshot isolation, partition evolution, and fast columnar pruning--directly to cloud storage. Yet, in our collective rush to build the "house", we quietly left the "lake" behind. Photo by mydaydream89 (CC0) via Wikimedia Commons With the rapid rise of multimodal Large Language Models (LLMs), interest in processing unstructured data directly within the lakehouse has grown rapidly. Multimodal LLMs enable modern data warehouses and lakehouses to analyze unstructured inputs--such as raw images, audio, and documents--much like traditional structured data, without requiring complex, bespoke machine learning staging pipelines. Because table formats were designed around structured rows and columns, unstructured media was left behind. Raw files were relegated back to external storage buckets, loosely connected to analytical tables through plain, unvalidated string URIs. The Apache Iceberg community recently took a major step toward reuniting these two worlds by merging support for the new FILE data type into the Iceberg specification. Limitations of using bare string URIs Historically, platforms referenced external unstructured assets by storing simple string paths inside tabular columns. While straightforward, relying on plain string URIs introduces several operational challenges: No file metadata in the table: A string path doesn't say what type a file is or how big it is. Engines guess types from file extensions, or make a separate call to objec

## Hitachi Vantara bets smarter storage can close AI's pilot-to-production gap

DevFeed: [Hitachi Vantara bets smarter storage can close AI's pilot-to-production gap](<https://devfeed.tech/articles/hitachi-vantara-bets-smarter-storage-can-close-ai-s-pilot-to-production-gap-144137.md>)

Original publisher: [Read original article](<https://siliconangle.com/2026/11/19/apache-iceberg-intelligent-storage-help-ai-reach-production-sc26/>)

Author: Kelly Knight

Published: 2026-11-19T18:39:55Z

Content type: news

Language: en

Sources: [SiliconANGLE](<https://devfeed.tech/sources/siliconangle.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [big-data](<https://devfeed.tech/topics/big-data.md>)

Tags: [accelerated-computing](<https://devfeed.tech/tags/accelerated-computing.md>), [access-control](<https://devfeed.tech/tags/access-control.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-data-centers](<https://devfeed.tech/tags/ai-data-centers.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [ai-supercomputers](<https://devfeed.tech/tags/ai-supercomputers.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [arun-prasad-raman](<https://devfeed.tech/tags/arun-prasad-raman.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [compute-express-link](<https://devfeed.tech/tags/compute-express-link.md>), [cube-event-coverage](<https://devfeed.tech/tags/cube-event-coverage.md>), [data](<https://devfeed.tech/tags/data.md>), [data-governance](<https://devfeed.tech/tags/data-governance.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [data-scientists](<https://devfeed.tech/tags/data-scientists.md>), [database](<https://devfeed.tech/tags/database.md>), [energy](<https://devfeed.tech/tags/energy.md>), [energy-efficiency](<https://devfeed.tech/tags/energy-efficiency.md>), [etl](<https://devfeed.tech/tags/etl.md>), [gap](<https://devfeed.tech/tags/gap.md>), [gemma-allen](<https://devfeed.tech/tags/gemma-allen.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [high-performance-computing](<https://devfeed.tech/tags/high-performance-computing.md>), [high-performance-storage](<https://devfeed.tech/tags/high-performance-storage.md>), [high-speed-networking](<https://devfeed.tech/tags/high-speed-networking.md>), [hitachi-iq](<https://devfeed.tech/tags/hitachi-iq.md>), [hitachi-vantara](<https://devfeed.tech/tags/hitachi-vantara.md>), [hpc](<https://devfeed.tech/tags/hpc.md>), [hpc-infrastructure](<https://devfeed.tech/tags/hpc-infrastructure.md>), [hpc-unites](<https://devfeed.tech/tags/hpc-unites.md>), [jeff-lundberg](<https://devfeed.tech/tags/jeff-lundberg.md>), [john-furrier](<https://devfeed.tech/tags/john-furrier.md>), [memory-pooling](<https://devfeed.tech/tags/memory-pooling.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [parallel-file-systems](<https://devfeed.tech/tags/parallel-file-systems.md>), [pilot](<https://devfeed.tech/tags/pilot.md>), [production](<https://devfeed.tech/tags/production.md>), [sc26](<https://devfeed.tech/tags/sc26.md>), [siliconangle](<https://devfeed.tech/tags/siliconangle.md>), [storage](<https://devfeed.tech/tags/storage.md>), [thecube](<https://devfeed.tech/tags/thecube.md>)

### AI overview

The excerpt identifies fragmented, ungoverned data and access controls as obstacles to moving AI projects into production, and describes Hitachi Vantara's approach of adding classification and governance directly to storage. Its VSP One object store can flag files containing personal information so organizations can exclude them from models or restrict cross-border replication. For analytics, the company describes storing CT scan metadata in Apache Iceberg and querying it directly--for example, to find abdominal scans from the past year--without a separate ETL process or database. These are company representatives' examples; the excerpt does not provide independent performance results.

### Source excerpt

Enterprises pushing AI from pilot to production are learning that data, not GPUs, is the hardest part, and open table formats such as Apache Iceberg are part of the answer. Demand for AI infrastructure shows few signs of slowing, but much of the information companies need is still scattered, ungoverned and out of reach. That [...] The post Hitachi Vantara bets smarter storage can close AI's pilot-to-production gap appeared first on SiliconANGLE.

## Amazon S3 Tables with dbt: Iceberg-based data transformations using AWS Glue

DevFeed: [Amazon S3 Tables with dbt: Iceberg-based data transformations using AWS Glue](<https://devfeed.tech/articles/amazon-s3-tables-with-dbt-iceberg-based-data-transformations-using-aws-glue-144273.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/amazon-s3-tables-with-dbt-iceberg-based-data-transformations-using-aws-glue/>)

Author: Yukiko Iwazumi

Published: 2026-11-19T17:58:50Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-quick-sight](<https://devfeed.tech/tags/amazon-quick-sight.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The walkthrough builds a dbt pipeline on Amazon S3 Tables through AWS Glue interactive sessions, with raw CSV data staged and transformed into analytics marts. It requires dbt-glue 1.12.4 or later for S3 Tables incremental models, Python 3.10-3.12, and AWS Glue 6.0 or later. In the example, `fact_orders` uses merge to retain each order's latest state, while `fact_order_history` uses append to retain changes. After source updates, the merge table has six orders, including the updated order 1004 and new order 1006; the history table grows from five to seven rows. The post describes S3 Tables support as experimental in dbt-glue 1.12.4 and recommends testing it in your environment.

### Source excerpt

Amazon S3 Tables, a capability of Amazon S3, bring built-in Apache Iceberg support to object storage. This post shows how to configure and run dbt transformations on S3 Tables with the AWS Glue dbt-glue adapter, using incremental merge and append strategies across a staging and mart architecture.

## Aderant modernizes analytics data pipeline with Apache Iceberg and AWS Lambda

DevFeed: [Aderant modernizes analytics data pipeline with Apache Iceberg and AWS Lambda](<https://devfeed.tech/articles/aderant-modernizes-analytics-data-pipeline-with-apache-iceberg-and-aws-lambda-141916.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/aderant-modernizes-analytics-data-pipeline-with-apache-iceberg-and-aws-lambda/>)

Author: Sham Birbahadur

Published: 2026-11-19T00:36:23Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [event-driven data processing](<https://devfeed.tech/topics/event-driven-data-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [AWS Database Migration Service](<https://devfeed.tech/topics/aws-database-migration-service.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [acid](<https://devfeed.tech/tags/acid.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [batch](<https://devfeed.tech/tags/batch.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [cost](<https://devfeed.tech/tags/cost.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [faster](<https://devfeed.tech/tags/faster.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [legal](<https://devfeed.tech/tags/legal.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [queries](<https://devfeed.tech/tags/queries.md>)

### AI overview

Aderant replaced Stridyn's 15-minute batch pipeline with an event-driven medallion architecture that processes CDC changes from 700 client environments in single-digit seconds to bronze-layer Apache Iceberg tables. AWS DMS delivers Parquet files to S3; S3 notifications and SQS invoke Lambda for ingestion, while scheduled AWS Glue jobs transform data through silver and gold layers. Incremental Glue jobs handle 77 standard gold tables; 34 complex tables use Iceberg materialized views and snapshot comparisons to send changed rows to native Amazon Redshift tables. Aderant reports 10-26x faster queries on native Redshift tables, under-100-millisecond dashboard queries, and 77 percent lower pipeline compute costs versus streaming alternatives. The account says this approach supports schema changes and concurrent tenant workloads; it also describes a possible future Kinesis path for high-volume tenants.

### Source excerpt

Learn how Aderant modernized the data pipeline behind its Stridyn legal analytics platform with Apache Iceberg, AWS Lambda, and AWS Glue, moving from 15-minute batch cycles to an event-driven medallion architecture that processes change data capture events in single-digit seconds for 700 tenants, with 10-26x faster queries and 77 percent lower cost.

## Long-term Amazon Redshift observability with Datadog and Amazon S3 Tables

DevFeed: [Long-term Amazon Redshift observability with Datadog and Amazon S3 Tables](<https://devfeed.tech/articles/long-term-amazon-redshift-observability-with-datadog-and-amazon-s3-tables-141880.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/long-term-amazon-redshift-observability-with-datadog-and-amazon-s3-tables/>)

Author: Raghu Kuppala

Published: 2026-11-18T22:57:27Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [datadog](<https://devfeed.tech/topics/datadog.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [elt](<https://devfeed.tech/topics/elt.md>)

Tags: [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-emr](<https://devfeed.tech/tags/amazon-emr.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [aws-cli](<https://devfeed.tech/tags/aws-cli.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-identity-and-access-management-iam](<https://devfeed.tech/tags/aws-identity-and-access-management-iam.md>), [aws-key-management-service-aws-kms](<https://devfeed.tech/tags/aws-key-management-service-aws-kms.md>), [data-observability](<https://devfeed.tech/tags/data-observability.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [history](<https://devfeed.tech/tags/history.md>), [integration](<https://devfeed.tech/tags/integration.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [observability](<https://devfeed.tech/tags/observability.md>), [performance](<https://devfeed.tech/tags/performance.md>), [query](<https://devfeed.tech/tags/query.md>)

### AI overview

Amazon Redshift system tables retain history for only 7 days. The S3 Tables integration continuously exports selected system table data as Apache Iceberg tables, letting customers set retention and query the history without collection queries consuming warehouse compute. Datadog Data Observability can read that history to build column-level lineage, normalized query profiles, and context for quality alerts across connected clusters and workgroups. It still connects to Redshift for metadata and runs configured aggregate quality monitors there. Setup requires enabling per-warehouse exports, integrating S3 Tables with the AWS Glue Data Catalog, and granting the Datadog IAM role read access; the feature is described as a preview requiring account-team access. Disabling exports creates a history gap that cannot be backfilled after Redshift's 7-day window, and customer-managed KMS encryption keys cannot be changed after table creation.

### Source excerpt

Amazon Redshift keeps only 7 days of system table history. This post pairs Datadog Data Observability with the new Amazon Redshift system table integration with Amazon S3 Tables, so you can retain query history for the long term and analyze lineage, query performance, and data quality without adding load to your warehouses.

## How United Airlines streams database changes to Apache Iceberg with Amazon Data Firehose

DevFeed: [How United Airlines streams database changes to Apache Iceberg with Amazon Data Firehose](<https://devfeed.tech/articles/how-united-airlines-streams-database-changes-to-apache-iceberg-with-amazon-data-firehose-141918.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/how-united-airlines-streams-database-changes-to-apache-iceberg-with-amazon-data-firehose/>)

Author: Guillaume Dupont

Published: 2026-11-18T20:37:39Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Amazon Aurora](<https://devfeed.tech/topics/amazon-aurora.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [streaming data to S3 with Parquet](<https://devfeed.tech/topics/streaming-data-to-s3-with-parquet.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [Visual Studio Code](<https://devfeed.tech/topics/visual-studio-code.md>)

Tags: [airlines](<https://devfeed.tech/tags/airlines.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-aurora](<https://devfeed.tech/tags/amazon-aurora.md>), [amazon-data-firehose](<https://devfeed.tech/tags/amazon-data-firehose.md>), [amazon-kinesis](<https://devfeed.tech/tags/amazon-kinesis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cloudformation](<https://devfeed.tech/tags/aws-cloudformation.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [dms](<https://devfeed.tech/tags/dms.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [kinesis-data-streams](<https://devfeed.tech/tags/kinesis-data-streams.md>), [streams](<https://devfeed.tech/tags/streams.md>), [united-airlines](<https://devfeed.tech/tags/united-airlines.md>)

### AI overview

United Airlines describes a near-real-time CDC pipeline that moves Aurora MySQL changes through AWS DMS, Kinesis Data Streams, Lambda, and Amazon Data Firehose into Apache Iceberg tables on S3. Firehose applies inserts, updates, and deletes as Iceberg snapshots; the open tables can be queried through AWS Glue, Redshift, or Athena. United reports 5x lower average query latency and 2x shorter downstream Glue job processing time in its testing, with a low-volume test reaching about 10 seconds end-to-end. Buffer settings trade latency against cost and file size. The walkthrough uses DMS Serverless, Athena for validation, and a simplified single-table CloudFormation template; United's production setup uses provisioned DMS and primarily Redshift Serverless. The template's shard count and other demo settings are not production-tuned, and schema evolution and multi-table replication require additional handling.

### Source excerpt

United Airlines built a near-real-time change data capture pipeline that replicates reference data from Amazon Aurora MySQL to open Apache Iceberg tables using AWS DMS, Amazon Kinesis Data Streams, and Amazon Data Firehose. This post shows how to reproduce the serverless, multi-engine analytics pattern with AWS CloudFormation.

## The Agent Can Guess. The Query Engine Can't.

DevFeed: [The Agent Can Guess. The Query Engine Can't.](<https://devfeed.tech/articles/the-agent-can-guess-the-query-engine-can-t-141975.md>)

Original publisher: [Read original article](<https://www.starburst.io/blog/gpu-acceleration-gpu-native-engine-torin-the-agent-can-guess-the-query-engine-cant>)

Published: 2026-11-18T16:05:22Z

Content type: article

Language: en

Sources: [Starburst Data](<https://devfeed.tech/sources/starburst-data.md>)

Topics: [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Query (disambiguation)](<https://devfeed.tech/topics/query.md>), [AI response variability](<https://devfeed.tech/topics/ai-response-variability.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [added](<https://devfeed.tech/tags/added.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [announcement](<https://devfeed.tech/tags/announcement.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [application](<https://devfeed.tech/tags/application.md>), [can](<https://devfeed.tech/tags/can.md>), [engine](<https://devfeed.tech/tags/engine.md>), [query](<https://devfeed.tech/tags/query.md>), [query-engine](<https://devfeed.tech/tags/query-engine.md>), [question](<https://devfeed.tech/tags/question.md>)

### AI overview

Torin is Starburst's GPU-native SQL query engine, designed to accelerate analytical work while returning the same rows as CPU execution. Trino continues to parse, optimize, and schedule queries; Torin runs supported operators on NVIDIA GPUs, while unsupported work stays on the CPU in the same query. It preserves CPU-compatible behavior by analyzing expressions for failure risks--for example, avoiding division by zero when a filter condition guards a division--and choosing safe evaluation strategies. `EXPLAIN ANALYZE` shows where query work ran. Torin is best suited to large scans, joins, aggregations, AI data preparation, and text or pattern filters over Parquet tables in Apache Iceberg, Hive, or Delta Lake. It requires a supported NVIDIA GPU with at least 24 GB memory; 80 GB or more is recommended. `DISTINCT` in `SELECT`, user-defined functions, complex types, non-Parquet scans, and inter-worker exchanges remain on CPU. Performance gains vary from none to an order of magnitude; small or lightweight queries may not keep a GPU busy. Torin is in limited general availability for Starburst Enterprise.

### Source excerpt

Ask an AI agent the same question twice and you may get two different answers. ...

## Testing Apache Iceberg Across Dremio, Spark, Flink, and Trino

DevFeed: [Testing Apache Iceberg Across Dremio, Spark, Flink, and Trino](<https://devfeed.tech/articles/testing-apache-iceberg-across-dremio-spark-flink-and-trino-138582.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/testing-apache-iceberg-across-dremio-spark-flink-and-trino/>)

Author: Alex Merced

Published: 2026-11-17T15:23:57Z

Content type: tutorial

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [compatibility](<https://devfeed.tech/tags/compatibility.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [flink](<https://devfeed.tech/tags/flink.md>), [spark](<https://devfeed.tech/tags/spark.md>), [testing](<https://devfeed.tech/tags/testing.md>), [trino](<https://devfeed.tech/tags/trino.md>)

### AI overview

A support matrix alone cannot establish that Dremio, Spark, Flink, and Trino interoperate on a specific Iceberg setup; test the exact engine, connector, catalog, storage, and format versions together. The proposed contract checks deterministic appends and snapshot reads with row counts and checksums, then validates schema and partition evolution, deletes, retries, commit conflicts, and injected failures. Capture snapshot and manifest metadata, query results, and catalog, engine, and storage logs so mismatches can be diagnosed by logical content rather than byte-for-byte manifest equality. Separate reader and writer certification, and gate experimental v3 features independently. A passing small CI suite is only evidence for its tested path; pre-release acceptance should cover realistic data, concurrency, and failures. The excerpt gives illustrative checks and troubleshooting scenarios, but says command syntax and engine-specific behavior must be verified in the relevant documentation.

### Source excerpt

Build an Apache Iceberg compatibility contract for Dremio, Spark, Flink, and Trino covering commits, DML, evolution, deletes, time travel, and recovery. The post Testing Apache Iceberg Across Dremio, Spark, Flink, and Trino appeared first on Dremio.

## Apache Iceberg Identifier Fields: Keys, CDC, Deletes, and Upserts

DevFeed: [Apache Iceberg Identifier Fields: Keys, CDC, Deletes, and Upserts](<https://devfeed.tech/articles/apache-iceberg-identifier-fields-keys-cdc-deletes-and-upserts-138578.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/apache-iceberg-identifier-fields-keys-cdc-deletes-and-upserts/>)

Author: Alex Merced

Published: 2026-11-17T15:23:36Z

Content type: article

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [identifier](<https://devfeed.tech/tags/identifier.md>), [keys](<https://devfeed.tech/tags/keys.md>)

### AI overview

Apache Iceberg identifier fields name primitive columns that identify a logical row, but they are table metadata, not an enforced primary key: uniqueness, null handling, event ordering, and deduplication remain the responsibility of writers and pipelines. Equality-delete files specify their own matching columns, which may differ from the identifier fields, so inspect them to confirm delete behavior. The excerpt describes Spark DDL for setting identifier fields and Flink upsert routing, whose exact behavior depends on connector versions. For CDC, it recommends testing duplicate and late events, checking delete files and compaction results, and preserving ordering through timestamps or sequence numbers. Before migrating to Iceberg v3 deletion vectors, verify engine and connector support and validate conversion in staging; existing files are not rewritten simply by changing identifier metadata.

### Source excerpt

Learn how Apache Iceberg identifier fields describe row identity, how engines use them for upserts and equality deletes, and what they do not guarantee. The post Apache Iceberg Identifier Fields: Keys, CDC, Deletes, and Upserts appeared first on Dremio.

## I benchmarked Snowflake Postgres mirroring for two days. Here are the numbers.

DevFeed: [I benchmarked Snowflake Postgres mirroring for two days. Here are the numbers.](<https://devfeed.tech/articles/i-benchmarked-snowflake-postgres-mirroring-for-two-days-here-are-the-numbers-136382.md>)

Original publisher: [Read original article](<https://medium.com/snowflake/i-benchmarked-snowflake-postgres-mirroring-for-two-days-here-are-the-numbers-b50eba22b8de?source=rss----34b6daafc07---4>)

Author: Ondřej Blaha

Published: 2026-11-17T01:00:34Z

Content type: article

Language: en

Sources: [Snowflake Builders](<https://devfeed.tech/sources/snowflake-builders.md>)

Topics: [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [elt](<https://devfeed.tech/topics/elt.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [data type](<https://devfeed.tech/topics/data-type.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [arrays](<https://devfeed.tech/tags/arrays.md>), [batch](<https://devfeed.tech/tags/batch.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [gotchas](<https://devfeed.tech/tags/gotchas.md>), [impact](<https://devfeed.tech/tags/impact.md>), [latency](<https://devfeed.tech/tags/latency.md>), [load](<https://devfeed.tech/tags/load.md>), [mirroring](<https://devfeed.tech/tags/mirroring.md>), [numbers](<https://devfeed.tech/tags/numbers.md>), [oltp](<https://devfeed.tech/tags/oltp.md>), [operations](<https://devfeed.tech/tags/operations.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [postgresql-replication](<https://devfeed.tech/tags/postgresql-replication.md>), [refresh](<https://devfeed.tech/tags/refresh.md>), [replication](<https://devfeed.tech/tags/replication.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [source](<https://devfeed.tech/tags/source.md>), [two-days](<https://devfeed.tech/tags/two-days.md>)

### AI overview

On a Snowflake Postgres STANDARD_M instance in one AWS region, mirrored rows reached the near-real-time `TABLE$live` view in about 26 seconds at the median and 37 seconds at p90 across refresh intervals; the regular table could lag by up to one interval. Under the tested write load, moving from a 30-second to a 2-minute interval cut apply-task cost 73%, while small OLTP transactions showed little source impact and batch inserts into mirrored tables were 20-30% slower. The author recommends starting at 2 minutes, using `$live` for operational freshness, adding primary keys before mirroring, and committing bulk loads in chunks of up to 100,000 rows. Important limits include lossy mappings for some PostgreSQL types, `$changes` history being cleared by column changes, and `RESTART_MIRROR` triggering a full table copy. Results are from one instance size and region with limited repetitions, and are not an SLA.

### Source excerpt

Latency distributions, cost per refresh interval, impact on the source database, bulk loads and operations. Measured on the smallest instance that supports mirroring. Snowflake Postgres, managed PostgreSQL running inside a Snowflake account, has been generally available since 24 February 2026, and data mirroring into Snowflake followed on 15 October 2026. I didn't just want to see that it works. I wanted real numbers under a simulated load, so that I can tell clients how it actually behaves and compare the cost with what we had expected. The marketing says "seconds" and "sub-minute", and nobody had published a latency distribution, a cost per interval or the impact on the source database, so I spent two days measuring them. TL;DR Freshness: new rows show up in TABLE$live after ~26 s (median), ~37 s (p90), no matter whether the refresh interval is 30 seconds or 10 minutes. The table itself lags behind $live by up to one refresh interval. Cost: the serverless apply task cost 0.063 credits/h idle at a 30-second interval and 0.25 credits/h under a steady write load. Switching to 2 minutes cut the loaded cost by 73 %, to 10 minutes by 88 %. Source impact: small OLTP transactions barely notice the mirror. Batch inserts into a mirrored table were 20-30 % slower than into a non-mirrored table on the same instance. Operations: SUSPEND_MIRROR took 2.5-4.7 s under load, RESTART_MIRROR re-copies every table despite saying "soft", adding an 8M-row table to a running mirror took under a minute, and the replication slot survived every suspend/resume of the instance. Gotchas: numeric without precision becomes FLOAT, json/jsonb become TEXT, 2-D arrays become NULL, PostgreSQL refuses ADD COLUMN ... DEFAULT <volatile> on mirrored tables, any ADD, DROP or RENAME COLUMN wipes that table's $changes history, and tables without a primary key turn insert-only. Why this matters Every team that runs an application on Postgres and analytics in Snowflake owns a pipeline between the two. It is usu

## Analyze Anthropic usage, spend, and access with AWS Glue, Amazon S3 Tables, and AWS Quick

DevFeed: [Analyze Anthropic usage, spend, and access with AWS Glue, Amazon S3 Tables, and AWS Quick](<https://devfeed.tech/articles/analyze-anthropic-usage-spend-and-access-with-aws-glue-amazon-s3-tables-and-aws-quick-136178.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/analyze-anthropic-usage-spend-and-access-with-aws-glue-amazon-s3-tables-and-aws-quick/>)

Author: Daniyal Alam

Published: 2026-11-16T20:09:24Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Claude](<https://devfeed.tech/topics/claude.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [anthropic](<https://devfeed.tech/topics/anthropic.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [JOIN](<https://devfeed.tech/topics/join.md>), [data ownership](<https://devfeed.tech/topics/data-ownership.md>), [AWS IAM](<https://devfeed.tech/topics/aws-iam.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [api](<https://devfeed.tech/tags/api.md>), [api-key](<https://devfeed.tech/tags/api-key.md>), [athena](<https://devfeed.tech/tags/athena.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-quick](<https://devfeed.tech/tags/aws-quick.md>), [s3](<https://devfeed.tech/tags/s3.md>), [s3-tables](<https://devfeed.tech/tags/s3-tables.md>), [tables](<https://devfeed.tech/tags/tables.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

AWS Glue's native Anthropic connector can load usage, cost, users, workspaces, and API key data into an AWS lakehouse without custom integration code. The walkthrough stores API keys in Secrets Manager, uses separate Admin and workspace connections for their different data scopes, and writes raw and curated Apache Iceberg tables to Amazon S3 Tables. AWS Glue materialized views join that data with business unit, outcome, and employee reference data; Athena queries the results, and AWS Quick can build dashboards directly from the views. The example surfaces spending by business unit, cost per outcome, and recently used keys created by former employees for ownership review--not automatic evidence of misuse. The article notes API rate limits, recommends bounded incremental reads, and specifies Glue materialized view requirements including Glue 5.1 or later, iam:PassRole, deterministic definitions, and inline subqueries because WITH-based views failed to refresh in its testing.

### Source excerpt

The new AWS Glue connector for Anthropic reads Claude usage, cost, and API key data into AWS without custom code. This post shows how to land it in Amazon S3 Tables as Apache Iceberg tables, join it with your business data, query it with Amazon Athena, and visualize AI spend and governance in AWS Quick.

## Multi-statement transactions for Apache Iceberg tables in Amazon Redshift

DevFeed: [Multi-statement transactions for Apache Iceberg tables in Amazon Redshift](<https://devfeed.tech/articles/multi-statement-transactions-for-apache-iceberg-tables-in-amazon-redshift-133388.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/multi-statement-transactions-for-apache-iceberg-tables-in-amazon-redshift/>)

Author: Shiyu Xu

Published: 2026-11-13T19:32:46Z

Content type: release

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [atomic](<https://devfeed.tech/tags/atomic.md>), [foundational-100](<https://devfeed.tech/tags/foundational-100.md>), [snapshot](<https://devfeed.tech/tags/snapshot.md>), [transactions](<https://devfeed.tech/tags/transactions.md>)

### AI overview

Amazon Redshift Patch 206 and later can group multiple DML statements against one Apache Iceberg table into a single `BEGIN ... COMMIT`, publishing one atomic snapshot. The transaction can include an audit before commit; failures and rollbacks publish no changes. All writes must target the same table using the same reference name, while reads may include uncommitted writes and each table's version is fixed on its first read. Run Iceberg DDL separately. Uncommitted intermediate files can become orphans; S3 Tables removes unreferenced files by default, while Glue tables require the orphan file deletion optimizer enabled in advance. In the reported test, batching 25 small statements reduced total runtime by 14.77% for INSERT and 11.29% for DELETE; heavy data work showed no meaningful speedup. At 32 concurrent writers, four-INSERT transactions achieved 3.59 times the committed-statement rate of auto-commit. Results vary with statement size, partitioning, and contention.

### Source excerpt

Amazon Redshift now supports multi-statement transactions for Apache Iceberg tables, so you can run several DML statements against one Iceberg table inside a single BEGIN...COMMIT and publish them as one atomic Iceberg snapshot. This post explains why one commit per statement makes multi-step pipelines fragile, what you can run in a transaction, and how it performs.

## Towards Unified File Management for Data Lakehouse Systems

DevFeed: [Towards Unified File Management for Data Lakehouse Systems](<https://devfeed.tech/articles/towards-unified-file-management-for-data-lakehouse-systems-132696.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15975>)

Author: Keonwoo Oh, Akshay Mehra, Joshua Kimball, Amol Deshpande

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [Data Management](<https://devfeed.tech/topics/data-management.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [scalable-architecture](<https://devfeed.tech/topics/scalable-architecture.md>), [rdbms](<https://devfeed.tech/topics/rdbms.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [database](<https://devfeed.tech/tags/database.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>), [garbage-collection](<https://devfeed.tech/tags/garbage-collection.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [latency](<https://devfeed.tech/tags/latency.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [metadata-management](<https://devfeed.tech/tags/metadata-management.md>), [raven](<https://devfeed.tech/tags/raven.md>), [snapshot-isolation](<https://devfeed.tech/tags/snapshot-isolation.md>), [tracking](<https://devfeed.tech/tags/tracking.md>), [transactions](<https://devfeed.tech/tags/transactions.md>), [versioning](<https://devfeed.tech/tags/versioning.md>)

### AI overview

Raven combines a transactional database catalog for transactional metadata with underlying Open Table Formats (OTFs) for analytical metadata, aiming to balance database scalability with file-based metadata management. Its extended Merge-on-Read mechanism is designed to reduce commit latency and metadata bloat, while snapshot isolation supports multi-table transactions. An extensible entity-relationship model unifies versioning and reference tracking to enable global garbage collection across asset types. The abstract reports lower commit latency and metadata bloat with Apache Iceberg and Delta Lake, catalog storage orders of magnitude smaller than database-only DuckLake, and scalable garbage collection with minimal disruption; it provides no evaluation details or quantitative measurements.

### Source excerpt

arXiv:2611.15975v1 Announce Type: new Abstract: Data lakehouse systems have garnered widespread adoption, yet their metadata management systems face a fundamental architectural dilemma. File-based OTFs (Open Table Formats) scale well, but suffer from metadata bloat, high commit latencies, the absence of multi-table transactions, and unsafe garbage collection. On the other hand, catalogs relying entirely on a transactional database incur severe scalability bottlenecks due to analytic file-level metadata. To reconcile this tension, we present Raven, an operational catalog that stores transactional metadata within a database while offloading analytical metadata to underlying OTF structures. Based on this hybrid architecture, Raven introduces an extended Merge-on-Read (MoR) mechanism to reduce commit latency and mitigate metadata bloat, alongside multi-table transactions under snapshot isolation. Furthermore, Raven unifies core data lakehouse semantics including versioning and reference tracking under an extensible ER (Entity-Relationship) metadata model, enabling safe global garbage collection across diverse types of assets. Empirical evaluations demonstrate that Raven substantially reduces commit latency and metadata bloat when integrated with Apache Iceberg and Delta Lake, maintains a catalog storage footprint orders of magnitude smaller than database-only systems like DuckLake, and achieves scalable garbage collection with minimal disruption.

## Migrating from Apache Hudi-based data lake to Apache Iceberg-based data lake

DevFeed: [Migrating from Apache Hudi-based data lake to Apache Iceberg-based data lake](<https://devfeed.tech/articles/migrating-from-apache-hudi-based-data-lake-to-apache-iceberg-based-data-lake-130716.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/migrating-from-apache-hudi-based-data-lake-to-apache-iceberg-based-data-lake/>)

Author: Nitin Kumar

Published: 2026-11-12T15:50:48Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-hudi](<https://devfeed.tech/tags/apache-hudi.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [athena](<https://devfeed.tech/tags/athena.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [based](<https://devfeed.tech/tags/based.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-lakes](<https://devfeed.tech/tags/data-lakes.md>), [files](<https://devfeed.tech/tags/files.md>), [full](<https://devfeed.tech/tags/full.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [hudi](<https://devfeed.tech/tags/hudi.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [insert](<https://devfeed.tech/tags/insert.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The excerpt presents three Hudi-to-Iceberg migration paths on AWS. In-place migration reuses Parquet files, saving rewrite time and storage but retaining the source schema, partitioning, and file layout. Iceberg's `add_files` suits a one-time conversion and requires Hudi cleanup first; for MoR, compact logs and clean old versions to avoid missing updates or duplicate rows. Apache XTable translates Hudi metadata and can sync ongoing writes, but MoR logs must be compacted before each conversion. Both approaches share data files, so avoid destructive Iceberg maintenance until cutover; `add_files` and `snapshot` also lack column statistics in Iceberg 1.9.0-1.10.0, with the issue fixed in 1.10.1. Full migration with CTAS or CREATE plus INSERT rewrites data, allowing schema and partition changes, including hidden partitioning, at higher compute, time, and storage cost. For large tables, partitioned INSERT batches support parallelism and resumability. The excerpt demonstrates AWS Glue notebooks, general S3 buckets, and Amazon S3 Tables, and recommends validating against Hudi before cutover.

### Source excerpt

Learn how to migrate transactional data lakes from Apache Hudi to Apache Iceberg on AWS. This post walks through in-place migration with the Iceberg add_files procedure and Apache XTable, plus full table migration with CTAS and INSERT, using AWS Glue notebooks and Amazon S3 Tables, then validates the results in Amazon Athena.

## How Open Is Open? Iceberg, Catalogs, and User Choice

DevFeed: [How Open Is Open? Iceberg, Catalogs, and User Choice](<https://devfeed.tech/articles/how-open-is-open-iceberg-catalogs-and-user-choice-129822.md>)

Original publisher: [Read original article](<https://www.dataengineeringweekly.com/p/how-open-is-open-iceberg-catalogs>)

Author: Ananth Packkildurai

Published: 2026-11-11T20:50:33Z

Content type: opinion

Language: en

Sources: [Data Engineering Weekly](<https://devfeed.tech/sources/data-engineering-weekly.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [API Governance](<https://devfeed.tech/topics/api-governance.md>), [Integration testing](<https://devfeed.tech/topics/integration-testing.md>)

Tags: [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-polaris](<https://devfeed.tech/tags/apache-polaris.md>), [api](<https://devfeed.tech/tags/api.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [change](<https://devfeed.tech/tags/change.md>), [choice](<https://devfeed.tech/tags/choice.md>), [commit](<https://devfeed.tech/tags/commit.md>), [control](<https://devfeed.tech/tags/control.md>), [engines](<https://devfeed.tech/tags/engines.md>), [giving](<https://devfeed.tech/tags/giving.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [next](<https://devfeed.tech/tags/next.md>), [open](<https://devfeed.tech/tags/open.md>), [providers](<https://devfeed.tech/tags/providers.md>), [test](<https://devfeed.tech/tags/test.md>), [users](<https://devfeed.tech/tags/users.md>)

### AI overview

Apache Iceberg's REST catalog protocol offers a foundation for catalog and engine choice, but an API connection alone does not establish production interoperability. The excerpt argues for a community-run, production-ready catalog as a common integration target, backed by public, versioned compatibility results that spell out supported operations, optional capabilities, configurations, and limitations. It also calls for controlled performance workloads covering metadata loading, scan planning, and concurrent commits, plus tested migration procedures addressing permissions, identities, credentials, catalog state, and active writers. Apache Polaris and Iceberg's REST Compatibility Kit are existing contributions; the proposed next step is to connect them to explicit expectations and evidence, while allowing alternative implementations and keeping the specification authoritative.

### Source excerpt

The next test of openness: giving users the freedom to choose engines, control their catalogs, and change providers.

## How Samsung Ads federated Snowflake Iceberg tables using Amazon Cognito

DevFeed: [How Samsung Ads federated Snowflake Iceberg tables using Amazon Cognito](<https://devfeed.tech/articles/how-samsung-ads-federated-snowflake-iceberg-tables-using-amazon-cognito-129562.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/how-samsung-ads-federated-snowflake-iceberg-tables-using-amazon-cognito/>)

Author: Balaji Selva Rajan

Published: 2026-11-11T19:48:02Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon Cognito](<https://devfeed.tech/topics/amazon-cognito.md>), [AWS Lake Formation](<https://devfeed.tech/topics/aws-lake-formation.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon VPC](<https://devfeed.tech/topics/amazon-vpc.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [AWS CloudShell](<https://devfeed.tech/topics/aws-cloudshell.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [ads](<https://devfeed.tech/tags/ads.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-cognito](<https://devfeed.tech/tags/amazon-cognito.md>), [amazon-route-53](<https://devfeed.tech/tags/amazon-route-53.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [amazon-sagemaker-ai](<https://devfeed.tech/tags/amazon-sagemaker-ai.md>), [amazon-vpc](<https://devfeed.tech/tags/amazon-vpc.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [authentication](<https://devfeed.tech/tags/authentication.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cli](<https://devfeed.tech/tags/aws-cli.md>), [aws-cloudformation](<https://devfeed.tech/tags/aws-cloudformation.md>), [aws-cloudshell](<https://devfeed.tech/tags/aws-cloudshell.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-lake-formation](<https://devfeed.tech/tags/aws-lake-formation.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [oauth2](<https://devfeed.tech/tags/oauth2.md>), [samsung](<https://devfeed.tech/tags/samsung.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Samsung Ads federated Snowflake-managed Iceberg tables into AWS Lake Formation so Amazon Athena can query the same S3 data without a second copy. An AWS Glue connection obtains OAuth2 client-credentials tokens from Amazon Cognito; a Pre Token Generation Lambda adds the `aud` claim Snowflake requires. The connection reaches Snowflake's Iceberg REST catalog over AWS PrivateLink, while Cognito's public token endpoint requires NAT access. Lake Formation governs access and vends scoped S3 credentials for Athena to read the Parquet files directly. Setup requires Snowflake Business-Critical or higher in the same Region, Snowflake-managed Iceberg tables, and AWS permissions for the Cognito, networking, and Lake Formation resources. The pattern does not apply Snowflake masking or row access policies to Athena reads; those controls must be recreated in Lake Formation and tested. Federate schemas containing only supported Snowflake-managed tables, and explicitly revoke Lake Formation grants when tables are removed. NAT and PrivateLink endpoints incur charges.

### Source excerpt

Learn how Samsung Ads federates Snowflake-managed Apache Iceberg tables into AWS Lake Formation and queries them from Amazon Athena using Amazon Cognito for OAuth2 machine-to-machine authentication, with no second copy of the data to maintain.

## SAP's latest API policy raises the stakes for your AI strategy | Blog | Fivetran

DevFeed: [SAP's latest API policy raises the stakes for your AI strategy | Blog | Fivetran](<https://devfeed.tech/articles/sap-s-latest-api-policy-raises-the-stakes-for-your-ai-strategy-blog-fivetran-136816.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/saps-latest-api-policy-raises-the-stakes-for-your-ai-strategy>)

Author: Taylor Brown

Published: 2026-11-11T16:05:32Z

Content type: opinion

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-strategy](<https://devfeed.tech/tags/ai-strategy.md>), [api](<https://devfeed.tech/tags/api.md>), [blog](<https://devfeed.tech/tags/blog.md>), [data](<https://devfeed.tech/tags/data.md>), [data-infrastructure](<https://devfeed.tech/tags/data-infrastructure.md>), [doesn-t](<https://devfeed.tech/tags/doesn-t.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [latest](<https://devfeed.tech/tags/latest.md>), [matters](<https://devfeed.tech/tags/matters.md>), [means](<https://devfeed.tech/tags/means.md>), [more](<https://devfeed.tech/tags/more.md>), [open](<https://devfeed.tech/tags/open.md>), [open-data](<https://devfeed.tech/tags/open-data.md>), [policy](<https://devfeed.tech/tags/policy.md>), [raises](<https://devfeed.tech/tags/raises.md>), [sap](<https://devfeed.tech/tags/sap.md>), [strategy](<https://devfeed.tech/tags/strategy.md>)

### AI overview

The article says SAP's policy bars API use by semi-autonomous or generative AI systems that plan or execute API calls, and restricts large-scale extraction to SAP-controlled pathways. It reports that Fivetran's SAP connectors are unaffected and that the policy is not a contract, with SAP targeting ODP RFC blocking from July and some exceptions through year-end. The author argues the broader move toward SAP-controlled integration could increase costs, limit cloud-service choice, tie business logic to the vendor, and constrain AI access. Fivetran presents Open Data Infrastructure as an alternative: decouple storage, compute, and integration; keep data in a customer-controlled lake or warehouse; and use open formats such as Iceberg and Delta Lake for portability across BI, ML, and AI tools.

### Source excerpt

What the new policy means, what it doesn't, and why Open Data Infrastructure matters more than ever.

## Bring high-performance analytics to your lakehouse with ClickHouse and Amazon S3 Tables

DevFeed: [Bring high-performance analytics to your lakehouse with ClickHouse and Amazon S3 Tables](<https://devfeed.tech/articles/bring-high-performance-analytics-to-your-lakehouse-with-clickhouse-and-amazon-s3-tables-131558.md>)

Original publisher: [Read original article](<https://clickhouse.com/blog/clickhouse-amazon-s3-tables>)

Author: AWS

Published: 2026-11-11T00:00:00Z

Content type: tutorial

Language: en

Sources: [ClickHouse Blog](<https://devfeed.tech/sources/clickhouse-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>)

Tags: [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-emr](<https://devfeed.tech/tags/amazon-emr.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-identity-and-access-management-iam](<https://devfeed.tech/tags/aws-identity-and-access-management-iam.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [retail-analytics](<https://devfeed.tech/tags/retail-analytics.md>)

### AI overview

ClickHouse can query Apache Iceberg data in Amazon S3 Tables through the Iceberg REST endpoint, transform it with SQL, and write curated results back without migrating the source data. The walkthrough uses ClickHouse's DataLakeCatalog engine and AWS IAM credentials to join retail purchases and returns into a reusable daily performance table; other Iceberg-compatible engines can then read that table. S3 Tables manage storage, metadata, and automatic maintenance, while ClickHouse performs analytics. The article also describes using ClickHouse as a tool for iterative AI agent queries and writes. The workflow requires self-managed ClickHouse 26.8 or later; this read/write path is not yet generally available in ClickHouse Cloud. Querying in place has different latency and throughput from local MergeTree tables: the article suggests it for exploration, batch transforms, and low-concurrency workloads, and materialization for consistently low-latency, high-concurrency dashboards or applications. For centralized catalog governance, fine-grained access across engines, or cross-account access, it points to the AWS Glue Iceberg REST endpoint.

### Source excerpt

Connect ClickHouse to Amazon S3 Tables to query Iceberg data in place, transform retail analytics, and write reusable results back for dashboards and AI agents.

## Is your Iceberg catalog healthy?

DevFeed: [Is your Iceberg catalog healthy?](<https://devfeed.tech/articles/is-your-iceberg-catalog-healthy-129096.md>)

Original publisher: [Read original article](<https://medium.com/snowflake/is-your-iceberg-catalog-healthy-939dc7817154?source=rss----34b6daafc07---4>)

Author: Daniel Hunt

Published: 2026-11-10T22:38:46Z

Content type: comparison

Language: en

Sources: [Snowflake Builders](<https://devfeed.tech/sources/snowflake-builders.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [vendor lock-in](<https://devfeed.tech/topics/vendor-lock-in.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-polaris](<https://devfeed.tech/tags/apache-polaris.md>), [architect](<https://devfeed.tech/tags/architect.md>), [blog](<https://devfeed.tech/tags/blog.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [completeness](<https://devfeed.tech/tags/completeness.md>), [contributing](<https://devfeed.tech/tags/contributing.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [development](<https://devfeed.tech/tags/development.md>), [disclosure](<https://devfeed.tech/tags/disclosure.md>), [experience](<https://devfeed.tech/tags/experience.md>), [extension](<https://devfeed.tech/tags/extension.md>), [focus](<https://devfeed.tech/tags/focus.md>), [functional](<https://devfeed.tech/tags/functional.md>), [functionality](<https://devfeed.tech/tags/functionality.md>), [governance](<https://devfeed.tech/tags/governance.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [open-source](<https://devfeed.tech/tags/open-source.md>)

### AI overview

The comparison rates Apache Polaris highest overall (10/10 for project health), with Apache Gravitino close behind (9/10); Lakekeeper scores 5/10, Unity Catalog OSS 3/10, and Nessie 0/10 under the author's criteria. Polaris and Gravitino lead on commit velocity and contributing-organisation counts, though commits outside their lead contributors account for 32% and 15%, respectively. Lakekeeper has eight contributing organisations but remains single-vendor. On capability, Polaris and Lakekeeper implement the full Iceberg REST catalog specification, including multi-table commits, authorization and scoped credential vending. Gravitino supports federation through connectors but lacks multi-table transactions; Unity Catalog OSS lacks named-role RBAC and documented migration from the managed service; Nessie labels REST catalog support experimental. The author presents the results as a reproducible, time-bounded assessment based on public project data and documentation, with unattributed contributors making organisation shares lower bounds.

### Source excerpt

A comparison of five open-source Iceberg catalogs on functionality and OSS project health. Apache Iceberg™ allows an organisation to own their data layer in an open format in their own storage, and has become the de facto open table format. Still, every engine must reach those Iceberg tables through an Iceberg catalog. A natural extension is an open-source Apache Iceberg catalog to complete the open lakehouse stack, acting as an open governance layer, so that access to your tables isn't dependent on any one vendor's managed service. In this blog, I aim to simplify the decision process when it comes to choosing an Iceberg catalog and therefore building an open lakehouse on Iceberg. I focus on the functional completeness of each catalog for Iceberg data and dive deep on OSS project health. After all, an OSS project with one vendor driving development carries similar lock-in risk to a proprietary product. My goal here is to provide transparency on the relative health of common OSS catalogs, specifically Apache Polaris™, Apache Gravitino™, Lakekeeper, Unity Catalog OSS and Project Nessie. The heart of the blog is a collation of public information from each project (which you can review in the public Git repo), but the criteria and analysis are informed by my experience as a specialist architect working with 50+ organisations implementing Apache Iceberg over the past couple of years. This is a companion post to Iceberg Catalogs: Below the Surface (updated November 2026), which scores six vendor-hosted catalogs and shares the same underlying repo. I split the two posts since the first one was becoming a dissertation! Disclosure: This is my personal opinion, not representative of the views of my employer. I'm a specialist architect at Snowflake, one of the vendors contributing towards the Apache Polaris project. Every claim below is sourced and reproducible. Don't take my word for it: test the claims, read the linked project docs, and please comment if you spot any inconsi

## Bring high-performance analytics to your lakehouse with ClickHouse and Amazon S3 Tables

DevFeed: [Bring high-performance analytics to your lakehouse with ClickHouse and Amazon S3 Tables](<https://devfeed.tech/articles/bring-high-performance-analytics-to-your-lakehouse-with-clickhouse-and-amazon-s3-tables-129190.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/storage/bring-high-performance-analytics-to-your-lakehouse-with-clickhouse-and-amazon-s3-tables/>)

Author: Karolina Ruiz Rogelj

Published: 2026-11-10T16:52:00Z

Content type: tutorial

Language: en

Sources: [AWS Storage Blog](<https://devfeed.tech/sources/aws-storage-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [connectivity](<https://devfeed.tech/tags/connectivity.md>), [data](<https://devfeed.tech/tags/data.md>), [engine](<https://devfeed.tech/tags/engine.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [open](<https://devfeed.tech/tags/open.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [performance](<https://devfeed.tech/tags/performance.md>), [query](<https://devfeed.tech/tags/query.md>), [s3](<https://devfeed.tech/tags/s3.md>), [s3-tables](<https://devfeed.tech/tags/s3-tables.md>), [tables](<https://devfeed.tech/tags/tables.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

ClickHouse can query and transform Apache Iceberg tables in Amazon S3 Tables without copying the data, then write curated results back as standard Iceberg tables for other compatible engines. It connects through the S3 Tables Iceberg REST endpoint using the DataLakeCatalog engine and AWS IAM credentials; S3 Tables handles storage, metadata, and automatic maintenance, while ClickHouse runs the analytics. The walkthrough requires self-managed ClickHouse 26.8 or later, an AWS account and table bucket, network access, and a least-privilege IAM role. This read-in-place approach suits exploration, batch transforms, and low-concurrency workloads; dashboards needing consistently low latency at high concurrency may require materializing data into MergeTree. The described S3 Tables read/write path is not yet generally available in ClickHouse Cloud. For centralized catalog governance, fine-grained access across engines, or cross-account access, the article points to AWS Glue's Iceberg REST endpoint.

### Source excerpt

Learn how to bring ClickHouse's high-performance engine directly to data in Amazon S3 Tables using bidirectional connectivity. Query your Iceberg tables in place, run transformations in ClickHouse SQL, and write curated results back, while S3 Tables remain the single open source of truth.

## How I Taught an AI to Read My Logs

DevFeed: [How I Taught an AI to Read My Logs](<https://devfeed.tech/articles/how-i-taught-an-ai-to-read-my-logs-127277.md>)

Original publisher: [Read original article](<https://medium.com/expedia-group-tech/how-i-taught-an-ai-to-read-my-logs-9c04f1cfa750?source=rss----38998a53046f---4>)

Author: Vincent DANIEL

Published: 2026-11-10T12:01:02Z

Content type: article

Language: en

Sources: [Expedia](<https://devfeed.tech/sources/expedia.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [aws](<https://devfeed.tech/tags/aws.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [execution-plan](<https://devfeed.tech/tags/execution-plan.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [spark](<https://devfeed.tech/tags/spark.md>)

### AI overview

Expedia Group describes two AI-assisted skills that diagnose Spark jobs from actual execution plans, traces, and catalog metadata instead of relying on configuration or code. The after-the-fact cost analysis correlates stage shuffle, spill and skew data with executor memory and SQL plans, identifying issues such as full-scan MERGE statements, driver or Python memory pressure, and stale Iceberg tables. A Trino MCP connection checks snapshot history and reconciles source DDL with the live catalog. A deployment skill can dry-run risky changes against a non-production table or branch, then monitor live stage progress about every ten seconds and choose recovery steps from the observed trace. The account reports over 90% lower daily compute cost on a couple of pipelines after Iceberg V3 migrations, and says plan analysis also uncovered ongoing data loss on multiple pipelines. Analysis has limits: used pre-signed dashboard URLs need a fresh request, and server-side traces are unavailable for jobs older than roughly 30 days.

### Source excerpt

Expedia Group Technology - DataAI-assisted data engineering at Expedia Group -- and less squinting at Spark execution plansPhoto by Queensland, Australia on Unsplash Anyone who has actually opened a Spark execution plan to figure out why a job is slow or expensive knows what the pain looks like. It's not one screen -- it's the Spark UI's stage DAG, then the SQL tab's physical plan with nested Exchange, BroadcastHashJoin, and SortMergeJoin operators stacked five levels deep, then a separate executors tab to check whether any of that actually correlates with memory pressure, then cross-referencing timestamps against a job's config history to rule out "did someone change something last week." Reading a plan correctly means holding all of that in your head at once, and most of it is stored in different tabs that don't talk to each other. Doing this well takes real practice, and doing it under time pressure -- a pipeline failed overnight, someone needs an answer before standup -- is exactly when people take the fastest wrong guess instead of the right slow one. This is the part of the job that AI turned out to be genuinely good at, and it's the part of the last two years worth focusing on. A brief origin point: Iceberg V3 The first place this paid off concretely was migrating tables to Apache Iceberg's V3 format after it shipped in mid-2025. The work itself was conversational -- reading a table's delete-file accumulation, reading the actual scan plan against that table, deciding whether a merge-on-read migration to deletion vectors was worth it, then checking the resulting plan again to confirm it worked. Daily compute cost on a couple of pipelines that got this treatment dropped by more than 90%. That's a good result, but the more important thing it proved was narrower: that reading a real execution plan and matching it against real table metadata, rather than trusting what the DDL or the job config claimed, was something an AI could do reliably enough to act on. (The deleti

## Amazon Athena introduces system tables for query execution observability

DevFeed: [Amazon Athena introduces system tables for query execution observability](<https://devfeed.tech/articles/amazon-athena-introduces-system-tables-for-query-execution-observability-126343.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/amazon-athena-introduces-system-tables-for-query-execution-observability/>)

Author: Pathik Shah

Published: 2026-11-09T18:06:44Z

Content type: release

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [observability](<https://devfeed.tech/topics/observability.md>), [AI for SQL query generation](<https://devfeed.tech/topics/ai-for-sql-query-generation.md>), [audit](<https://devfeed.tech/topics/audit.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-skill](<https://devfeed.tech/tags/agent-skill.md>), [agent-toolkit](<https://devfeed.tech/tags/agent-toolkit.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [attribution](<https://devfeed.tech/tags/attribution.md>), [audit](<https://devfeed.tech/tags/audit.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [cost](<https://devfeed.tech/tags/cost.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [observability](<https://devfeed.tech/tags/observability.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [sql](<https://devfeed.tech/tags/sql.md>), [trends](<https://devfeed.tech/tags/trends.md>)

### AI overview

Amazon Athena's new `query_executions` system table records each completed query, including failures, with identity, workgroup, timing, data scanned, DPU count, and outcome details. Enable system tables in the Athena console; records become queryable within minutes as Apache Iceberg data in an S3 Tables bucket in your account. Athena manages table maintenance, while you control encryption, access, and retention; no default retention is set. The table supports cost attribution, performance trends, audit, error analysis, and capacity reservation analysis. Data is read-only for account principals and can also be queried by Iceberg-compatible engines. For natural-language exploration, AWS describes using an MCP-compatible agent with the Athena system tables skill and read-only IAM access; the skill provides schemas and partition-aware query examples.

### Source excerpt

Amazon Athena now offers system tables: SQL-queryable, per-query execution metadata stored as Apache Iceberg in your own account. This post walks through what system tables are, how to turn them on, and how to use them for usage analysis, cost attribution, performance trends, audit, and capacity optimization without custom pipelines.

[Next page](<https://devfeed.tech/topics/apache-iceberg.md?cursor=WyIyMDI2LTExLTA5VDE4OjA2OjQ0KzAwOjAwIiwgImE3NTc1M2UxLTc0ZTMtNGMzMy1hODFjLTY5Zjg0ODhjZDljNiJd>)