# Amazon S3 Tables

Amazon S3 Tables is a fully managed Apache Iceberg table service in Amazon S3 that automates data-lake table maintenance and supports Iceberg-compatible query engines.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Amazon S3 Tables with dbt: Iceberg-based data transformations using AWS Glue

DevFeed: [Amazon S3 Tables with dbt: Iceberg-based data transformations using AWS Glue](<https://devfeed.tech/articles/amazon-s3-tables-with-dbt-iceberg-based-data-transformations-using-aws-glue-144273.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/amazon-s3-tables-with-dbt-iceberg-based-data-transformations-using-aws-glue/>)

Author: Yukiko Iwazumi

Published: 2026-11-19T17:58:50Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-quick-sight](<https://devfeed.tech/tags/amazon-quick-sight.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The walkthrough builds a dbt pipeline on Amazon S3 Tables through AWS Glue interactive sessions, with raw CSV data staged and transformed into analytics marts. It requires dbt-glue 1.12.4 or later for S3 Tables incremental models, Python 3.10-3.12, and AWS Glue 6.0 or later. In the example, `fact_orders` uses merge to retain each order's latest state, while `fact_order_history` uses append to retain changes. After source updates, the merge table has six orders, including the updated order 1004 and new order 1006; the history table grows from five to seven rows. The post describes S3 Tables support as experimental in dbt-glue 1.12.4 and recommends testing it in your environment.

### Source excerpt

Amazon S3 Tables, a capability of Amazon S3, bring built-in Apache Iceberg support to object storage. This post shows how to configure and run dbt transformations on S3 Tables with the AWS Glue dbt-glue adapter, using incremental merge and append strategies across a staging and mart architecture.

## Finding Amazon S3 cost savings with natural language using AWS Quick and S3 Tables

DevFeed: [Finding Amazon S3 cost savings with natural language using AWS Quick and S3 Tables](<https://devfeed.tech/articles/finding-amazon-s3-cost-savings-with-natural-language-using-aws-quick-and-s3-tables-143610.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/storage/finding-amazon-s3-cost-savings-with-natural-language-using-aws-quick-and-s3-tables/>)

Author: Santosh Setty

Published: 2026-11-18T23:59:46Z

Content type: tutorial

Language: en

Sources: [AWS Storage Blog](<https://devfeed.tech/sources/aws-storage-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [cloud cost management](<https://devfeed.tech/topics/cloud-cost-management.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [text2sql](<https://devfeed.tech/topics/text2sql.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [amazon-simple-storage-service-amazon-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-amazon-s3.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cloud-storage](<https://devfeed.tech/tags/aws-cloud-storage.md>), [aws-quick](<https://devfeed.tech/tags/aws-quick.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cost-savings](<https://devfeed.tech/tags/cost-savings.md>), [costs](<https://devfeed.tech/tags/costs.md>), [data](<https://devfeed.tech/tags/data.md>), [growth](<https://devfeed.tech/tags/growth.md>), [logs](<https://devfeed.tech/tags/logs.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [overhead](<https://devfeed.tech/tags/overhead.md>), [query](<https://devfeed.tech/tags/query.md>), [questions](<https://devfeed.tech/tags/questions.md>), [replication](<https://devfeed.tech/tags/replication.md>), [request](<https://devfeed.tech/tags/request.md>), [s3](<https://devfeed.tech/tags/s3.md>), [s3-tables](<https://devfeed.tech/tags/s3-tables.md>), [tables](<https://devfeed.tech/tags/tables.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Combining S3 Storage Lens metrics with object-level server access logs in S3 Tables lets AWS Quick answer natural-language questions about storage costs across buckets. Storage Lens provides bytes, object counts, and request metrics by storage class and prefix; access logs show object reads but not storage size. The walkthrough exports daily metrics to the aws-s3 table bucket, sends logs through CloudWatch Logs to aws-cloudwatch, then connects both to Quick. Example analyses identify cold prefixes, incomplete multipart uploads, noncurrent versions, cross-Region replication and reads, and rising GET traffic. Setup requires Quick Enterprise, and initial data may take 1-2 days to become useful; logs include sensitive requester and object details. Quick's findings are recommendations: verify the SQL or query in Athena and confirm retention needs before applying permanent expiration rules.

### Source excerpt

Across thousands of S3 buckets, the data you need to cut storage costs sits in different places. Learn how to bring Amazon S3 Storage Lens metrics and server access logs together in S3 Tables and query them with AWS Quick. Find cold data, wasted storage, replication overhead, and request growth by asking questions in natural language.

## Long-term Amazon Redshift observability with Datadog and Amazon S3 Tables

DevFeed: [Long-term Amazon Redshift observability with Datadog and Amazon S3 Tables](<https://devfeed.tech/articles/long-term-amazon-redshift-observability-with-datadog-and-amazon-s3-tables-141880.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/long-term-amazon-redshift-observability-with-datadog-and-amazon-s3-tables/>)

Author: Raghu Kuppala

Published: 2026-11-18T22:57:27Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [datadog](<https://devfeed.tech/topics/datadog.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [elt](<https://devfeed.tech/topics/elt.md>)

Tags: [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-emr](<https://devfeed.tech/tags/amazon-emr.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [aws-cli](<https://devfeed.tech/tags/aws-cli.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-identity-and-access-management-iam](<https://devfeed.tech/tags/aws-identity-and-access-management-iam.md>), [aws-key-management-service-aws-kms](<https://devfeed.tech/tags/aws-key-management-service-aws-kms.md>), [data-observability](<https://devfeed.tech/tags/data-observability.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [history](<https://devfeed.tech/tags/history.md>), [integration](<https://devfeed.tech/tags/integration.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [observability](<https://devfeed.tech/tags/observability.md>), [performance](<https://devfeed.tech/tags/performance.md>), [query](<https://devfeed.tech/tags/query.md>)

### AI overview

Amazon Redshift system tables retain history for only 7 days. The S3 Tables integration continuously exports selected system table data as Apache Iceberg tables, letting customers set retention and query the history without collection queries consuming warehouse compute. Datadog Data Observability can read that history to build column-level lineage, normalized query profiles, and context for quality alerts across connected clusters and workgroups. It still connects to Redshift for metadata and runs configured aggregate quality monitors there. Setup requires enabling per-warehouse exports, integrating S3 Tables with the AWS Glue Data Catalog, and granting the Datadog IAM role read access; the feature is described as a preview requiring account-team access. Disabling exports creates a history gap that cannot be backfilled after Redshift's 7-day window, and customer-managed KMS encryption keys cannot be changed after table creation.

### Source excerpt

Amazon Redshift keeps only 7 days of system table history. This post pairs Datadog Data Observability with the new Amazon Redshift system table integration with Amazon S3 Tables, so you can retain query history for the long term and analyze lineage, query performance, and data quality without adding load to your warehouses.

## Orchestrate DuckDB analytics pipelines with Amazon MWAA

DevFeed: [Orchestrate DuckDB analytics pipelines with Amazon MWAA](<https://devfeed.tech/articles/orchestrate-duckdb-analytics-pipelines-with-amazon-mwaa-141881.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/orchestrate-duckdb-analytics-pipelines-with-amazon-mwaa/>)

Author: Vinod Jayendra

Published: 2026-11-18T20:57:09Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [parquet](<https://devfeed.tech/topics/parquet.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-managed-workflows-for-apache-airflow-amazon-mwaa](<https://devfeed.tech/tags/amazon-managed-workflows-for-apache-airflow-amazon-mwaa.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [data](<https://devfeed.tech/tags/data.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [effective](<https://devfeed.tech/tags/effective.md>), [embedded](<https://devfeed.tech/tags/embedded.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [learn-how-to](<https://devfeed.tech/tags/learn-how-to.md>), [model](<https://devfeed.tech/tags/model.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [retry](<https://devfeed.tech/tags/retry.md>), [s3](<https://devfeed.tech/tags/s3.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Amazon MWAA can orchestrate bounded SQL analytics by running DuckDB inside Airflow worker tasks, without a separate query cluster. The walkthrough reads S3 Parquet files, deduplicates events, uses ASOF LEFT JOINs to apply rates and ownership effective at each event time, and merges results into three Iceberg tables in Amazon S3 Tables. A second task validates counts, uniqueness, and allocation states. On supported MWAA Provisioned environments running Airflow 3.3.1, DuckDB and its providers plus the duckdb_default AWS-aware connection are preinstalled; the execution role supplies temporary credentials. The example's unchanged-input reruns retained 300,000 unique events and duplicate-free summaries, but the excerpt says this does not establish recovery from partial failures or concurrent external writes. Each task shares one worker's CPU, memory, and storage, so limits and concurrency need tuning with representative data. Use Glue or EMR for steps needing horizontal scaling or more capacity than one worker provides. Three table commits are not one transaction, and Iceberg update/delete support can vary by version.

### Source excerpt

Learn how to orchestrate embedded DuckDB analytics pipelines with Amazon MWAA. This post shows how to run DuckDB inside Airflow tasks to read Parquet data from Amazon S3, apply time-effective model rates, and publish Apache Iceberg tables to Amazon S3 Tables, with retry-safe publishing and worker-sizing considerations.

## Analyze Anthropic usage, spend, and access with AWS Glue, Amazon S3 Tables, and AWS Quick

DevFeed: [Analyze Anthropic usage, spend, and access with AWS Glue, Amazon S3 Tables, and AWS Quick](<https://devfeed.tech/articles/analyze-anthropic-usage-spend-and-access-with-aws-glue-amazon-s3-tables-and-aws-quick-136178.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/analyze-anthropic-usage-spend-and-access-with-aws-glue-amazon-s3-tables-and-aws-quick/>)

Author: Daniyal Alam

Published: 2026-11-16T20:09:24Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Claude](<https://devfeed.tech/topics/claude.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [anthropic](<https://devfeed.tech/topics/anthropic.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [JOIN](<https://devfeed.tech/topics/join.md>), [data ownership](<https://devfeed.tech/topics/data-ownership.md>), [AWS IAM](<https://devfeed.tech/topics/aws-iam.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [api](<https://devfeed.tech/tags/api.md>), [api-key](<https://devfeed.tech/tags/api-key.md>), [athena](<https://devfeed.tech/tags/athena.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-quick](<https://devfeed.tech/tags/aws-quick.md>), [s3](<https://devfeed.tech/tags/s3.md>), [s3-tables](<https://devfeed.tech/tags/s3-tables.md>), [tables](<https://devfeed.tech/tags/tables.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

AWS Glue's native Anthropic connector can load usage, cost, users, workspaces, and API key data into an AWS lakehouse without custom integration code. The walkthrough stores API keys in Secrets Manager, uses separate Admin and workspace connections for their different data scopes, and writes raw and curated Apache Iceberg tables to Amazon S3 Tables. AWS Glue materialized views join that data with business unit, outcome, and employee reference data; Athena queries the results, and AWS Quick can build dashboards directly from the views. The example surfaces spending by business unit, cost per outcome, and recently used keys created by former employees for ownership review--not automatic evidence of misuse. The article notes API rate limits, recommends bounded incremental reads, and specifies Glue materialized view requirements including Glue 5.1 or later, iam:PassRole, deterministic definitions, and inline subqueries because WITH-based views failed to refresh in its testing.

### Source excerpt

The new AWS Glue connector for Anthropic reads Claude usage, cost, and API key data into AWS without custom code. This post shows how to land it in Amazon S3 Tables as Apache Iceberg tables, join it with your business data, query it with Amazon Athena, and visualize AI spend and governance in AWS Quick.

## Multi-statement transactions for Apache Iceberg tables in Amazon Redshift

DevFeed: [Multi-statement transactions for Apache Iceberg tables in Amazon Redshift](<https://devfeed.tech/articles/multi-statement-transactions-for-apache-iceberg-tables-in-amazon-redshift-133388.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/multi-statement-transactions-for-apache-iceberg-tables-in-amazon-redshift/>)

Author: Shiyu Xu

Published: 2026-11-13T19:32:46Z

Content type: release

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [atomic](<https://devfeed.tech/tags/atomic.md>), [foundational-100](<https://devfeed.tech/tags/foundational-100.md>), [snapshot](<https://devfeed.tech/tags/snapshot.md>), [transactions](<https://devfeed.tech/tags/transactions.md>)

### AI overview

Amazon Redshift Patch 206 and later can group multiple DML statements against one Apache Iceberg table into a single `BEGIN ... COMMIT`, publishing one atomic snapshot. The transaction can include an audit before commit; failures and rollbacks publish no changes. All writes must target the same table using the same reference name, while reads may include uncommitted writes and each table's version is fixed on its first read. Run Iceberg DDL separately. Uncommitted intermediate files can become orphans; S3 Tables removes unreferenced files by default, while Glue tables require the orphan file deletion optimizer enabled in advance. In the reported test, batching 25 small statements reduced total runtime by 14.77% for INSERT and 11.29% for DELETE; heavy data work showed no meaningful speedup. At 32 concurrent writers, four-INSERT transactions achieved 3.59 times the committed-statement rate of auto-commit. Results vary with statement size, partitioning, and contention.

### Source excerpt

Amazon Redshift now supports multi-statement transactions for Apache Iceberg tables, so you can run several DML statements against one Iceberg table inside a single BEGIN...COMMIT and publish them as one atomic Iceberg snapshot. This post explains why one commit per statement makes multi-step pipelines fragile, what you can run in a transaction, and how it performs.

## Migrating from Apache Hudi-based data lake to Apache Iceberg-based data lake

DevFeed: [Migrating from Apache Hudi-based data lake to Apache Iceberg-based data lake](<https://devfeed.tech/articles/migrating-from-apache-hudi-based-data-lake-to-apache-iceberg-based-data-lake-130716.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/migrating-from-apache-hudi-based-data-lake-to-apache-iceberg-based-data-lake/>)

Author: Nitin Kumar

Published: 2026-11-12T15:50:48Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-hudi](<https://devfeed.tech/tags/apache-hudi.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [athena](<https://devfeed.tech/tags/athena.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [based](<https://devfeed.tech/tags/based.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-lakes](<https://devfeed.tech/tags/data-lakes.md>), [files](<https://devfeed.tech/tags/files.md>), [full](<https://devfeed.tech/tags/full.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [hudi](<https://devfeed.tech/tags/hudi.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [insert](<https://devfeed.tech/tags/insert.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The excerpt presents three Hudi-to-Iceberg migration paths on AWS. In-place migration reuses Parquet files, saving rewrite time and storage but retaining the source schema, partitioning, and file layout. Iceberg's `add_files` suits a one-time conversion and requires Hudi cleanup first; for MoR, compact logs and clean old versions to avoid missing updates or duplicate rows. Apache XTable translates Hudi metadata and can sync ongoing writes, but MoR logs must be compacted before each conversion. Both approaches share data files, so avoid destructive Iceberg maintenance until cutover; `add_files` and `snapshot` also lack column statistics in Iceberg 1.9.0-1.10.0, with the issue fixed in 1.10.1. Full migration with CTAS or CREATE plus INSERT rewrites data, allowing schema and partition changes, including hidden partitioning, at higher compute, time, and storage cost. For large tables, partitioned INSERT batches support parallelism and resumability. The excerpt demonstrates AWS Glue notebooks, general S3 buckets, and Amazon S3 Tables, and recommends validating against Hudi before cutover.

### Source excerpt

Learn how to migrate transactional data lakes from Apache Hudi to Apache Iceberg on AWS. This post walks through in-place migration with the Iceberg add_files procedure and Apache XTable, plus full table migration with CTAS and INSERT, using AWS Glue notebooks and Amazon S3 Tables, then validates the results in Amazon Athena.

## Governance compliance beyond the catalog in Amazon SageMaker Unified Studio

DevFeed: [Governance compliance beyond the catalog in Amazon SageMaker Unified Studio](<https://devfeed.tech/articles/governance-compliance-beyond-the-catalog-in-amazon-sagemaker-unified-studio-129560.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/governance-compliance-beyond-the-catalog-in-amazon-sagemaker-unified-studio/>)

Author: Amom Luiz

Published: 2026-11-11T16:08:47Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [Data visualization](<https://devfeed.tech/topics/data-visualization.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data ownership](<https://devfeed.tech/topics/data-ownership.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-eventbridge](<https://devfeed.tech/tags/amazon-eventbridge.md>), [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [amazon-sagemaker-unified-studio](<https://devfeed.tech/tags/amazon-sagemaker-unified-studio.md>), [apis](<https://devfeed.tech/tags/apis.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [forms](<https://devfeed.tech/tags/forms.md>), [glossary](<https://devfeed.tech/tags/glossary.md>), [governance](<https://devfeed.tech/tags/governance.md>), [governance-compliance](<https://devfeed.tech/tags/governance-compliance.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [join](<https://devfeed.tech/tags/join.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [policy](<https://devfeed.tech/tags/policy.md>), [score](<https://devfeed.tech/tags/score.md>), [subscription](<https://devfeed.tech/tags/subscription.md>), [table](<https://devfeed.tech/tags/table.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The post describes a daily enrichment pipeline that joins SageMaker Unified Studio's asset metadata export with catalog APIs and an S3 policy file, then writes per-asset governance records to an Iceberg table. It combines built-in required-form rules from GetAssetType with organization-defined requirements, and adds glossary associations, data quality signals, descriptions, and active subscription counts. Teams can query weighted compliance scores and trends in Athena, build Quick Sight dashboards, and schedule alerts; subscription counts help prioritize low-scoring assets with wider use. The five score dimensions and weights are configurable examples, not prescribed policy, and organizations must define what compliance means for them. The sample test processed 29 Glue table assets in about 30 seconds; the post suggests parallelization, more Lambda memory, or Step Functions for larger catalogs.

### Source excerpt

Amazon SageMaker provides rich governance signals such as metadata forms, data quality scores, glossary coverage, and subscription data, but they live in separate APIs. This post shows how to join them into a single enriched Iceberg table and build a compliance scorecard you can query, score, and adapt to your governance policy.

## Bring high-performance analytics to your lakehouse with ClickHouse and Amazon S3 Tables

DevFeed: [Bring high-performance analytics to your lakehouse with ClickHouse and Amazon S3 Tables](<https://devfeed.tech/articles/bring-high-performance-analytics-to-your-lakehouse-with-clickhouse-and-amazon-s3-tables-131558.md>)

Original publisher: [Read original article](<https://clickhouse.com/blog/clickhouse-amazon-s3-tables>)

Author: AWS

Published: 2026-11-11T00:00:00Z

Content type: tutorial

Language: en

Sources: [ClickHouse Blog](<https://devfeed.tech/sources/clickhouse-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>)

Tags: [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-emr](<https://devfeed.tech/tags/amazon-emr.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-identity-and-access-management-iam](<https://devfeed.tech/tags/aws-identity-and-access-management-iam.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [retail-analytics](<https://devfeed.tech/tags/retail-analytics.md>)

### AI overview

ClickHouse can query Apache Iceberg data in Amazon S3 Tables through the Iceberg REST endpoint, transform it with SQL, and write curated results back without migrating the source data. The walkthrough uses ClickHouse's DataLakeCatalog engine and AWS IAM credentials to join retail purchases and returns into a reusable daily performance table; other Iceberg-compatible engines can then read that table. S3 Tables manage storage, metadata, and automatic maintenance, while ClickHouse performs analytics. The article also describes using ClickHouse as a tool for iterative AI agent queries and writes. The workflow requires self-managed ClickHouse 26.8 or later; this read/write path is not yet generally available in ClickHouse Cloud. Querying in place has different latency and throughput from local MergeTree tables: the article suggests it for exploration, batch transforms, and low-concurrency workloads, and materialization for consistently low-latency, high-concurrency dashboards or applications. For centralized catalog governance, fine-grained access across engines, or cross-account access, it points to the AWS Glue Iceberg REST endpoint.

### Source excerpt

Connect ClickHouse to Amazon S3 Tables to query Iceberg data in place, transform retail analytics, and write reusable results back for dashboards and AI agents.

## Bring high-performance analytics to your lakehouse with ClickHouse and Amazon S3 Tables

DevFeed: [Bring high-performance analytics to your lakehouse with ClickHouse and Amazon S3 Tables](<https://devfeed.tech/articles/bring-high-performance-analytics-to-your-lakehouse-with-clickhouse-and-amazon-s3-tables-129190.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/storage/bring-high-performance-analytics-to-your-lakehouse-with-clickhouse-and-amazon-s3-tables/>)

Author: Karolina Ruiz Rogelj

Published: 2026-11-10T16:52:00Z

Content type: tutorial

Language: en

Sources: [AWS Storage Blog](<https://devfeed.tech/sources/aws-storage-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [connectivity](<https://devfeed.tech/tags/connectivity.md>), [data](<https://devfeed.tech/tags/data.md>), [engine](<https://devfeed.tech/tags/engine.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [open](<https://devfeed.tech/tags/open.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [performance](<https://devfeed.tech/tags/performance.md>), [query](<https://devfeed.tech/tags/query.md>), [s3](<https://devfeed.tech/tags/s3.md>), [s3-tables](<https://devfeed.tech/tags/s3-tables.md>), [tables](<https://devfeed.tech/tags/tables.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

ClickHouse can query and transform Apache Iceberg tables in Amazon S3 Tables without copying the data, then write curated results back as standard Iceberg tables for other compatible engines. It connects through the S3 Tables Iceberg REST endpoint using the DataLakeCatalog engine and AWS IAM credentials; S3 Tables handles storage, metadata, and automatic maintenance, while ClickHouse runs the analytics. The walkthrough requires self-managed ClickHouse 26.8 or later, an AWS account and table bucket, network access, and a least-privilege IAM role. This read-in-place approach suits exploration, batch transforms, and low-concurrency workloads; dashboards needing consistently low latency at high concurrency may require materializing data into MergeTree. The described S3 Tables read/write path is not yet generally available in ClickHouse Cloud. For centralized catalog governance, fine-grained access across engines, or cross-account access, the article points to AWS Glue's Iceberg REST endpoint.

### Source excerpt

Learn how to bring ClickHouse's high-performance engine directly to data in Amazon S3 Tables using bidirectional connectivity. Query your Iceberg tables in place, run transformations in ClickHouse SQL, and write curated results back, while S3 Tables remain the single open source of truth.

## Automate ETL pipeline creation with Amazon Bedrock AgentCore and AWS Glue

DevFeed: [Automate ETL pipeline creation with Amazon Bedrock AgentCore and AWS Glue](<https://devfeed.tech/articles/automate-etl-pipeline-creation-with-amazon-bedrock-agentcore-and-aws-glue-126345.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/automate-etl-pipeline-creation-with-amazon-bedrock-agentcore-and-aws-glue/>)

Author: Mike Brouwers

Published: 2026-11-09T23:15:31Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AI for data automation](<https://devfeed.tech/topics/ai-for-data-automation.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Amazon Bedrock AgentCore](<https://devfeed.tech/topics/amazon-bedrock-agentcore.md>), [Amazon Bedrock](<https://devfeed.tech/topics/amazon-bedrock.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [Amazon Cognito](<https://devfeed.tech/topics/amazon-cognito.md>), [AWS CloudShell](<https://devfeed.tech/topics/aws-cloudshell.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-bedrock-agentcore](<https://devfeed.tech/tags/amazon-bedrock-agentcore.md>), [amazon-eventbridge](<https://devfeed.tech/tags/amazon-eventbridge.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [anthropic-claude](<https://devfeed.tech/tags/anthropic-claude.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [approval-gates](<https://devfeed.tech/tags/approval-gates.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [etl](<https://devfeed.tech/tags/etl.md>), [etl-pipeline](<https://devfeed.tech/tags/etl-pipeline.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Agentic Data Ingestion automates file or database-table ETL setup by proposing schema mappings, generating PySpark transformations, testing them in AWS Glue Spark, and creating and running AWS Glue jobs through a Step Functions workflow. Human approval gates review mappings and tested code; low-confidence mappings receive extra context, and a retry agent may rerun selected failed jobs while skipping cases where retries could duplicate writes or cannot fix the error. Generated code is kept reviewable with target-specific Jinja2 templates, an allowlist for the model-written transformation block, and checks that required job arguments resolve. Tests use a 100-row file sample in a matching Glue interactive session; database tests use schema-generated rows because the session cannot access databases. In the measured US West (Oregon) sample deployment, runs cost about $0.10-0.25 and idle operation about $15-20 monthly; wider sources and lower-cost models were not measured. The solution requires Claude Sonnet 4.6 access and, outside US regions, a model configuration change; production identity federation and role-based access are listed as future hardening steps.

### Source excerpt

Agentic Data Ingestion is an open-source solution that uses AI agents on Amazon Bedrock to automate data pipeline creation, from file upload to queryable tables. Point it at a file or database table, and foundation models propose the field mapping, generate and test the AWS Glue transform, and run the job behind human approval gates.

## Amazon Athena introduces system tables for query execution observability

DevFeed: [Amazon Athena introduces system tables for query execution observability](<https://devfeed.tech/articles/amazon-athena-introduces-system-tables-for-query-execution-observability-126343.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/amazon-athena-introduces-system-tables-for-query-execution-observability/>)

Author: Pathik Shah

Published: 2026-11-09T18:06:44Z

Content type: release

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [observability](<https://devfeed.tech/topics/observability.md>), [AI for SQL query generation](<https://devfeed.tech/topics/ai-for-sql-query-generation.md>), [audit](<https://devfeed.tech/topics/audit.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-skill](<https://devfeed.tech/tags/agent-skill.md>), [agent-toolkit](<https://devfeed.tech/tags/agent-toolkit.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [attribution](<https://devfeed.tech/tags/attribution.md>), [audit](<https://devfeed.tech/tags/audit.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [cost](<https://devfeed.tech/tags/cost.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [observability](<https://devfeed.tech/tags/observability.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [sql](<https://devfeed.tech/tags/sql.md>), [trends](<https://devfeed.tech/tags/trends.md>)

### AI overview

Amazon Athena's new `query_executions` system table records each completed query, including failures, with identity, workgroup, timing, data scanned, DPU count, and outcome details. Enable system tables in the Athena console; records become queryable within minutes as Apache Iceberg data in an S3 Tables bucket in your account. Athena manages table maintenance, while you control encryption, access, and retention; no default retention is set. The table supports cost attribution, performance trends, audit, error analysis, and capacity reservation analysis. Data is read-only for account principals and can also be queried by Iceberg-compatible engines. For natural-language exploration, AWS describes using an MCP-compatible agent with the Athena system tables skill and read-only IAM access; the skill provides schemas and partition-aware query examples.

### Source excerpt

Amazon Athena now offers system tables: SQL-queryable, per-query execution metadata stored as Apache Iceberg in your own account. This post walks through what system tables are, how to turn them on, and how to use them for usage analysis, cost attribution, performance trends, audit, and capacity optimization without custom pipelines.

## AWS Weekly Roundup: AgentCore Runtime V2, Amazon DynamoDB bulk put items, and more (November 9, 2026)

DevFeed: [AWS Weekly Roundup: AgentCore Runtime V2, Amazon DynamoDB bulk put items, and more (November 9, 2026)](<https://devfeed.tech/articles/aws-weekly-roundup-agentcore-runtime-v2-amazon-dynamodb-bulk-put-items-and-more-november-9-2026-126534.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/aws/aws-weekly-roundup-agentcore-runtime-v2-amazon-dynamodb-bulk-put-items-and-more-november-9-2026/>)

Author: Daniel Abib

Published: 2026-11-09T17:55:26Z

Content type: news

Language: en

Sources: [AWS News Blog](<https://devfeed.tech/sources/aws-news-blog.md>)

Topics: [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon Bedrock AgentCore](<https://devfeed.tech/topics/amazon-bedrock-agentcore.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Amazon DynamoDB](<https://devfeed.tech/topics/amazon-dynamodb.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon EC2](<https://devfeed.tech/topics/amazon-ec2.md>), [aws inspector](<https://devfeed.tech/topics/aws-inspector.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [AWS Fargate](<https://devfeed.tech/topics/aws-fargate.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [agentcore](<https://devfeed.tech/tags/agentcore.md>), [agentcore-runtime](<https://devfeed.tech/tags/agentcore-runtime.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-aurora](<https://devfeed.tech/tags/amazon-aurora.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-bedrock-agentcore](<https://devfeed.tech/tags/amazon-bedrock-agentcore.md>), [amazon-dynamodb](<https://devfeed.tech/tags/amazon-dynamodb.md>), [amazon-ec2](<https://devfeed.tech/tags/amazon-ec2.md>), [amazon-elastic-container-registry](<https://devfeed.tech/tags/amazon-elastic-container-registry.md>), [amazon-elastic-container-service](<https://devfeed.tech/tags/amazon-elastic-container-service.md>), [amazon-inspector](<https://devfeed.tech/tags/amazon-inspector.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [amazon-sagemaker-unified-studio](<https://devfeed.tech/tags/amazon-sagemaker-unified-studio.md>), [amazon-simple-notification-service-sns](<https://devfeed.tech/tags/amazon-simple-notification-service-sns.md>), [amazon-simple-storage-service-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-s3.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-direct-connect](<https://devfeed.tech/tags/aws-direct-connect.md>), [aws-fargate](<https://devfeed.tech/tags/aws-fargate.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [aws-security-hub](<https://devfeed.tech/tags/aws-security-hub.md>), [hive](<https://devfeed.tech/tags/hive.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [news](<https://devfeed.tech/tags/news.md>), [s3-tables](<https://devfeed.tech/tags/s3-tables.md>), [sync](<https://devfeed.tech/tags/sync.md>), [week-in-review](<https://devfeed.tech/tags/week-in-review.md>), [weekly](<https://devfeed.tech/tags/weekly.md>), [weekly-roundup](<https://devfeed.tech/tags/weekly-roundup.md>)

### AI overview

AWS highlights several managed updates designed to reduce migration work or improve existing workloads. S3 Tables can migrate Parquet Hive tables from the Glue Data Catalog to Iceberg through start, incremental sync, and cutover; the source remains writable until cutover, with compatibility validation and charges only for copied data. AgentCore Runtime V2 pages memory on demand and restores prepared snapshots; AWS reports faster cold starts in its tests, while noting a higher per-unit rate, several-minute updates, a 120-second `/ping` health requirement, and availability in 10 Regions. DynamoDB bulk put loads S3 files asynchronously into existing tables, supports per-record conditions and an accelerated tier, but cancellation does not undo completed writes. Other updates include AgentCore Web Answer, which returns synthesized cited answers while Web Search returns ranked results for a model to synthesize; Lambda Kafka queue mode, which enables parallel processing beyond partition count but gives up ordering and uses at-least-once delivery; Equinix Interconnect preview; and Security Hub SBOM inventory.

### Source excerpt

Most of the cost of adopting something new isn't the new thing, it's the move. Moving a Hive table to Apache Iceberg has typically meant weeks of engineering work. Last week, Amazon S3 Tables turned that into three steps: start, sync as often as you need, complete. The source stays writable except for a few [...]

## Salesforce to Iceberg: How to Load CRM Objects and Keep Tables Current

DevFeed: [Salesforce to Iceberg: How to Load CRM Objects and Keep Tables Current](<https://devfeed.tech/articles/salesforce-to-iceberg-how-to-load-crm-objects-and-keep-tables-current-121708.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/salesforce-to-iceberg>)

Author: Seán Whelan

Published: 2026-11-06T12:26:52Z

Content type: tutorial

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [etl](<https://devfeed.tech/topics/etl.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [compare](<https://devfeed.tech/tags/compare.md>), [copy](<https://devfeed.tech/tags/copy.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [load](<https://devfeed.tech/tags/load.md>), [salesforce](<https://devfeed.tech/tags/salesforce.md>), [zero-copy](<https://devfeed.tech/tags/zero-copy.md>)

### AI overview

Keeping Salesforce data current in Iceberg requires an initial Bulk API 2.0 load followed by REST `queryAll` sweeps over `SystemModstamp` and `MERGE INTO` operations keyed on `Id`. Sweeps consume the org's shared API allocation; `IsDeleted` records need explicit treatment, purged records are not returned, and formula fields that depend on parent records need scheduled rereads. The document compares a custom Spark/dlt pipeline, AWS Glue zero-ETL for hourly AWS syncs, Snowflake Openflow (Preview), and a managed pipeline; Data 360 zero copy reads existing Iceberg tables rather than copying Salesforce objects into a catalog you control. It describes an Estuary route that polls every five minutes by default and merges on a 30-minute schedule, with daily formula refreshes and EMR Serverless Spark. Its stated limits include unsupported external and big objects, double-typed currency and percent fields, and variant columns requiring Spark 4 unless disabled. Polling and merge frequency trade freshness against API use, compute, and file maintenance.

### Source excerpt

Salesforce to Iceberg: compare 4 ways to load CRM objects into Iceberg tables, see where Data 360 zero copy fits, and handle deletes and formulas.

## Amazon S3 Tables now offer managed migration of Apache Hive tables to Apache Iceberg

DevFeed: [Amazon S3 Tables now offer managed migration of Apache Hive tables to Apache Iceberg](<https://devfeed.tech/articles/amazon-s3-tables-now-offer-managed-migration-of-apache-hive-tables-to-apache-iceberg-118925.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/aws/amazon-s3-tables-now-offer-managed-migration-of-apache-hive-tables-to-apache-iceberg/>)

Author: Channy Yun (윤석찬)

Published: 2026-11-04T23:34:52Z

Content type: release

Language: en

Sources: [AWS News Blog](<https://devfeed.tech/sources/aws-news-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>)

Tags: [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-hive](<https://devfeed.tech/tags/apache-hive.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [launch](<https://devfeed.tech/tags/launch.md>), [migration](<https://devfeed.tech/tags/migration.md>), [news](<https://devfeed.tech/tags/news.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

Amazon S3 Tables can migrate AWS Glue-registered Hive tables stored as Parquet to managed Apache Iceberg tables in the same AWS account and Region. S3 Tables checks compatibility and source objects before copying, then handles type conversions, recompression, validation, and compaction. The destination remains read-only during migration; you can sync changes repeatedly without stopping source writes, then complete cutover, which takes the source table offline for a few minutes. The service charges for data copied, with no Hive-to-Iceberg conversion fee, and leaves S3 Intelligent-Tiering objects in their current tier. The feature is available in the listed US, Asia Pacific, and European Regions.

### Source excerpt

You can now migrate existing Apache Hive datasets in Amazon S3 to fully managed Apache Iceberg tables in Amazon S3 Tables, with no pipelines to build or compute to provision, while production workloads keep running.

## Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg

DevFeed: [Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg](<https://devfeed.tech/articles/migrate-google-bigquery-to-an-open-lakehouse-on-aws-with-apache-iceberg-114202.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/migrate-google-bigquery-to-an-open-lakehouse-on-aws-with-apache-iceberg/>)

Author: Saman Irfan

Published: 2026-11-02T17:21:17Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [AWS Lake Formation](<https://devfeed.tech/topics/aws-lake-formation.md>), [Amazon Bedrock Knowledge Bases](<https://devfeed.tech/topics/amazon-bedrock-knowledge-bases.md>), [AWS Step Functions](<https://devfeed.tech/topics/aws-step-functions.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [data](<https://devfeed.tech/tags/data.md>), [design](<https://devfeed.tech/tags/design.md>), [future](<https://devfeed.tech/tags/future.md>), [google](<https://devfeed.tech/tags/google.md>), [google-bigquery](<https://devfeed.tech/tags/google-bigquery.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [migrate](<https://devfeed.tech/tags/migrate.md>), [migration](<https://devfeed.tech/tags/migration.md>), [open](<https://devfeed.tech/tags/open.md>), [optimize](<https://devfeed.tech/tags/optimize.md>), [over](<https://devfeed.tech/tags/over.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

The article presents a phased path from BigQuery to an AWS lakehouse using Apache Iceberg tables on Amazon S3, with governance through AWS Glue Data Catalog and Lake Formation and engines selected by workload. It recommends assessing tables, query complexity, consumers, volumes, and access controls; designing storage, partitioning, ingestion, and governance; then converting a representative slice and validating results, performance, and cost. For bulk transfer, export BigQuery tables to Parquet in Google Cloud Storage, move them to S3 with AWS DataSync, create or register Iceberg tables, and synchronize incremental changes during transition. Cutover includes parallel validation and deciding whether each consumer will reconnect, be rebuilt, or relay data. The article distinguishes managed S3 Tables, which handle maintenance automatically, from self-managed Iceberg on general-purpose S3, which requires tuning. It also cautions that BigQuery scan-based and Redshift compute-based pricing are not directly comparable, recommending workload-based benchmarks and monthly TCO estimates. Post-migration steps include query and cost optimization.

### Source excerpt

Learn how to migrate Google BigQuery workloads to an open lakehouse on AWS built on Apache Iceberg and Amazon S3. This post explains why an open lakehouse future-proofs your data, presents the target architecture on AWS, and walks through a step-by-step migration: assess, design, convert, migrate, cut over, and optimize.

## Why an AI agent's revenue query was $3.5 million wrong, and how AWS fixed it with dbt Semantic Layer

DevFeed: [Why an AI agent's revenue query was $3.5 million wrong, and how AWS fixed it with dbt Semantic Layer](<https://devfeed.tech/articles/why-an-ai-agent-s-revenue-query-was-3-5-million-wrong-and-how-aws-fixed-it-with-dbt-semantic-layer-102636.md>)

Original publisher: [Read original article](<https://www.getdbt.com/blog/aws-dbt-semantic-layer-ai-agent-revenue-accuracy>)

Author: Daniel Poppy

Published: 2026-10-27T18:41:00Z

Content type: news

Language: en

Sources: [dbt Labs](<https://devfeed.tech/sources/dbt-labs.md>)

Topics: [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-revenue](<https://devfeed.tech/tags/ai-revenue.md>), [aws](<https://devfeed.tech/tags/aws.md>), [business](<https://devfeed.tech/tags/business.md>), [business-context](<https://devfeed.tech/tags/business-context.md>), [context](<https://devfeed.tech/tags/context.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [dbt-semantic-layer](<https://devfeed.tech/tags/dbt-semantic-layer.md>), [errors](<https://devfeed.tech/tags/errors.md>), [layer](<https://devfeed.tech/tags/layer.md>), [learn](<https://devfeed.tech/tags/learn.md>), [prevent](<https://devfeed.tech/tags/prevent.md>), [query](<https://devfeed.tech/tags/query.md>), [revenue](<https://devfeed.tech/tags/revenue.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [semantic-layer](<https://devfeed.tech/tags/semantic-layer.md>)

### AI overview

A raw SQL query overstated revenue by 30.8%: it summed pre-discount, pre-refund amounts and included refunds incorrectly, cancelled and pending orders, and internal test accounts. The AWS presenters' proposed fix separates metric governance from query execution. The dbt Semantic Layer resolves questions against certified metrics and dimensions, while MetricFlow compiles the SQL; the Redshift MCP Server executes it as read-only, single-statement queries, with IAM and database permissions adding safeguards. Responses include the metric, SQL, source model, dbt test results, and Redshift query ID. The presenters said the same metric definitions worked across Redshift, a Redshift and Iceberg query, and dbt's Athena and DuckDB adapters without Redshift. Developers could review SQL before execution, while business users received cited plain-language answers.

### Source excerpt

A plausible AI revenue query was $3.5 million wrong. AWS explains how business context helps prevent those errors.

## Data governance with annotations in Amazon S3

DevFeed: [Data governance with annotations in Amazon S3](<https://devfeed.tech/articles/data-governance-with-annotations-in-amazon-s3-101933.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/data-governance-with-annotations-in-amazon-s3/>)

Author: Aditya Jain

Published: 2026-10-26T18:00:26Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Authorization](<https://devfeed.tech/topics/authorization.md>), [pii](<https://devfeed.tech/topics/pii.md>), [ai-governance](<https://devfeed.tech/topics/ai-governance.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-eventbridge](<https://devfeed.tech/tags/amazon-eventbridge.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [amazon-simple-storage-service-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-s3.md>), [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [annotations](<https://devfeed.tech/tags/annotations.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [api](<https://devfeed.tech/tags/api.md>), [apis](<https://devfeed.tech/tags/apis.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [athena](<https://devfeed.tech/tags/athena.md>), [audit](<https://devfeed.tech/tags/audit.md>), [audit-trail](<https://devfeed.tech/tags/audit-trail.md>), [classification](<https://devfeed.tech/tags/classification.md>), [data-governance](<https://devfeed.tech/tags/data-governance.md>), [governance](<https://devfeed.tech/tags/governance.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [pii](<https://devfeed.tech/tags/pii.md>), [retention](<https://devfeed.tech/tags/retention.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Amazon S3 annotations keep structured governance context on a specific object version, where it can be updated without rewriting the object and read through annotation APIs or queried at scale with an S3 Metadata Iceberg table and Athena. The pattern uses an enrichment process for new uploads, scheduled updates with an ETag condition, and S3 Batch Operations for backfill; generated labels need validation before they drive consequential actions. Key constraints shape deployment: in versioned buckets, ordinary deletes leave annotations on prior versions; Object Lock blocks annotation changes during retention; multipart copies require an opt-in to preserve annotations; annotation overwrites and deletes have no history; and SSE-C objects are unsupported. Separate read, write, and delete permissions, configure annotation replication explicitly, and account for asynchronous table refresh and annotation replication, as well as storage and request costs. Annotations suit rich, mutable context, but not relational queries, policy or lifecycle filtering on annotation values, or very high write rates to one object.

### Source excerpt

Attach governance context such as classification, retention, residency, and PII flags directly to your Amazon S3 objects with S3 annotations. This post walks through the end-to-end pattern and the handful of S3 behaviors that decide whether your governance design is correct.

## How Seltz cut data costs 40% with Amazon S3 Tables

DevFeed: [How Seltz cut data costs 40% with Amazon S3 Tables](<https://devfeed.tech/articles/how-seltz-cut-data-costs-40-with-amazon-s3-tables-96261.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/storage/how-seltz-cut-data-costs-40-with-amazon-s3-tables/>)

Author: Vivien de Saint Pern

Published: 2026-10-22T19:47:31Z

Content type: article

Language: en

Sources: [AWS Storage Blog](<https://devfeed.tech/sources/aws-storage-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>)

Tags: [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [automatic](<https://devfeed.tech/tags/automatic.md>), [compaction](<https://devfeed.tech/tags/compaction.md>), [cost](<https://devfeed.tech/tags/cost.md>), [costs](<https://devfeed.tech/tags/costs.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [cut-data](<https://devfeed.tech/tags/cut-data.md>), [data](<https://devfeed.tech/tags/data.md>), [data-costs](<https://devfeed.tech/tags/data-costs.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [incremental](<https://devfeed.tech/tags/incremental.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [s3](<https://devfeed.tech/tags/s3.md>), [s3-tables](<https://devfeed.tech/tags/s3-tables.md>), [tables](<https://devfeed.tech/tags/tables.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Seltz reports 40% lower infrastructure cost with Amazon S3 Tables than with its self-managed pipeline, based on cost modeling at its current scale; the figure excludes engineering effort. Its Knowledge Pipeline writes each enrichment stage's intermediate results to a separate Iceberg table, and a downstream indexer builds search index files. S3 Tables automatically compacts small writes and removes expired or unreferenced data, while atomic commits, independent table access, and file-level pruning support incremental processing. Seltz uses Athena for SQL monitoring and DuckDB for local inspection. The article also outlines a walkthrough using PyIceberg and Athena, requiring an AWS account with table-bucket permissions, Python 3.9+, and configured Athena.

### Source excerpt

Learn how Seltz uses Amazon S3 Tables as managed Apache Iceberg storage between the stages of its billion-document Knowledge Pipeline. The team gets automatic compaction, lifecycle management, and incremental table semantics without operating that infrastructure, cutting infrastructure cost 40 percent compared to a self-managed pipeline.

## Amazon S3 is now available on the Vercel Marketplace and v0 by Vercel

DevFeed: [Amazon S3 is now available on the Vercel Marketplace and v0 by Vercel](<https://devfeed.tech/articles/amazon-s3-is-now-available-on-the-vercel-marketplace-and-v0-by-vercel-87251.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/aws/amazon-s3-vercel-marketplace-v0/>)

Author: Daniel Abib

Published: 2026-10-14T19:30:48Z

Content type: release

Language: en

Sources: [AWS News Blog](<https://devfeed.tech/sources/aws-news-blog.md>)

Topics: [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Vercel](<https://devfeed.tech/topics/vercel.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [amazon-simple-storage-service-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-s3.md>), [marketplace](<https://devfeed.tech/tags/marketplace.md>), [vectors](<https://devfeed.tech/tags/vectors.md>), [vercel](<https://devfeed.tech/tags/vercel.md>)

### AI overview

Amazon S3, S3 Tables, and S3 Vectors are now available through the Vercel Marketplace and v0, letting developers provision storage resources within Vercel. The article describes setup, temporary AWS credentials through OIDC, and code examples for object storage, Iceberg analytics, and vector search. v0 can also provision resources and generate application code from natural-language prompts.

### Source excerpt

Amazon Simple Storage Service (Amazon S3), including Amazon S3 Tables and Amazon S3 Vectors, is natively available on the Vercel Marketplace and v0 by Vercel.

## Resolving concurrent write conflicts for Apache Iceberg tables in Amazon Redshift

DevFeed: [Resolving concurrent write conflicts for Apache Iceberg tables in Amazon Redshift](<https://devfeed.tech/articles/resolving-concurrent-write-conflicts-for-apache-iceberg-tables-in-amazon-redshift-86417.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/resolving-concurrent-write-conflicts-for-apache-iceberg-tables-in-amazon-redshift/>)

Author: Sanjay Devireddy

Published: 2026-10-13T20:16:48Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [database-consistency](<https://devfeed.tech/topics/database-consistency.md>)

Tags: [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [concurrency-control](<https://devfeed.tech/tags/concurrency-control.md>), [conflict-resolution](<https://devfeed.tech/tags/conflict-resolution.md>), [foundational-100](<https://devfeed.tech/tags/foundational-100.md>)

### AI overview

Amazon Redshift now automatically rebases many concurrent writes to Apache Iceberg tables when metadata shows that the changes do not overlap. The article explains how Redshift checks conflicts at operation, partition, and file levels, and describes benchmark results for append-only writers. In the reported 16-writer test, automatic resolution completed 12 times more commits in the same period than basic optimistic concurrency mode, with no rejected commits in the measured configurations.

### Source excerpt

Amazon Redshift now automatically resolves many concurrent writes to Apache Iceberg tables that don't actually overlap, committing changes that previously failed and had to retry. Learn why these writes conflict, how the engine rebases non-overlapping ones across operation, partition, and file checks, and the throughput gains it delivers.

## Agentic antimicrobial resistance surveillance with AWS HealthOmics and Amazon Bedrock AgentCore

DevFeed: [Agentic antimicrobial resistance surveillance with AWS HealthOmics and Amazon Bedrock AgentCore](<https://devfeed.tech/articles/agentic-antimicrobial-resistance-surveillance-with-aws-healthomics-and-amazon-bedrock-agentcore-85601.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/agentic-antimicrobial-resistance-surveillance-with-aws-healthomics-and-amazon-bedrock-agentcore/>)

Author: Nicolo Cosimo Albanese

Published: 2026-10-12T14:34:57Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Amazon Bedrock AgentCore](<https://devfeed.tech/topics/amazon-bedrock-agentcore.md>), [AWS IoT Core](<https://devfeed.tech/topics/aws-iot-core.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon DynamoDB](<https://devfeed.tech/topics/amazon-dynamodb.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [Amazon EC2](<https://devfeed.tech/topics/amazon-ec2.md>)

Tags: [amazon-bedrock-agentcore](<https://devfeed.tech/tags/amazon-bedrock-agentcore.md>), [amazon-dynamodb](<https://devfeed.tech/tags/amazon-dynamodb.md>), [amazon-eventbridge](<https://devfeed.tech/tags/amazon-eventbridge.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-fargate](<https://devfeed.tech/tags/aws-fargate.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-healthomics](<https://devfeed.tech/tags/aws-healthomics.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [bedrock](<https://devfeed.tech/tags/bedrock.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [it](<https://devfeed.tech/tags/it.md>), [language](<https://devfeed.tech/tags/language.md>), [processing](<https://devfeed.tech/tags/processing.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

This reference architecture processes public sequencing data with AWS HealthOmics, harmonizes and stores resistance gene results, and uses an Amazon Bedrock AgentCore agent to answer researchers' questions in plain language. The post describes its ingestion, genomics processing, storage, and conversational query components, and notes that it is a demonstration rather than a clinical system.

### Source excerpt

This post presents a reference architecture that turns public sequencing data into plain-language answers about antimicrobial resistance (AMR) genes. It pairs AWS HealthOmics for managed genomics processing with a grounded Amazon Bedrock AgentCore agent, so researchers can ask questions without writing bioinformatics code or SQL.

## Cost-effective ETL with DuckDB and Amazon S3 Tables on AWS Glue

DevFeed: [Cost-effective ETL with DuckDB and Amazon S3 Tables on AWS Glue](<https://devfeed.tech/articles/cost-effective-etl-with-duckdb-and-amazon-s3-tables-on-aws-glue-65454.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/cost-effective-etl-with-duckdb-and-amazon-s3-tables-on-aws-glue/>)

Author: Bezuayehu Wate

Published: 2026-10-06T16:13:02Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [parquet](<https://devfeed.tech/topics/parquet.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The tutorial explains how to run DuckDB in AWS Glue 6.0 to read Parquet from Amazon S3, transform data with SQL, and write Iceberg tables to Amazon S3 Tables. It covers configuration, permissions, worker sizing, monitoring, and recovery from partial writes. For the demonstrated 8.4-million-row workload, DuckDB took approximately half the runtime and cost of an equivalent Spark job on the same Glue configuration; distributed workloads remain a use case for Spark.

### Source excerpt

Learn how to pair DuckDB with AWS Glue 6.0 to run SQL-centric ETL on a single worker, reading Parquet from Amazon S3 and writing Apache Iceberg tables to Amazon S3 Tables. This post walks through a complete, runnable example and compares measured cost and runtime against an equivalent Apache Spark job on the same Glue runtime.

## Query across accounts and table formats with multi-catalog in Amazon EMR

DevFeed: [Query across accounts and table formats with multi-catalog in Amazon EMR](<https://devfeed.tech/articles/query-across-accounts-and-table-formats-with-multi-catalog-in-amazon-emr-65456.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/query-across-accounts-and-table-formats-with-multi-catalog-in-amazon-emr/>)

Author: Suthan Phillips

Published: 2026-10-06T16:05:27Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>)

Tags: [access-control](<https://devfeed.tech/tags/access-control.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-emr](<https://devfeed.tech/tags/amazon-emr.md>), [amazon-emr-on-eks](<https://devfeed.tech/tags/amazon-emr-on-eks.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-hudi](<https://devfeed.tech/tags/apache-hudi.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [api](<https://devfeed.tech/tags/api.md>), [applications](<https://devfeed.tech/tags/applications.md>), [aws-emr](<https://devfeed.tech/tags/aws-emr.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-key-management-service](<https://devfeed.tech/tags/aws-key-management-service.md>), [aws-lake-formation](<https://devfeed.tech/tags/aws-lake-formation.md>), [data-catalog](<https://devfeed.tech/tags/data-catalog.md>), [data-mesh](<https://devfeed.tech/tags/data-mesh.md>), [database](<https://devfeed.tech/tags/database.md>), [databases](<https://devfeed.tech/tags/databases.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>)

### AI overview

Amazon EMR 8.1.0 adds multi-catalog support that routes Iceberg, Delta Lake, Hudi, and Hive tables through one catalog and enables queries joining data across AWS accounts without copying it. The tutorial demonstrates configuring the RedirectingSessionCatalog and optionally using Glue catalog resolution for discovering remote catalogs at query time.

### Source excerpt

Amazon EMR 8.1.0 adds multi-catalog support through the RedirectingSessionCatalog, so you can query Iceberg, Delta Lake, Hudi, and Hive tables through a single catalog and join data across AWS accounts without copying it. This post shows how to put these capabilities into practice on Amazon EMR Serverless.

[Next page](<https://devfeed.tech/topics/amazon-s3-tables.md?cursor=WyIyMDI2LTEwLTA2VDE2OjA1OjI3KzAwOjAwIiwgIjlmY2Y0MmQzLWFhYzgtNDQ0OS1hMDQzLTQ0YzY2MjAxYzBkZSJd>)