# AWS Big Data Blog

Official Big Data Blog of Amazon Web Services

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Introducing Amazon EMR Serverless Flex: Run non-urgent Spark jobs at a discount

DevFeed: [Introducing Amazon EMR Serverless Flex: Run non-urgent Spark jobs at a discount](<https://devfeed.tech/articles/introducing-amazon-emr-serverless-flex-run-non-urgent-spark-jobs-at-a-discount-144577.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/introducing-amazon-emr-serverless-flex-run-non-urgent-spark-jobs-at-a-discount/>)

Author: Neil Mukerje

Published: 2026-11-19T20:58:07Z

Content type: release

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [Amazon EC2](<https://devfeed.tech/topics/amazon-ec2.md>)

Tags: [amazon-emr](<https://devfeed.tech/tags/amazon-emr.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [batch](<https://devfeed.tech/tags/batch.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [billing](<https://devfeed.tech/tags/billing.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [discount](<https://devfeed.tech/tags/discount.md>), [driver](<https://devfeed.tech/tags/driver.md>), [flex](<https://devfeed.tech/tags/flex.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [pricing](<https://devfeed.tech/tags/pricing.md>), [retries](<https://devfeed.tech/tags/retries.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [spark](<https://devfeed.tech/tags/spark.md>), [standard](<https://devfeed.tech/tags/standard.md>)

### AI overview

Amazon EMR Serverless Flex runs non-urgent Spark batch jobs on spare capacity at a fixed 60% discount on worker vCPU and memory charges compared with Standard. It automates Availability Zone and instance selection, protects the driver on a Standard worker, and retries tasks when Flex workers are reclaimed; optional Serverless Storage can reduce recomputation for shuffle-heavy jobs. Flex may start more slowly and run longer than Standard, and a job fails if no executor joins the driver within 20 minutes. In the described 103-query benchmark, all queries completed without resubmission; runtime was 7% longer at 1 TB and 30% longer at 3 TB, while estimated cost fell about 63% at both scales. Results vary with workload and spare capacity. Flex is enabled per job, so teams can compare it with Standard on flexible-deadline workloads.

### Source excerpt

Amazon EMR Serverless Flex is a new execution option that runs non-urgent Spark batch jobs on spare compute capacity at a fixed 60% discount compared to standard pricing. Flex handles Availability Zone selection, instance diversification, driver protection, and automatic retries, so jobs complete even when spare capacity is reclaimed.

## Amazon S3 Tables with dbt: Iceberg-based data transformations using AWS Glue

DevFeed: [Amazon S3 Tables with dbt: Iceberg-based data transformations using AWS Glue](<https://devfeed.tech/articles/amazon-s3-tables-with-dbt-iceberg-based-data-transformations-using-aws-glue-144273.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/amazon-s3-tables-with-dbt-iceberg-based-data-transformations-using-aws-glue/>)

Author: Yukiko Iwazumi

Published: 2026-11-19T17:58:50Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-quick-sight](<https://devfeed.tech/tags/amazon-quick-sight.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The walkthrough builds a dbt pipeline on Amazon S3 Tables through AWS Glue interactive sessions, with raw CSV data staged and transformed into analytics marts. It requires dbt-glue 1.12.4 or later for S3 Tables incremental models, Python 3.10-3.12, and AWS Glue 6.0 or later. In the example, `fact_orders` uses merge to retain each order's latest state, while `fact_order_history` uses append to retain changes. After source updates, the merge table has six orders, including the updated order 1004 and new order 1006; the history table grows from five to seven rows. The post describes S3 Tables support as experimental in dbt-glue 1.12.4 and recommends testing it in your environment.

### Source excerpt

Amazon S3 Tables, a capability of Amazon S3, bring built-in Apache Iceberg support to object storage. This post shows how to configure and run dbt transformations on S3 Tables with the AWS Glue dbt-glue adapter, using incremental merge and append strategies across a staging and mart architecture.

## Introducing automatic version upgrades for AWS Glue jobs

DevFeed: [Introducing automatic version upgrades for AWS Glue jobs](<https://devfeed.tech/articles/introducing-automatic-version-upgrades-for-aws-glue-jobs-141879.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/introducing-automatic-version-upgrades-for-aws-glue-jobs/>)

Author: Keerthi Chadalavada

Published: 2026-11-19T00:45:54Z

Content type: release

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [version](<https://devfeed.tech/topics/version.md>), [Amazon Bedrock](<https://devfeed.tech/topics/amazon-bedrock.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [automatic](<https://devfeed.tech/tags/automatic.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [changes](<https://devfeed.tech/tags/changes.md>), [fleet](<https://devfeed.tech/tags/fleet.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [job](<https://devfeed.tech/tags/job.md>), [jobs](<https://devfeed.tech/tags/jobs.md>), [latest](<https://devfeed.tech/tags/latest.md>), [learn](<https://devfeed.tech/tags/learn.md>), [set](<https://devfeed.tech/tags/set.md>), [stable](<https://devfeed.tech/tags/stable.md>), [the-latest](<https://devfeed.tech/tags/the-latest.md>), [upgrade](<https://devfeed.tech/tags/upgrade.md>), [upgrades](<https://devfeed.tech/tags/upgrades.md>), [version](<https://devfeed.tech/tags/version.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

AWS Glue can automatically assess and upgrade eligible Spark ETL jobs when their `GlueVersion` is set to `auto`. Jobs continue running on their current version while Glue checks for known breaking changes; if no code changes are needed, it sends an eligibility summary for human approval. After approval, Glue validates upgraded runs for correctness, performance, and cost against earlier runs. A failed upgraded run is retried on the previous version and the job reverts. Jobs must use Glue 2.0 or later, have `MaxRetries` of at least 1, have job run queuing off, and be safe to retry. The console supports approval job by job; an AI agent with the managing-glue-jobs skill can assess and manage a fleet, with confirmation for changes and optional email notifications. Jobs requiring code changes need a separate generative AI upgrade first. The article says auto upgrade is available in commercial AWS Regions; region details are in the Glue User Guide.

### Source excerpt

Today we're announcing automatic version upgrades for AWS Glue jobs. Set a job's version to auto, and AWS Glue checks for breaking changes, asks for your approval, then moves the job to the latest stable version and validates the upgraded runs. Learn how auto upgrade works and how to use it across a fleet of jobs with an AI agent.

## Long-term Amazon Redshift observability with Datadog and Amazon S3 Tables

DevFeed: [Long-term Amazon Redshift observability with Datadog and Amazon S3 Tables](<https://devfeed.tech/articles/long-term-amazon-redshift-observability-with-datadog-and-amazon-s3-tables-141880.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/long-term-amazon-redshift-observability-with-datadog-and-amazon-s3-tables/>)

Author: Raghu Kuppala

Published: 2026-11-18T22:57:27Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [datadog](<https://devfeed.tech/topics/datadog.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [elt](<https://devfeed.tech/topics/elt.md>)

Tags: [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-emr](<https://devfeed.tech/tags/amazon-emr.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [aws-cli](<https://devfeed.tech/tags/aws-cli.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-identity-and-access-management-iam](<https://devfeed.tech/tags/aws-identity-and-access-management-iam.md>), [aws-key-management-service-aws-kms](<https://devfeed.tech/tags/aws-key-management-service-aws-kms.md>), [data-observability](<https://devfeed.tech/tags/data-observability.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [history](<https://devfeed.tech/tags/history.md>), [integration](<https://devfeed.tech/tags/integration.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [observability](<https://devfeed.tech/tags/observability.md>), [performance](<https://devfeed.tech/tags/performance.md>), [query](<https://devfeed.tech/tags/query.md>)

### AI overview

Amazon Redshift system tables retain history for only 7 days. The S3 Tables integration continuously exports selected system table data as Apache Iceberg tables, letting customers set retention and query the history without collection queries consuming warehouse compute. Datadog Data Observability can read that history to build column-level lineage, normalized query profiles, and context for quality alerts across connected clusters and workgroups. It still connects to Redshift for metadata and runs configured aggregate quality monitors there. Setup requires enabling per-warehouse exports, integrating S3 Tables with the AWS Glue Data Catalog, and granting the Datadog IAM role read access; the feature is described as a preview requiring account-team access. Disabling exports creates a history gap that cannot be backfilled after Redshift's 7-day window, and customer-managed KMS encryption keys cannot be changed after table creation.

### Source excerpt

Amazon Redshift keeps only 7 days of system table history. This post pairs Datadog Data Observability with the new Amazon Redshift system table integration with Amazon S3 Tables, so you can retain query history for the long term and analyze lineage, query performance, and data quality without adding load to your warehouses.

## Orchestrate DuckDB analytics pipelines with Amazon MWAA

DevFeed: [Orchestrate DuckDB analytics pipelines with Amazon MWAA](<https://devfeed.tech/articles/orchestrate-duckdb-analytics-pipelines-with-amazon-mwaa-141881.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/orchestrate-duckdb-analytics-pipelines-with-amazon-mwaa/>)

Author: Vinod Jayendra

Published: 2026-11-18T20:57:09Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [parquet](<https://devfeed.tech/topics/parquet.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-managed-workflows-for-apache-airflow-amazon-mwaa](<https://devfeed.tech/tags/amazon-managed-workflows-for-apache-airflow-amazon-mwaa.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [data](<https://devfeed.tech/tags/data.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [effective](<https://devfeed.tech/tags/effective.md>), [embedded](<https://devfeed.tech/tags/embedded.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [learn-how-to](<https://devfeed.tech/tags/learn-how-to.md>), [model](<https://devfeed.tech/tags/model.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [retry](<https://devfeed.tech/tags/retry.md>), [s3](<https://devfeed.tech/tags/s3.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Amazon MWAA can orchestrate bounded SQL analytics by running DuckDB inside Airflow worker tasks, without a separate query cluster. The walkthrough reads S3 Parquet files, deduplicates events, uses ASOF LEFT JOINs to apply rates and ownership effective at each event time, and merges results into three Iceberg tables in Amazon S3 Tables. A second task validates counts, uniqueness, and allocation states. On supported MWAA Provisioned environments running Airflow 3.3.1, DuckDB and its providers plus the duckdb_default AWS-aware connection are preinstalled; the execution role supplies temporary credentials. The example's unchanged-input reruns retained 300,000 unique events and duplicate-free summaries, but the excerpt says this does not establish recovery from partial failures or concurrent external writes. Each task shares one worker's CPU, memory, and storage, so limits and concurrency need tuning with representative data. Use Glue or EMR for steps needing horizontal scaling or more capacity than one worker provides. Three table commits are not one transaction, and Iceberg update/delete support can vary by version.

### Source excerpt

Learn how to orchestrate embedded DuckDB analytics pipelines with Amazon MWAA. This post shows how to run DuckDB inside Airflow tasks to read Parquet data from Amazon S3, apply time-effective model rates, and publish Apache Iceberg tables to Amazon S3 Tables, with retry-safe publishing and worker-sizing considerations.

## Introducing Amazon OpenSearch Memory: Persistent memory for AI agents

DevFeed: [Introducing Amazon OpenSearch Memory: Persistent memory for AI agents](<https://devfeed.tech/articles/introducing-amazon-opensearch-memory-persistent-memory-for-ai-agents-141878.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/introducing-amazon-opensearch-memory-persistent-memory-for-ai-agents/>)

Author: Ganesh Natarajan

Published: 2026-11-18T20:50:32Z

Content type: release

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [opensearch](<https://devfeed.tech/topics/opensearch.md>), [Amazon OpenSearch Service](<https://devfeed.tech/topics/amazon-opensearch-service.md>), [Amazon Bedrock](<https://devfeed.tech/topics/amazon-bedrock.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-opensearch-serverless](<https://devfeed.tech/tags/amazon-opensearch-serverless.md>), [amazon-opensearch-service](<https://devfeed.tech/tags/amazon-opensearch-service.md>), [amazon-titan](<https://devfeed.tech/tags/amazon-titan.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [api](<https://devfeed.tech/tags/api.md>), [aws-sdk-for-python](<https://devfeed.tech/tags/aws-sdk-for-python.md>), [context](<https://devfeed.tech/tags/context.md>), [cross-session-memory](<https://devfeed.tech/tags/cross-session-memory.md>), [events](<https://devfeed.tech/tags/events.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [memory](<https://devfeed.tech/tags/memory.md>), [opensearch](<https://devfeed.tech/tags/opensearch.md>), [persistent-memory](<https://devfeed.tech/tags/persistent-memory.md>), [persistent-memory-for-ai](<https://devfeed.tech/tags/persistent-memory-for-ai.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [session](<https://devfeed.tech/tags/session.md>)

### AI overview

Amazon OpenSearch Memory adds persistent, cross-session memory to AI agents through serverless spaces containing verbatim events and memories derived from them. Applications write each turn and retrieve relevant memories or exact events before responding; event reads are immediate and searchability follows within seconds, while memory extraction runs asynchronously and may return no results for a new user at first. The service handles indexing, consolidation, retrieval, isolation, encryption, and retention, with actor and optional session controls; applications still add retrieved context to prompts and should review agent-written memories. In one LongMemEval-S run, the configured workflow used about 13,500 input tokens per question versus 167,000 for full history and scored 93.0% accuracy versus 95.2%; it beat BM25 raw-turn retrieval by 4.4 points. Results depend on the tested models and settings. General availability is in four AWS Regions.

### Source excerpt

Amazon OpenSearch Memory is now generally available. Built on Amazon OpenSearch Serverless, it adds persistent, cross-session memory to AI agents. It stores interactions as searchable events, derives memories, and retrieves relevant context, so you can give agents long-term memory without building your own memory system.

## Reduce the complexity and cost of real-time processing with queues in Amazon Kinesis Data Streams

DevFeed: [Reduce the complexity and cost of real-time processing with queues in Amazon Kinesis Data Streams](<https://devfeed.tech/articles/reduce-the-complexity-and-cost-of-real-time-processing-with-queues-in-amazon-kinesis-data-streams-140167.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/reduce-the-complexity-and-cost-of-real-time-processing-with-queues-in-amazon-kinesis-data-streams/>)

Author: Dirk Fröhner

Published: 2026-11-17T23:17:16Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-kinesis](<https://devfeed.tech/tags/amazon-kinesis.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [backlog](<https://devfeed.tech/tags/backlog.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [demand](<https://devfeed.tech/tags/demand.md>), [instead](<https://devfeed.tech/tags/instead.md>), [kinesis](<https://devfeed.tech/tags/kinesis.md>), [kinesis-data-streams](<https://devfeed.tech/tags/kinesis-data-streams.md>), [offers](<https://devfeed.tech/tags/offers.md>), [one](<https://devfeed.tech/tags/one.md>), [processing](<https://devfeed.tech/tags/processing.md>), [queues](<https://devfeed.tech/tags/queues.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [reduce](<https://devfeed.tech/tags/reduce.md>), [streams](<https://devfeed.tech/tags/streams.md>), [time](<https://devfeed.tech/tags/time.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Amazon Kinesis Data Streams queues provide per-record delivery tracking for applications that can process records independently. Consumers receive batches of up to 10 records, process them, and acknowledge them; unacknowledged records become available again after a configurable visibility timeout. Delivery is at least once and ordering is best effort, so processing should be idempotent. Records that exceed the default six receives without acknowledgment go to a dead-letter state, where operators can inspect, restore, or discard them. Queues scale consumer instances with backlog rather than shard count and remove the need for a KCL lease table and per-shard custom metrics. Multiple queues, stream consumers, and managed delivery can use the same stream without a relay or duplicate ingest. Use direct stream consumers when processing depends on record order or prior state; use queues for independent work. AWS reports up to 43% lower processing cost, but its synthetic benchmark used an I/O-bound handler; savings depend on workload and configuration, and CPU-bound work narrows the compute benefit.

### Source excerpt

Amazon Kinesis Data Streams now offers queues, a new way to consume the records of an on-demand stream. A queue delivers records one by one, tracks each record's delivery state, redelivers unacknowledged records, sets persistently failing records aside as dead letters, and scales with your backlog instead of your shard count.

## Amazon MSK Express brokers available on AWS Graviton4-based M8g instances

DevFeed: [Amazon MSK Express brokers available on AWS Graviton4-based M8g instances](<https://devfeed.tech/articles/amazon-msk-express-brokers-available-on-aws-graviton4-based-m8g-instances-140165.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/amazon-msk-express-brokers-available-on-aws-graviton4-based-m8g-instances/>)

Author: Charishma Makineni

Published: 2026-11-17T22:07:48Z

Content type: release

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Graviton](<https://devfeed.tech/topics/graviton.md>), [Amazon EC2](<https://devfeed.tech/topics/amazon-ec2.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-managed-streaming-for-apache-kafka-amazon-msk](<https://devfeed.tech/tags/amazon-managed-streaming-for-apache-kafka-amazon-msk.md>), [amazon-msk](<https://devfeed.tech/tags/amazon-msk.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [aws](<https://devfeed.tech/tags/aws.md>), [based](<https://devfeed.tech/tags/based.md>), [broker](<https://devfeed.tech/tags/broker.md>), [client](<https://devfeed.tech/tags/client.md>), [consumer](<https://devfeed.tech/tags/consumer.md>), [cross](<https://devfeed.tech/tags/cross.md>), [express](<https://devfeed.tech/tags/express.md>), [faster](<https://devfeed.tech/tags/faster.md>), [foundational-100](<https://devfeed.tech/tags/foundational-100.md>), [graviton](<https://devfeed.tech/tags/graviton.md>), [graviton4](<https://devfeed.tech/tags/graviton4.md>), [latency](<https://devfeed.tech/tags/latency.md>), [p99](<https://devfeed.tech/tags/p99.md>), [pricing](<https://devfeed.tech/tags/pricing.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [size](<https://devfeed.tech/tags/size.md>), [traffic](<https://devfeed.tech/tags/traffic.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Amazon MSK Express brokers on Graviton4-based M8g instances combine lower costs with better results than M7g in the described tests: broker-hour pricing is 15% lower, broker-side cross-AZ client traffic is uncharged (cutting total cross-AZ transfer cost by 50%), and P99 produce latency was 21-54% lower at the same tested load. Brokers used 9-11 percentage points less CPU, and consumers cleared up to three times more backlog in 30 minutes. Results depend on workload settings; the catch-up test used large brokers only. M8g adds a 24xlarge option, while shared sizes retain the same per-broker storage and network throughput. Teams can create an M8g cluster, update an M7g Express cluster in place one broker at a time, or migrate from Standard or self-managed Kafka using MSK Replicator. Rack-aware consumers can reduce cross-AZ charges further, though reading from follower replicas may add some end-to-end latency.

### Source excerpt

Amazon MSK Express brokers are now available on AWS Graviton4-based M8g instances. Compared with Express on M7g, M8g delivers 15% lower broker-hour pricing, lower P99 produce latency, faster consumer recovery, a new 24xlarge size, and no broker-side charge for cross-AZ client traffic.

## Deliver petabytes of Apache Kafka data to Amazon S3 at up to 60% lower cost with Amazon MSK Express brokers

DevFeed: [Deliver petabytes of Apache Kafka data to Amazon S3 at up to 60% lower cost with Amazon MSK Express brokers](<https://devfeed.tech/articles/deliver-petabytes-of-apache-kafka-data-to-amazon-s3-at-up-to-60-lower-cost-with-amazon-msk-express-brokers-136179.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/deliver-petabytes-of-apache-kafka-data-to-amazon-s3-at-up-to-60-lower-cost-with-amazon-msk-express-brokers/>)

Author: Diksha Dubey

Published: 2026-11-16T20:42:44Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>)

Tags: [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-managed-streaming-for-apache-kafka-amazon-msk](<https://devfeed.tech/tags/amazon-managed-streaming-for-apache-kafka-amazon-msk.md>), [amazon-msk](<https://devfeed.tech/tags/amazon-msk.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-simple-storage-service-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-s3.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [channel](<https://devfeed.tech/tags/channel.md>), [connect](<https://devfeed.tech/tags/connect.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [delivered](<https://devfeed.tech/tags/delivered.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [express](<https://devfeed.tech/tags/express.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [one](<https://devfeed.tech/tags/one.md>), [post](<https://devfeed.tech/tags/post.md>), [run](<https://devfeed.tech/tags/run.md>), [s3](<https://devfeed.tech/tags/s3.md>), [up](<https://devfeed.tech/tags/up.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Amazon MSK Express brokers deliver Kafka topic records to general purpose Amazon S3 buckets through a channel, without Kafka Connect. The article says delivery is billed per byte, adds no broker egress load, and can cost up to 60% less than self-managed alternatives. Brokers automatically scale storage and rebalance partitions; MSK manages recovery and updates. Channels batch records into S3 objects, preserving producer formats, with configurable compression, storage class, and object key layout. JSON and String records are validated; failures go to an S3 dead-letter queue while delivery continues. The article also describes CloudWatch metrics and logs, setup through the console, APIs, CLI, or an Agent Skill, and recommends overlapping a new channel with an existing connector until delivered data is confirmed. Review supported configurations and prerequisites before production use.

### Source excerpt

Amazon MSK Express brokers now deliver Apache Kafka topic data to Amazon S3 at up to 60% lower cost, with no Kafka Connect to run. You create a channel on a topic and pay only for the data delivered. In this post, we cover the two challenges of delivering Kafka data to Amazon S3 today and how Express brokers and data delivery to Amazon S3 address each one.

## Analyze Anthropic usage, spend, and access with AWS Glue, Amazon S3 Tables, and AWS Quick

DevFeed: [Analyze Anthropic usage, spend, and access with AWS Glue, Amazon S3 Tables, and AWS Quick](<https://devfeed.tech/articles/analyze-anthropic-usage-spend-and-access-with-aws-glue-amazon-s3-tables-and-aws-quick-136178.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/analyze-anthropic-usage-spend-and-access-with-aws-glue-amazon-s3-tables-and-aws-quick/>)

Author: Daniyal Alam

Published: 2026-11-16T20:09:24Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Claude](<https://devfeed.tech/topics/claude.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [anthropic](<https://devfeed.tech/topics/anthropic.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [JOIN](<https://devfeed.tech/topics/join.md>), [data ownership](<https://devfeed.tech/topics/data-ownership.md>), [AWS IAM](<https://devfeed.tech/topics/aws-iam.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [api](<https://devfeed.tech/tags/api.md>), [api-key](<https://devfeed.tech/tags/api-key.md>), [athena](<https://devfeed.tech/tags/athena.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-quick](<https://devfeed.tech/tags/aws-quick.md>), [s3](<https://devfeed.tech/tags/s3.md>), [s3-tables](<https://devfeed.tech/tags/s3-tables.md>), [tables](<https://devfeed.tech/tags/tables.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

AWS Glue's native Anthropic connector can load usage, cost, users, workspaces, and API key data into an AWS lakehouse without custom integration code. The walkthrough stores API keys in Secrets Manager, uses separate Admin and workspace connections for their different data scopes, and writes raw and curated Apache Iceberg tables to Amazon S3 Tables. AWS Glue materialized views join that data with business unit, outcome, and employee reference data; Athena queries the results, and AWS Quick can build dashboards directly from the views. The example surfaces spending by business unit, cost per outcome, and recently used keys created by former employees for ownership review--not automatic evidence of misuse. The article notes API rate limits, recommends bounded incremental reads, and specifies Glue materialized view requirements including Glue 5.1 or later, iam:PassRole, deterministic definitions, and inline subqueries because WITH-based views failed to refresh in its testing.

### Source excerpt

The new AWS Glue connector for Anthropic reads Claude usage, cost, and API key data into AWS without custom code. This post shows how to land it in Amazon S3 Tables as Apache Iceberg tables, join it with your business data, query it with Amazon Athena, and visualize AI spend and governance in AWS Quick.

## Introducing enhanced snapshots for Amazon Managed Service for Apache Flink

DevFeed: [Introducing enhanced snapshots for Amazon Managed Service for Apache Flink](<https://devfeed.tech/articles/introducing-enhanced-snapshots-for-amazon-managed-service-for-apache-flink-136180.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/introducing-enhanced-snapshots-for-amazon-managed-service-for-apache-flink/>)

Author: Francisco Morillo

Published: 2026-11-16T18:17:18Z

Content type: release

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Deployment Strategies](<https://devfeed.tech/topics/deployment-strategies.md>), [zero downtime deployment](<https://devfeed.tech/topics/zero-downtime-deployment.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-managed-service-for-apache-flink](<https://devfeed.tech/tags/amazon-managed-service-for-apache-flink.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [blue-green-deployment](<https://devfeed.tech/tags/blue-green-deployment.md>), [disaster-recovery](<https://devfeed.tech/tags/disaster-recovery.md>), [migrations](<https://devfeed.tech/tags/migrations.md>), [snapshots](<https://devfeed.tech/tags/snapshots.md>)

### AI overview

Enhanced snapshots are standalone AWS resources that let Amazon Managed Service for Apache Flink applications reuse state: start another application from a snapshot, export it to S3, or import a Flink savepoint. The text describes blue/green deployments, second-Region recovery, and migration from self-managed Flink. Blue/green cutover requires accounting for overlapping output and possible duplicates; both applications incur compute while running. Cross-Region recovery uses scheduled exports and S3 replication, trading more frequent exports for lower recovery work and higher operation, storage, and replication costs. Migration and restore require compatible Flink versions and matching operator UIDs; restores also require compatible state types and maxParallelism, and imported savepoints have validation prerequisites. Enhanced snapshots require Flink 1.20 or later, have separate IAM permissions, and persist until deleted, accruing storage charges. The text says snapshot storage is priced like application snapshots, with no added export/import charge; S3 charges still apply.

### Source excerpt

Enhanced snapshots for Amazon Managed Service for Apache Flink are now generally available. An enhanced snapshot is an AWS resource with its own ARN, decoupled from the application that created it, so you can start another application from it, export it to Amazon S3, and import savepoints for blue/green deployments, disaster recovery, and migrations.

## Upgrading and troubleshooting Amazon MWAA with AWS agent skills

DevFeed: [Upgrading and troubleshooting Amazon MWAA with AWS agent skills](<https://devfeed.tech/articles/upgrading-and-troubleshooting-amazon-mwaa-with-aws-agent-skills-136182.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/upgrading-and-troubleshooting-amazon-mwaa-with-aws-agent-skills/>)

Author: Kamen Sharlandjiev

Published: 2026-11-16T17:19:03Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Agent Skills](<https://devfeed.tech/topics/agent-skills.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-skills](<https://devfeed.tech/tags/agent-skills.md>), [agent-toolkit](<https://devfeed.tech/tags/agent-toolkit.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-coding-assistant](<https://devfeed.tech/tags/ai-coding-assistant.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-managed-workflows-for-apache-airflow-amazon-mwaa](<https://devfeed.tech/tags/amazon-managed-workflows-for-apache-airflow-amazon-mwaa.md>), [aws](<https://devfeed.tech/tags/aws.md>), [change](<https://devfeed.tech/tags/change.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-assistant](<https://devfeed.tech/tags/coding-assistant.md>), [environment](<https://devfeed.tech/tags/environment.md>), [failures](<https://devfeed.tech/tags/failures.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [safety](<https://devfeed.tech/tags/safety.md>), [skills](<https://devfeed.tech/tags/skills.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [toolkit](<https://devfeed.tech/tags/toolkit.md>), [troubleshoot](<https://devfeed.tech/tags/troubleshoot.md>), [troubleshooting](<https://devfeed.tech/tags/troubleshooting.md>), [upgrade](<https://devfeed.tech/tags/upgrade.md>)

### AI overview

AWS's MWAA agent skills guide coding assistants through workflow authoring, testing, troubleshooting, and Airflow upgrades. The upgrade skill is for provisioned environments; it plans version steps, including a 2.11.x bridge for older sources moving to Airflow 3, validates DAGs, and supports blue/green or in-place strategies. Authoring, testing, and debugging also support MWAA Serverless, which uses YAML workflows. Skills can edit and deploy workflows, so the article recommends testing in development, keeping code under version control, using least-privilege credentials, and confirming every environment change and version jump. The AWS MCP Server supplies current documentation and sandboxed, audited CLI execution.

### Source excerpt

AWS publishes MWAA agent skills in the open source AWS agent toolkit: procedures your AI coding assistant can run to upgrade Amazon MWAA from Airflow 2.x to 3.x and troubleshoot failures, with a safety confirmation on every environment change.

## Understanding span queries in Amazon OpenSearch Service

DevFeed: [Understanding span queries in Amazon OpenSearch Service](<https://devfeed.tech/articles/understanding-span-queries-in-amazon-opensearch-service-133390.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/understanding-span-queries-in-amazon-opensearch-service/>)

Author: Jon Handler

Published: 2026-11-13T22:00:29Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [opensearch](<https://devfeed.tech/topics/opensearch.md>), [Amazon OpenSearch Service](<https://devfeed.tech/topics/amazon-opensearch-service.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>)

Tags: [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-opensearch-service](<https://devfeed.tech/tags/amazon-opensearch-service.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cloud](<https://devfeed.tech/tags/aws-cloud.md>), [edit-distance](<https://devfeed.tech/tags/edit-distance.md>), [expert-400](<https://devfeed.tech/tags/expert-400.md>), [opensearch](<https://devfeed.tech/tags/opensearch.md>), [queries](<https://devfeed.tech/tags/queries.md>), [query](<https://devfeed.tech/tags/query.md>), [span](<https://devfeed.tech/tags/span.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Span queries let Amazon OpenSearch Service users express term-order, position, proximity, and containment constraints that phrase queries cannot reliably capture. Phrase queries match contiguous terms; `match_phrase` slop allows positional swaps, which can produce matches with terms in an unexpected order. Span queries combine span-only clauses such as `span_term`, `span_near`, `span_first`, `span_not`, `span_containing`, and `span_within`; `in_order` controls clause ordering, while containment queries differ in whether they return the larger or smaller span. The examples show anchoring a term near a field's start and requiring an exact phrase to occur inside a broader span. This precision can help legal, regulatory, patent, scientific, and compliance searches where placement matters, and offers a query-time option when text lacks a structured model. Span queries are more verbose and require careful composition.

### Source excerpt

Among OpenSearch query types, span queries give you precise control over where terms appear in a document and how they relate to one another. Learn how span queries work, when to use them instead of phrase queries, and how to compose them to enforce exact term order, position, and containment that phrase queries cannot express.

## Optimizing PySpark UDFs on AWS Glue 6.0 with native Arrow UDFs

DevFeed: [Optimizing PySpark UDFs on AWS Glue 6.0 with native Arrow UDFs](<https://devfeed.tech/articles/optimizing-pyspark-udfs-on-aws-glue-6-0-with-native-arrow-udfs-133389.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/optimizing-pyspark-udfs-on-aws-glue-6-0-with-native-arrow-udfs/>)

Author: Iván Fernández Perea

Published: 2026-11-13T20:58:25Z

Content type: comparison

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [apache-arrow](<https://devfeed.tech/topics/apache-arrow.md>), [pandas](<https://devfeed.tech/topics/pandas.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [arrow](<https://devfeed.tech/tags/arrow.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [can](<https://devfeed.tech/tags/can.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [jvm](<https://devfeed.tech/tags/jvm.md>), [native](<https://devfeed.tech/tags/native.md>), [optimized](<https://devfeed.tech/tags/optimized.md>), [optimizing](<https://devfeed.tech/tags/optimizing.md>), [pandas](<https://devfeed.tech/tags/pandas.md>), [pyspark](<https://devfeed.tech/tags/pyspark.md>), [python](<https://devfeed.tech/tags/python.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [runtime](<https://devfeed.tech/tags/runtime.md>), [spark](<https://devfeed.tech/tags/spark.md>), [sql](<https://devfeed.tech/tags/sql.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

In AWS Glue 6.0 tests on 10M-100M rows, built-in Spark SQL was fastest when applicable; among UDFs, native Arrow led when its logic fit PyArrow compute kernels, while pandas UDFs were the flexible fallback. Native Arrow processes pyarrow arrays directly, avoiding the pandas conversion; pandas UDFs use Arrow transport and batch NumPy work; Arrow-optimized scalar UDFs improve transport but still run Python per row. For math-heavy work at 100M rows, the native Arrow UDF took 3.13 seconds, versus 3.83 for pandas and 11.01 for Arrow-optimized Python. Iterative logic unavailable in SQL or PyArrow kernels left pandas UDFs as the fastest tested option. A controlled test found NumPy through RDD.mapPartitions slower than standard Python UDF, because it materialized Python objects and left DataFrame execution. Results used 10 G.1X workers; absolute timings may vary. Cache handling and warm-up materially affected measurements.

### Source excerpt

Python UDFs in PySpark can be slow because data is serialized between the JVM and the Python runtime. We benchmark five approaches on AWS Glue 6.0 (standard Python, Arrow-optimized Python, vectorized pandas, native Arrow UDFs, and Spark SQL) across 10M to 100M rows, and separate transport cost from compute cost to explain the ranking.

## Multi-statement transactions for Apache Iceberg tables in Amazon Redshift

DevFeed: [Multi-statement transactions for Apache Iceberg tables in Amazon Redshift](<https://devfeed.tech/articles/multi-statement-transactions-for-apache-iceberg-tables-in-amazon-redshift-133388.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/multi-statement-transactions-for-apache-iceberg-tables-in-amazon-redshift/>)

Author: Shiyu Xu

Published: 2026-11-13T19:32:46Z

Content type: release

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [atomic](<https://devfeed.tech/tags/atomic.md>), [foundational-100](<https://devfeed.tech/tags/foundational-100.md>), [snapshot](<https://devfeed.tech/tags/snapshot.md>), [transactions](<https://devfeed.tech/tags/transactions.md>)

### AI overview

Amazon Redshift Patch 206 and later can group multiple DML statements against one Apache Iceberg table into a single `BEGIN ... COMMIT`, publishing one atomic snapshot. The transaction can include an audit before commit; failures and rollbacks publish no changes. All writes must target the same table using the same reference name, while reads may include uncommitted writes and each table's version is fixed on its first read. Run Iceberg DDL separately. Uncommitted intermediate files can become orphans; S3 Tables removes unreferenced files by default, while Glue tables require the orphan file deletion optimizer enabled in advance. In the reported test, batching 25 small statements reduced total runtime by 14.77% for INSERT and 11.29% for DELETE; heavy data work showed no meaningful speedup. At 32 concurrent writers, four-INSERT transactions achieved 3.59 times the committed-statement rate of auto-commit. Results vary with statement size, partitioning, and contention.

### Source excerpt

Amazon Redshift now supports multi-statement transactions for Apache Iceberg tables, so you can run several DML statements against one Iceberg table inside a single BEGIN...COMMIT and publish them as one atomic Iceberg snapshot. This post explains why one commit per statement makes multi-step pipelines fragile, what you can run in a transaction, and how it performs.

## Build a self-healing ETL pipeline with Amazon Bedrock AgentCore

DevFeed: [Build a self-healing ETL pipeline with Amazon Bedrock AgentCore](<https://devfeed.tech/articles/build-a-self-healing-etl-pipeline-with-amazon-bedrock-agentcore-133386.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/build-a-self-healing-etl-pipeline-with-amazon-bedrock-agentcore/>)

Author: Kevin Tinn

Published: 2026-11-13T19:08:10Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon Bedrock AgentCore](<https://devfeed.tech/topics/amazon-bedrock-agentcore.md>), [etl](<https://devfeed.tech/topics/etl.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon Bedrock Knowledge Bases](<https://devfeed.tech/topics/amazon-bedrock-knowledge-bases.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [knowledge-engineering](<https://devfeed.tech/topics/knowledge-engineering.md>), [aws-sns](<https://devfeed.tech/topics/aws-sns.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [agentcore](<https://devfeed.tech/tags/agentcore.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-bedrock-agentcore](<https://devfeed.tech/tags/amazon-bedrock-agentcore.md>), [automatically](<https://devfeed.tech/tags/automatically.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [bedrock](<https://devfeed.tech/tags/bedrock.md>), [bedrock-agentcore](<https://devfeed.tech/tags/bedrock-agentcore.md>), [build](<https://devfeed.tech/tags/build.md>), [can](<https://devfeed.tech/tags/can.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [domain](<https://devfeed.tech/tags/domain.md>), [errors](<https://devfeed.tech/tags/errors.md>), [etl](<https://devfeed.tech/tags/etl.md>), [etl-pipeline](<https://devfeed.tech/tags/etl-pipeline.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [self-healing](<https://devfeed.tech/tags/self-healing.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

The described AWS Glue pipeline routes records that fail schema validation to a four-agent system on Amazon Bedrock AgentCore. A correction agent queries a Bedrock Knowledge Base for domain guidance and looks up precomputed strategies in S3 Vectors in parallel, selecting the higher-confidence result. At the default 0.7 threshold, it applies qualifying corrections; otherwise, the orchestrator sends an SNS alert for human review. A persistence agent writes corrected records to both corrected and transformed data lake zones. The sample annotates records with the chosen strategy rather than changing field values; direct transformations require extending its correction function. With repository credentials configured, a remediation agent can analyze the source error and open a GitHub pull request; without them, that step is skipped. The post outlines AWS CDK deployment and sample-data workflow, and notes required AWS services, model access, Python 3.10+, and a container runtime for local testing.

### Source excerpt

Build a self-healing ETL pipeline on AWS Glue and Amazon Bedrock AgentCore that automatically corrects bad records using domain knowledge, writes the fixes back to the data lake, and can open pull requests to prevent the same errors from recurring.

## Migrating from Apache Hudi-based data lake to Apache Iceberg-based data lake

DevFeed: [Migrating from Apache Hudi-based data lake to Apache Iceberg-based data lake](<https://devfeed.tech/articles/migrating-from-apache-hudi-based-data-lake-to-apache-iceberg-based-data-lake-130716.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/migrating-from-apache-hudi-based-data-lake-to-apache-iceberg-based-data-lake/>)

Author: Nitin Kumar

Published: 2026-11-12T15:50:48Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-hudi](<https://devfeed.tech/tags/apache-hudi.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [athena](<https://devfeed.tech/tags/athena.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [based](<https://devfeed.tech/tags/based.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-lakes](<https://devfeed.tech/tags/data-lakes.md>), [files](<https://devfeed.tech/tags/files.md>), [full](<https://devfeed.tech/tags/full.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [hudi](<https://devfeed.tech/tags/hudi.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [insert](<https://devfeed.tech/tags/insert.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The excerpt presents three Hudi-to-Iceberg migration paths on AWS. In-place migration reuses Parquet files, saving rewrite time and storage but retaining the source schema, partitioning, and file layout. Iceberg's `add_files` suits a one-time conversion and requires Hudi cleanup first; for MoR, compact logs and clean old versions to avoid missing updates or duplicate rows. Apache XTable translates Hudi metadata and can sync ongoing writes, but MoR logs must be compacted before each conversion. Both approaches share data files, so avoid destructive Iceberg maintenance until cutover; `add_files` and `snapshot` also lack column statistics in Iceberg 1.9.0-1.10.0, with the issue fixed in 1.10.1. Full migration with CTAS or CREATE plus INSERT rewrites data, allowing schema and partition changes, including hidden partitioning, at higher compute, time, and storage cost. For large tables, partitioned INSERT batches support parallelism and resumability. The excerpt demonstrates AWS Glue notebooks, general S3 buckets, and Amazon S3 Tables, and recommends validating against Hudi before cutover.

### Source excerpt

Learn how to migrate transactional data lakes from Apache Hudi to Apache Iceberg on AWS. This post walks through in-place migration with the Iceberg add_files procedure and Apache XTable, plus full table migration with CTAS and INSERT, using AWS Glue notebooks and Amazon S3 Tables, then validates the results in Amazon Athena.

## Managed cross-Region disaster recovery for Amazon OpenSearch Service

DevFeed: [Managed cross-Region disaster recovery for Amazon OpenSearch Service](<https://devfeed.tech/articles/managed-cross-region-disaster-recovery-for-amazon-opensearch-service-129794.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/managed-cross-region-disaster-recovery-for-amazon-opensearch-service/>)

Author: Cedric Pelvet

Published: 2026-11-12T00:45:45Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon OpenSearch Service](<https://devfeed.tech/topics/amazon-opensearch-service.md>), [Disaster Recovery](<https://devfeed.tech/topics/disaster-recovery.md>), [opensearch](<https://devfeed.tech/topics/opensearch.md>), [read-replica](<https://devfeed.tech/topics/read-replica.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>)

Tags: [ai-workloads](<https://devfeed.tech/tags/ai-workloads.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-opensearch-service](<https://devfeed.tech/tags/amazon-opensearch-service.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [api](<https://devfeed.tech/tags/api.md>), [aws-command-line-interface-aws-cli](<https://devfeed.tech/tags/aws-command-line-interface-aws-cli.md>), [aws-key-management-service-aws-kms](<https://devfeed.tech/tags/aws-key-management-service-aws-kms.md>), [aws-management-console](<https://devfeed.tech/tags/aws-management-console.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [business-continuity](<https://devfeed.tech/tags/business-continuity.md>), [cross](<https://devfeed.tech/tags/cross.md>), [disaster-recovery](<https://devfeed.tech/tags/disaster-recovery.md>), [failover](<https://devfeed.tech/tags/failover.md>), [foundational-100](<https://devfeed.tech/tags/foundational-100.md>), [region](<https://devfeed.tech/tags/region.md>), [rpo](<https://devfeed.tech/tags/rpo.md>), [rto](<https://devfeed.tech/tags/rto.md>)

### AI overview

Amazon OpenSearch Service now provides managed cross-Region disaster recovery for domains running OpenSearch 3.7 or later. Pair same-account domains in different Regions with matching capacity; the service copies existing data and continuously replicates changes, with both domains readable and only the primary writable. Planned switchover coordinates a role swap with zero data loss; failover promotes the secondary without waiting, and may lose writes still in replication transit, bounded by replication lag. The stated targets are RPO under one minute and RTO under 20 minutes; example tests saw 8-10 seconds of loss on failover and writes resume in about a minute. Applications must redirect writes after promotion. Index templates, settings, roles, and ISM policies are not replicated and must be managed on both domains. UltraWarm, Writable Warm, and the OpenSearch Optimized engine for log analytics are unsupported. There is no additional service charge, though secondary-domain capacity and cross-Region transfer are billed.

### Source excerpt

Amazon OpenSearch Service now offers managed cross-Region disaster recovery for domains on OpenSearch 3.7 or later. It replicates data from a primary domain to a secondary domain in another Region with API-driven switchover and failover, providing an RPO under 1 minute and an RTO under 20 minutes.

## Optimizing Apache HBase availability on Amazon EMR, Part 2: Diagnose and tune compaction

DevFeed: [Optimizing Apache HBase availability on Amazon EMR, Part 2: Diagnose and tune compaction](<https://devfeed.tech/articles/optimizing-apache-hbase-availability-on-amazon-emr-part-2-diagnose-and-tune-compaction-129563.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/optimizing-apache-hbase-availability-on-amazon-emr-part-2-diagnose-and-tune-compaction/>)

Author: Reekunj Parmar

Published: 2026-11-11T22:06:59Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [amazon-emr](<https://devfeed.tech/tags/amazon-emr.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [availability](<https://devfeed.tech/tags/availability.md>), [aws-support](<https://devfeed.tech/tags/aws-support.md>), [compaction](<https://devfeed.tech/tags/compaction.md>), [diagnose](<https://devfeed.tech/tags/diagnose.md>), [hbase](<https://devfeed.tech/tags/hbase.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [latency](<https://devfeed.tech/tags/latency.md>), [learn-how-to](<https://devfeed.tech/tags/learn-how-to.md>), [optimizing](<https://devfeed.tech/tags/optimizing.md>), [performance](<https://devfeed.tech/tags/performance.md>), [series](<https://devfeed.tech/tags/series.md>), [spikes](<https://devfeed.tech/tags/spikes.md>), [support](<https://devfeed.tech/tags/support.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The article recommends confirming compaction pressure before changing settings: check the affected RegionServer's compaction queue and store files in the HBase UI, rule out GC and host resource issues, and compare region read latency with a canary baseline. For HBase on S3 with EMR 7.10-7.13, first set `fs.s3a.experimental.input.fadvise` to `sequential`; then lower compaction throughput bounds, tune frequency, define off-peak hours, or disable automatic major compaction and schedule it manually. Change one parameter at a time and restart HBase as required. Manual scheduling means deleted cells and markers remain until major compaction, so monitor store file counts and avoid running manual compactions on read replicas. To reduce compaction work, limit region size, address write hotspotting, reduce small flushes, and size BucketCache appropriately; increasing MemStore settings can increase read-replica lag and requires sufficient heap. Contact AWS Support if two rounds of tuning do not improve latency or before risky metadata repairs.

### Source excerpt

Compaction keeps HBase read performance healthy, but when it runs during peak hours it drives latency spikes, write stalls, and RegionTooBusyException errors. In Part 2 of this series, you learn how to confirm compaction is the cause on Amazon EMR, tune the right parameters in the least-invasive order, and know when to stop and contact AWS Support.

## Build a data foundation for agentic AI on AWS: From lakehouse to MCP

DevFeed: [Build a data foundation for agentic AI on AWS: From lakehouse to MCP](<https://devfeed.tech/articles/build-a-data-foundation-for-agentic-ai-on-aws-from-lakehouse-to-mcp-129558.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/build-a-data-foundation-for-agentic-ai-on-aws-from-lakehouse-to-mcp/>)

Author: Adrien Lebret

Published: 2026-11-11T21:03:43Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AI-Ready Data](<https://devfeed.tech/topics/ai-ready-data.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Amazon Bedrock Knowledge Bases](<https://devfeed.tech/topics/amazon-bedrock-knowledge-bases.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon Bedrock](<https://devfeed.tech/topics/amazon-bedrock.md>), [AWS IAM](<https://devfeed.tech/topics/aws-iam.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [Amazon OpenSearch Service](<https://devfeed.tech/topics/amazon-opensearch-service.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-data](<https://devfeed.tech/tags/ai-data.md>), [aws](<https://devfeed.tech/tags/aws.md>), [build](<https://devfeed.tech/tags/build.md>), [building](<https://devfeed.tech/tags/building.md>), [context](<https://devfeed.tech/tags/context.md>), [context-engineering](<https://devfeed.tech/tags/context-engineering.md>), [data](<https://devfeed.tech/tags/data.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [governance](<https://devfeed.tech/tags/governance.md>), [interoperability](<https://devfeed.tech/tags/interoperability.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [like](<https://devfeed.tech/tags/like.md>), [mcp](<https://devfeed.tech/tags/mcp.md>), [model](<https://devfeed.tech/tags/model.md>), [model-context-protocol](<https://devfeed.tech/tags/model-context-protocol.md>), [uncategorized](<https://devfeed.tech/tags/uncategorized.md>)

### AI overview

The article presents a four-pillar AWS data foundation for agentic AI: a unified lakehouse, context engineering, interoperability, and governance. Amazon SageMaker Lakehouse and S3 Tables bring operational, analytical, and AI data into a governed foundation with Apache Iceberg, zero-ETL ingestion, and federated queries. Context engineering combines knowledge bases, short- and long-term memory, and tools that fetch selected data, addressing stateless models' limited context windows. For collaboration, A2A supports agent-to-agent communication over HTTP and JSON, while MCP standardizes agent access to resources, tools, and prompts; Amazon Bedrock AgentCore Gateway centralizes connections, authentication, audit logging, and permission-scoped semantic tool discovery. Amazon SageMaker Data Catalog adds dataset discovery, lineage, quality indicators, and publish-and-subscribe access controls. The article recommends establishing the lakehouse and ingestion first, developing context capabilities before scaling orchestration, and treating governance as an ongoing effort from the start.

### Source excerpt

Building a data foundation ready for agentic AI relies on a unified lakehouse and interoperability protocols like Model Context Protocol (MCP). This post walks through four pillars for building an agentic AI data foundation on AWS: a unified lakehouse, context engineering, multi-agent interoperability, and data governance.

## Build a multi-team AI and analytics environment on Amazon SageMaker Unified Studio with Terraform

DevFeed: [Build a multi-team AI and analytics environment on Amazon SageMaker Unified Studio with Terraform](<https://devfeed.tech/articles/build-a-multi-team-ai-and-analytics-environment-on-amazon-sagemaker-unified-studio-with-terraform-129559.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/build-a-multi-team-ai-and-analytics-environment-on-amazon-sagemaker-unified-studio-with-terraform/>)

Author: Harshi Surampalli

Published: 2026-11-11T19:58:49Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [Terraform](<https://devfeed.tech/topics/terraform.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Cloud Networking](<https://devfeed.tech/topics/cloud-networking.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [Amazon Bedrock Knowledge Bases](<https://devfeed.tech/topics/amazon-bedrock-knowledge-bases.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [amazon-sagemaker-unified-studio](<https://devfeed.tech/tags/amazon-sagemaker-unified-studio.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apply](<https://devfeed.tech/tags/apply.md>), [blueprints](<https://devfeed.tech/tags/blueprints.md>), [build](<https://devfeed.tech/tags/build.md>), [code](<https://devfeed.tech/tags/code.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analytics](<https://devfeed.tech/tags/data-analytics.md>), [domains](<https://devfeed.tech/tags/domains.md>), [environment](<https://devfeed.tech/tags/environment.md>), [generative](<https://devfeed.tech/tags/generative.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [governance](<https://devfeed.tech/tags/governance.md>), [studio](<https://devfeed.tech/tags/studio.md>), [team](<https://devfeed.tech/tags/team.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [terraform](<https://devfeed.tech/tags/terraform.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

The Terraform module provisions a shared, governed Amazon SageMaker Unified Studio domain with team-specific project profiles for analytics, machine learning, and generative AI. It combines AWS and AWS Cloud Control providers, enables selected blueprints, and can create least-privilege service roles or use supplied roles; network settings can default to discovery of the default VPC and subnets. The walkthrough requires Terraform 1.7 or later, AWS credentials with resource-management permissions, and Unified Studio in the target Region; IAM Identity Center is needed only for SSO membership. One apply creates the domain, profiles, and an initial project, while further team projects require additional project module blocks or portal creation. Membership supports SSO users and groups plus same-account IAM users and roles. IAM principals need SageMakerStudioUserIAMConsolePolicy or equivalent, which the module does not attach; cross-account role access may be limited and should be verified for the Region. Costs mainly arise from using Redshift Serverless, EMR Serverless, Bedrock, and S3.

### Source excerpt

Learn how to use the Amazon SageMaker Unified Studio Terraform module to provision a multi-team AI and analytics platform (domains, blueprints, project profiles, and projects) for data analytics, machine learning, and generative AI teams in a single terraform apply, with shared governance and least-privilege security defined entirely as code.

## How Samsung Ads federated Snowflake Iceberg tables using Amazon Cognito

DevFeed: [How Samsung Ads federated Snowflake Iceberg tables using Amazon Cognito](<https://devfeed.tech/articles/how-samsung-ads-federated-snowflake-iceberg-tables-using-amazon-cognito-129562.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/how-samsung-ads-federated-snowflake-iceberg-tables-using-amazon-cognito/>)

Author: Balaji Selva Rajan

Published: 2026-11-11T19:48:02Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon Cognito](<https://devfeed.tech/topics/amazon-cognito.md>), [AWS Lake Formation](<https://devfeed.tech/topics/aws-lake-formation.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon VPC](<https://devfeed.tech/topics/amazon-vpc.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [AWS CloudShell](<https://devfeed.tech/topics/aws-cloudshell.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [ads](<https://devfeed.tech/tags/ads.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-cognito](<https://devfeed.tech/tags/amazon-cognito.md>), [amazon-route-53](<https://devfeed.tech/tags/amazon-route-53.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [amazon-sagemaker-ai](<https://devfeed.tech/tags/amazon-sagemaker-ai.md>), [amazon-vpc](<https://devfeed.tech/tags/amazon-vpc.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [authentication](<https://devfeed.tech/tags/authentication.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cli](<https://devfeed.tech/tags/aws-cli.md>), [aws-cloudformation](<https://devfeed.tech/tags/aws-cloudformation.md>), [aws-cloudshell](<https://devfeed.tech/tags/aws-cloudshell.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-lake-formation](<https://devfeed.tech/tags/aws-lake-formation.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [oauth2](<https://devfeed.tech/tags/oauth2.md>), [samsung](<https://devfeed.tech/tags/samsung.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Samsung Ads federated Snowflake-managed Iceberg tables into AWS Lake Formation so Amazon Athena can query the same S3 data without a second copy. An AWS Glue connection obtains OAuth2 client-credentials tokens from Amazon Cognito; a Pre Token Generation Lambda adds the `aud` claim Snowflake requires. The connection reaches Snowflake's Iceberg REST catalog over AWS PrivateLink, while Cognito's public token endpoint requires NAT access. Lake Formation governs access and vends scoped S3 credentials for Athena to read the Parquet files directly. Setup requires Snowflake Business-Critical or higher in the same Region, Snowflake-managed Iceberg tables, and AWS permissions for the Cognito, networking, and Lake Formation resources. The pattern does not apply Snowflake masking or row access policies to Athena reads; those controls must be recreated in Lake Formation and tested. Federate schemas containing only supported Snowflake-managed tables, and explicitly revoke Lake Formation grants when tables are removed. NAT and PrivateLink endpoints incur charges.

### Source excerpt

Learn how Samsung Ads federates Snowflake-managed Apache Iceberg tables into AWS Lake Formation and queries them from Amazon Athena using Amazon Cognito for OAuth2 machine-to-machine authentication, with no second copy of the data to maintain.

## Allocate shared Amazon MSK costs to Kafka topics for cost visibility

DevFeed: [Allocate shared Amazon MSK costs to Kafka topics for cost visibility](<https://devfeed.tech/articles/allocate-shared-amazon-msk-costs-to-kafka-topics-for-cost-visibility-129557.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/allocate-shared-amazon-msk-costs-to-kafka-topics-for-cost-visibility/>)

Author: Durai Krishnan

Published: 2026-11-11T19:46:04Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [cloud cost management](<https://devfeed.tech/topics/cloud-cost-management.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [Amazon Managed Service for Prometheus](<https://devfeed.tech/topics/amazon-managed-service-for-prometheus.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Prometheus](<https://devfeed.tech/topics/prometheus.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [aws-sns](<https://devfeed.tech/topics/aws-sns.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-eventbridge](<https://devfeed.tech/tags/amazon-eventbridge.md>), [amazon-managed-service-for-prometheus](<https://devfeed.tech/tags/amazon-managed-service-for-prometheus.md>), [amazon-managed-streaming-for-apache-kafka-amazon-msk](<https://devfeed.tech/tags/amazon-managed-streaming-for-apache-kafka-amazon-msk.md>), [amazon-msk](<https://devfeed.tech/tags/amazon-msk.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-simple-storage-service-amazon-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-amazon-s3.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cost-and-usage-report](<https://devfeed.tech/tags/aws-cost-and-usage-report.md>), [billing](<https://devfeed.tech/tags/billing.md>), [cloudwatch](<https://devfeed.tech/tags/cloudwatch.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cost-and-usage-report](<https://devfeed.tech/tags/cost-and-usage-report.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [prometheus](<https://devfeed.tech/tags/prometheus.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [topic](<https://devfeed.tech/tags/topic.md>), [visibility](<https://devfeed.tech/tags/visibility.md>)

### AI overview

The pipeline estimates each Kafka topic's share of an Amazon MSK Provisioned cluster's daily cost by joining cluster-level CUR charges with per-topic ingress metrics, mapping cluster names to ARNs, then allocating cost in proportion to bytes in. A daily Lambda writes Parquet results to S3 for Athena and Quick Sight. Enhanced Monitoring publishes per-topic CloudWatch metrics with little setup, but metric charges grow with topic-broker pairs. Open Monitoring exposes broker metrics at no additional charge; a Fargate scraper sends them to Amazon Managed Service for Prometheus, whose ingestion costs grow with samples. Both options use the same allocation logic. Bytes in is only a proxy: it misses retention, replication, compression, and consumer fan-out. Idle topics receive no row, so active topics may absorb the full cost; if no topics report traffic, the function splits cost evenly across configured topics. CUR can lag up to 24 hours, and metrics exist only after they are enabled. The method applies to Provisioned clusters; Serverless is out of scope.

### Source excerpt

Amazon MSK bills at the cluster level, with no per-topic breakdown. This post shows how to allocate shared MSK cluster costs to individual Kafka topics by combining AWS Cost and Usage Report billing with per-topic CloudWatch or Prometheus metrics, giving each team a usage-based view of its spend.

## Governance compliance beyond the catalog in Amazon SageMaker Unified Studio

DevFeed: [Governance compliance beyond the catalog in Amazon SageMaker Unified Studio](<https://devfeed.tech/articles/governance-compliance-beyond-the-catalog-in-amazon-sagemaker-unified-studio-129560.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/governance-compliance-beyond-the-catalog-in-amazon-sagemaker-unified-studio/>)

Author: Amom Luiz

Published: 2026-11-11T16:08:47Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [Data visualization](<https://devfeed.tech/topics/data-visualization.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data ownership](<https://devfeed.tech/topics/data-ownership.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-eventbridge](<https://devfeed.tech/tags/amazon-eventbridge.md>), [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [amazon-sagemaker-unified-studio](<https://devfeed.tech/tags/amazon-sagemaker-unified-studio.md>), [apis](<https://devfeed.tech/tags/apis.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [forms](<https://devfeed.tech/tags/forms.md>), [glossary](<https://devfeed.tech/tags/glossary.md>), [governance](<https://devfeed.tech/tags/governance.md>), [governance-compliance](<https://devfeed.tech/tags/governance-compliance.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [join](<https://devfeed.tech/tags/join.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [policy](<https://devfeed.tech/tags/policy.md>), [score](<https://devfeed.tech/tags/score.md>), [subscription](<https://devfeed.tech/tags/subscription.md>), [table](<https://devfeed.tech/tags/table.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The post describes a daily enrichment pipeline that joins SageMaker Unified Studio's asset metadata export with catalog APIs and an S3 policy file, then writes per-asset governance records to an Iceberg table. It combines built-in required-form rules from GetAssetType with organization-defined requirements, and adds glossary associations, data quality signals, descriptions, and active subscription counts. Teams can query weighted compliance scores and trends in Athena, build Quick Sight dashboards, and schedule alerts; subscription counts help prioritize low-scoring assets with wider use. The five score dimensions and weights are configurable examples, not prescribed policy, and organizations must define what compliance means for them. The sample test processed 29 Glue table assets in about 30 seconds; the post suggests parallelization, more Lambda memory, or Step Functions for larger catalogs.

### Source excerpt

Amazon SageMaker provides rich governance signals such as metadata forms, data quality scores, glossary coverage, and subscription data, but they live in separate APIs. This post shows how to join them into a single enriched Iceberg table and build a compliance scorecard you can query, score, and adapt to your governance policy.

[Next page](<https://devfeed.tech/sources/aws-big-data-blog.md?cursor=WyIyMDI2LTExLTExVDE2OjA4OjQ3KzAwOjAwIiwgIjAwNmE4Y2UyLTM1ZDItNDhhZC1hYjkyLWFlYTVjNjU1ZDhiMiJd>)