# AWS Glue

AWS Glue is a serverless data integration service for discovering, preparing, combining, and processing data, including ETL pipelines built with Apache Spark, Python, and Scala.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Amazon S3 Tables with dbt: Iceberg-based data transformations using AWS Glue

DevFeed: [Amazon S3 Tables with dbt: Iceberg-based data transformations using AWS Glue](<https://devfeed.tech/articles/amazon-s3-tables-with-dbt-iceberg-based-data-transformations-using-aws-glue-144273.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/amazon-s3-tables-with-dbt-iceberg-based-data-transformations-using-aws-glue/>)

Author: Yukiko Iwazumi

Published: 2026-11-19T17:58:50Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-quick-sight](<https://devfeed.tech/tags/amazon-quick-sight.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The walkthrough builds a dbt pipeline on Amazon S3 Tables through AWS Glue interactive sessions, with raw CSV data staged and transformed into analytics marts. It requires dbt-glue 1.12.4 or later for S3 Tables incremental models, Python 3.10-3.12, and AWS Glue 6.0 or later. In the example, `fact_orders` uses merge to retain each order's latest state, while `fact_order_history` uses append to retain changes. After source updates, the merge table has six orders, including the updated order 1004 and new order 1006; the history table grows from five to seven rows. The post describes S3 Tables support as experimental in dbt-glue 1.12.4 and recommends testing it in your environment.

### Source excerpt

Amazon S3 Tables, a capability of Amazon S3, bring built-in Apache Iceberg support to object storage. This post shows how to configure and run dbt transformations on S3 Tables with the AWS Glue dbt-glue adapter, using incremental merge and append strategies across a staging and mart architecture.

## Introducing automatic version upgrades for AWS Glue jobs

DevFeed: [Introducing automatic version upgrades for AWS Glue jobs](<https://devfeed.tech/articles/introducing-automatic-version-upgrades-for-aws-glue-jobs-141879.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/introducing-automatic-version-upgrades-for-aws-glue-jobs/>)

Author: Keerthi Chadalavada

Published: 2026-11-19T00:45:54Z

Content type: release

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [version](<https://devfeed.tech/topics/version.md>), [Amazon Bedrock](<https://devfeed.tech/topics/amazon-bedrock.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [automatic](<https://devfeed.tech/tags/automatic.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [changes](<https://devfeed.tech/tags/changes.md>), [fleet](<https://devfeed.tech/tags/fleet.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [job](<https://devfeed.tech/tags/job.md>), [jobs](<https://devfeed.tech/tags/jobs.md>), [latest](<https://devfeed.tech/tags/latest.md>), [learn](<https://devfeed.tech/tags/learn.md>), [set](<https://devfeed.tech/tags/set.md>), [stable](<https://devfeed.tech/tags/stable.md>), [the-latest](<https://devfeed.tech/tags/the-latest.md>), [upgrade](<https://devfeed.tech/tags/upgrade.md>), [upgrades](<https://devfeed.tech/tags/upgrades.md>), [version](<https://devfeed.tech/tags/version.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

AWS Glue can automatically assess and upgrade eligible Spark ETL jobs when their `GlueVersion` is set to `auto`. Jobs continue running on their current version while Glue checks for known breaking changes; if no code changes are needed, it sends an eligibility summary for human approval. After approval, Glue validates upgraded runs for correctness, performance, and cost against earlier runs. A failed upgraded run is retried on the previous version and the job reverts. Jobs must use Glue 2.0 or later, have `MaxRetries` of at least 1, have job run queuing off, and be safe to retry. The console supports approval job by job; an AI agent with the managing-glue-jobs skill can assess and manage a fleet, with confirmation for changes and optional email notifications. Jobs requiring code changes need a separate generative AI upgrade first. The article says auto upgrade is available in commercial AWS Regions; region details are in the Glue User Guide.

### Source excerpt

Today we're announcing automatic version upgrades for AWS Glue jobs. Set a job's version to auto, and AWS Glue checks for breaking changes, asks for your approval, then moves the job to the latest stable version and validates the upgraded runs. Learn how auto upgrade works and how to use it across a fleet of jobs with an AI agent.

## Aderant modernizes analytics data pipeline with Apache Iceberg and AWS Lambda

DevFeed: [Aderant modernizes analytics data pipeline with Apache Iceberg and AWS Lambda](<https://devfeed.tech/articles/aderant-modernizes-analytics-data-pipeline-with-apache-iceberg-and-aws-lambda-141916.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/aderant-modernizes-analytics-data-pipeline-with-apache-iceberg-and-aws-lambda/>)

Author: Sham Birbahadur

Published: 2026-11-19T00:36:23Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [event-driven data processing](<https://devfeed.tech/topics/event-driven-data-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [AWS Database Migration Service](<https://devfeed.tech/topics/aws-database-migration-service.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [acid](<https://devfeed.tech/tags/acid.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [batch](<https://devfeed.tech/tags/batch.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [cost](<https://devfeed.tech/tags/cost.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [faster](<https://devfeed.tech/tags/faster.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [legal](<https://devfeed.tech/tags/legal.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [queries](<https://devfeed.tech/tags/queries.md>)

### AI overview

Aderant replaced Stridyn's 15-minute batch pipeline with an event-driven medallion architecture that processes CDC changes from 700 client environments in single-digit seconds to bronze-layer Apache Iceberg tables. AWS DMS delivers Parquet files to S3; S3 notifications and SQS invoke Lambda for ingestion, while scheduled AWS Glue jobs transform data through silver and gold layers. Incremental Glue jobs handle 77 standard gold tables; 34 complex tables use Iceberg materialized views and snapshot comparisons to send changed rows to native Amazon Redshift tables. Aderant reports 10-26x faster queries on native Redshift tables, under-100-millisecond dashboard queries, and 77 percent lower pipeline compute costs versus streaming alternatives. The account says this approach supports schema changes and concurrent tenant workloads; it also describes a possible future Kinesis path for high-volume tenants.

### Source excerpt

Learn how Aderant modernized the data pipeline behind its Stridyn legal analytics platform with Apache Iceberg, AWS Lambda, and AWS Glue, moving from 15-minute batch cycles to an event-driven medallion architecture that processes change data capture events in single-digit seconds for 700 tenants, with 10-26x faster queries and 77 percent lower cost.

## How United Airlines streams database changes to Apache Iceberg with Amazon Data Firehose

DevFeed: [How United Airlines streams database changes to Apache Iceberg with Amazon Data Firehose](<https://devfeed.tech/articles/how-united-airlines-streams-database-changes-to-apache-iceberg-with-amazon-data-firehose-141918.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/how-united-airlines-streams-database-changes-to-apache-iceberg-with-amazon-data-firehose/>)

Author: Guillaume Dupont

Published: 2026-11-18T20:37:39Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Amazon Aurora](<https://devfeed.tech/topics/amazon-aurora.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [streaming data to S3 with Parquet](<https://devfeed.tech/topics/streaming-data-to-s3-with-parquet.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [Visual Studio Code](<https://devfeed.tech/topics/visual-studio-code.md>)

Tags: [airlines](<https://devfeed.tech/tags/airlines.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-aurora](<https://devfeed.tech/tags/amazon-aurora.md>), [amazon-data-firehose](<https://devfeed.tech/tags/amazon-data-firehose.md>), [amazon-kinesis](<https://devfeed.tech/tags/amazon-kinesis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cloudformation](<https://devfeed.tech/tags/aws-cloudformation.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [dms](<https://devfeed.tech/tags/dms.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [kinesis-data-streams](<https://devfeed.tech/tags/kinesis-data-streams.md>), [streams](<https://devfeed.tech/tags/streams.md>), [united-airlines](<https://devfeed.tech/tags/united-airlines.md>)

### AI overview

United Airlines describes a near-real-time CDC pipeline that moves Aurora MySQL changes through AWS DMS, Kinesis Data Streams, Lambda, and Amazon Data Firehose into Apache Iceberg tables on S3. Firehose applies inserts, updates, and deletes as Iceberg snapshots; the open tables can be queried through AWS Glue, Redshift, or Athena. United reports 5x lower average query latency and 2x shorter downstream Glue job processing time in its testing, with a low-volume test reaching about 10 seconds end-to-end. Buffer settings trade latency against cost and file size. The walkthrough uses DMS Serverless, Athena for validation, and a simplified single-table CloudFormation template; United's production setup uses provisioned DMS and primarily Redshift Serverless. The template's shard count and other demo settings are not production-tuned, and schema evolution and multi-table replication require additional handling.

### Source excerpt

United Airlines built a near-real-time change data capture pipeline that replicates reference data from Amazon Aurora MySQL to open Apache Iceberg tables using AWS DMS, Amazon Kinesis Data Streams, and Amazon Data Firehose. This post shows how to reproduce the serverless, multi-engine analytics pattern with AWS CloudFormation.

## Fivetran Automated Cloud Data Integration Now Deploys on AWS | Blog | Fivetran

DevFeed: [Fivetran Automated Cloud Data Integration Now Deploys on AWS | Blog | Fivetran](<https://devfeed.tech/articles/fivetran-automated-cloud-data-integration-now-deploys-on-aws-blog-fivetran-141316.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/fivetran-now-deploys-on-aws>)

Author: Amy Peterson

Published: 2026-11-18T06:13:15Z

Content type: release

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [data mobility](<https://devfeed.tech/topics/data-mobility.md>), [Security, Identity, & Compliance](<https://devfeed.tech/topics/security-identity-compliance.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [about](<https://devfeed.tech/tags/about.md>), [addition](<https://devfeed.tech/tags/addition.md>), [all](<https://devfeed.tech/tags/all.md>), [all-in](<https://devfeed.tech/tags/all-in.md>), [automated](<https://devfeed.tech/tags/automated.md>), [aws](<https://devfeed.tech/tags/aws.md>), [blog](<https://devfeed.tech/tags/blog.md>), [can](<https://devfeed.tech/tags/can.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-data](<https://devfeed.tech/tags/cloud-data.md>), [cloud-services](<https://devfeed.tech/tags/cloud-services.md>), [costs](<https://devfeed.tech/tags/costs.md>), [customers](<https://devfeed.tech/tags/customers.md>), [data](<https://devfeed.tech/tags/data.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [integration](<https://devfeed.tech/tags/integration.md>), [services](<https://devfeed.tech/tags/services.md>)

### AI overview

Fivetran now lets customers on its Enterprise and Business Critical plans choose AWS or Google Cloud to run its services. AWS deployment is intended to help meet cloud-provider requirements and reduce data transfer costs by keeping processing in the same cloud region. The article states AWS support is available in the U.S., EU, U.K., and Australia, with more geographies planned. It lists SOC 2 Type II, ISO 27001, HIPAA (with a business associate agreement for covered customers), and PCI DSS Level 1 validation, and describes security as a shared responsibility with AWS. Fivetran says components can run across different clouds and be managed from one console; other provider options were still in development.

### Source excerpt

Joint customers can now stay within AWS for all cloud services and minimize data movement costs.

## Fivetran Named a 2019 Gartner Peer Insights Customers' Choice for Data Integration Tools | Blog | Fivetran

DevFeed: [Fivetran Named a 2019 Gartner Peer Insights Customers' Choice for Data Integration Tools | Blog | Fivetran](<https://devfeed.tech/articles/fivetran-named-a-2019-gartner-peer-insights-customers-choice-for-data-integration-tools-blog-fivetran-140226.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/fivetran-named-a-2019-gartner-peer-insights-customers-choice-for-data-integration-tools>)

Author: Fivetran

Published: 2026-11-18T02:46:08Z

Content type: release

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Data Infrastructure](<https://devfeed.tech/topics/data-infrastructure.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>)

Tags: [2019](<https://devfeed.tech/tags/2019.md>), [about](<https://devfeed.tech/tags/about.md>), [accelerate](<https://devfeed.tech/tags/accelerate.md>), [affiliates](<https://devfeed.tech/tags/affiliates.md>), [all](<https://devfeed.tech/tags/all.md>), [analyst](<https://devfeed.tech/tags/analyst.md>), [analysts](<https://devfeed.tech/tags/analysts.md>), [blog](<https://devfeed.tech/tags/blog.md>), [choice](<https://devfeed.tech/tags/choice.md>), [customers](<https://devfeed.tech/tags/customers.md>), [data](<https://devfeed.tech/tags/data.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [gartner](<https://devfeed.tech/tags/gartner.md>), [insights](<https://devfeed.tech/tags/insights.md>), [integration](<https://devfeed.tech/tags/integration.md>), [peer](<https://devfeed.tech/tags/peer.md>), [rating](<https://devfeed.tech/tags/rating.md>), [tools](<https://devfeed.tech/tags/tools.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Fivetran says it received the 2019 Gartner Peer Insights Customers' Choice distinction for data integration tools, citing customer feedback and a 4.4/5 rating as of November 20, 2019. The company describes connectors that adapt to schema and API changes to support reliable data delivery, and says it is expanding connector coverage, adding in-warehouse transformations, and improving reliability and support. Quoted users report easier data ingestion and setup, including for NetSuite, reduced pipeline setup effort, and responsive troubleshooting support. Gartner's disclaimer says the distinction reflects subjective end-user reviews and a documented methodology; it is not Gartner's endorsement.

### Source excerpt

Fivetran customers have spoken! With a 4.4/5 rating on Gartner Peer Insights as of November 20, 2019, we've been recognized in data integration tools.

## Analyze Anthropic usage, spend, and access with AWS Glue, Amazon S3 Tables, and AWS Quick

DevFeed: [Analyze Anthropic usage, spend, and access with AWS Glue, Amazon S3 Tables, and AWS Quick](<https://devfeed.tech/articles/analyze-anthropic-usage-spend-and-access-with-aws-glue-amazon-s3-tables-and-aws-quick-136178.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/analyze-anthropic-usage-spend-and-access-with-aws-glue-amazon-s3-tables-and-aws-quick/>)

Author: Daniyal Alam

Published: 2026-11-16T20:09:24Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Claude](<https://devfeed.tech/topics/claude.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [anthropic](<https://devfeed.tech/topics/anthropic.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [JOIN](<https://devfeed.tech/topics/join.md>), [data ownership](<https://devfeed.tech/topics/data-ownership.md>), [AWS IAM](<https://devfeed.tech/topics/aws-iam.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [api](<https://devfeed.tech/tags/api.md>), [api-key](<https://devfeed.tech/tags/api-key.md>), [athena](<https://devfeed.tech/tags/athena.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-quick](<https://devfeed.tech/tags/aws-quick.md>), [s3](<https://devfeed.tech/tags/s3.md>), [s3-tables](<https://devfeed.tech/tags/s3-tables.md>), [tables](<https://devfeed.tech/tags/tables.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

AWS Glue's native Anthropic connector can load usage, cost, users, workspaces, and API key data into an AWS lakehouse without custom integration code. The walkthrough stores API keys in Secrets Manager, uses separate Admin and workspace connections for their different data scopes, and writes raw and curated Apache Iceberg tables to Amazon S3 Tables. AWS Glue materialized views join that data with business unit, outcome, and employee reference data; Athena queries the results, and AWS Quick can build dashboards directly from the views. The example surfaces spending by business unit, cost per outcome, and recently used keys created by former employees for ownership review--not automatic evidence of misuse. The article notes API rate limits, recommends bounded incremental reads, and specifies Glue materialized view requirements including Glue 5.1 or later, iam:PassRole, deterministic definitions, and inline subqueries because WITH-based views failed to refresh in its testing.

### Source excerpt

The new AWS Glue connector for Anthropic reads Claude usage, cost, and API key data into AWS without custom code. This post shows how to land it in Amazon S3 Tables as Apache Iceberg tables, join it with your business data, query it with Amazon Athena, and visualize AI spend and governance in AWS Quick.

## Optimizing PySpark UDFs on AWS Glue 6.0 with native Arrow UDFs

DevFeed: [Optimizing PySpark UDFs on AWS Glue 6.0 with native Arrow UDFs](<https://devfeed.tech/articles/optimizing-pyspark-udfs-on-aws-glue-6-0-with-native-arrow-udfs-133389.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/optimizing-pyspark-udfs-on-aws-glue-6-0-with-native-arrow-udfs/>)

Author: Iván Fernández Perea

Published: 2026-11-13T20:58:25Z

Content type: comparison

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [apache-arrow](<https://devfeed.tech/topics/apache-arrow.md>), [pandas](<https://devfeed.tech/topics/pandas.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [arrow](<https://devfeed.tech/tags/arrow.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [can](<https://devfeed.tech/tags/can.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [jvm](<https://devfeed.tech/tags/jvm.md>), [native](<https://devfeed.tech/tags/native.md>), [optimized](<https://devfeed.tech/tags/optimized.md>), [optimizing](<https://devfeed.tech/tags/optimizing.md>), [pandas](<https://devfeed.tech/tags/pandas.md>), [pyspark](<https://devfeed.tech/tags/pyspark.md>), [python](<https://devfeed.tech/tags/python.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [runtime](<https://devfeed.tech/tags/runtime.md>), [spark](<https://devfeed.tech/tags/spark.md>), [sql](<https://devfeed.tech/tags/sql.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

In AWS Glue 6.0 tests on 10M-100M rows, built-in Spark SQL was fastest when applicable; among UDFs, native Arrow led when its logic fit PyArrow compute kernels, while pandas UDFs were the flexible fallback. Native Arrow processes pyarrow arrays directly, avoiding the pandas conversion; pandas UDFs use Arrow transport and batch NumPy work; Arrow-optimized scalar UDFs improve transport but still run Python per row. For math-heavy work at 100M rows, the native Arrow UDF took 3.13 seconds, versus 3.83 for pandas and 11.01 for Arrow-optimized Python. Iterative logic unavailable in SQL or PyArrow kernels left pandas UDFs as the fastest tested option. A controlled test found NumPy through RDD.mapPartitions slower than standard Python UDF, because it materialized Python objects and left DataFrame execution. Results used 10 G.1X workers; absolute timings may vary. Cache handling and warm-up materially affected measurements.

### Source excerpt

Python UDFs in PySpark can be slow because data is serialized between the JVM and the Python runtime. We benchmark five approaches on AWS Glue 6.0 (standard Python, Arrow-optimized Python, vectorized pandas, native Arrow UDFs, and Spark SQL) across 10M to 100M rows, and separate transport cost from compute cost to explain the ranking.

## How Thermo Fisher Scientific built AI-powered patient matching on AWS

DevFeed: [How Thermo Fisher Scientific built AI-powered patient matching on AWS](<https://devfeed.tech/articles/how-thermo-fisher-scientific-built-ai-powered-patient-matching-on-aws-133534.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/how-thermo-fisher-scientific-built-ai-powered-patient-matching-on-aws/>)

Author: Kaushal Patel

Published: 2026-11-13T20:26:58Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [AWS Step Functions](<https://devfeed.tech/topics/aws-step-functions.md>), [Entity resolution](<https://devfeed.tech/topics/entity-resolution.md>), [database-optimization](<https://devfeed.tech/topics/database-optimization.md>)

Tags: [aggregate](<https://devfeed.tech/tags/aggregate.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [ai-workloads](<https://devfeed.tech/tags/ai-workloads.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-documentdb](<https://devfeed.tech/tags/amazon-documentdb.md>), [amazon-elastic-container-service](<https://devfeed.tech/tags/amazon-elastic-container-service.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-simple-storage-service-amazon-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-amazon-s3.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-support](<https://devfeed.tech/tags/aws-support.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data](<https://devfeed.tech/tags/data.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [launch](<https://devfeed.tech/tags/launch.md>), [matching](<https://devfeed.tech/tags/matching.md>), [modernizing](<https://devfeed.tech/tags/modernizing.md>), [platform](<https://devfeed.tech/tags/platform.md>), [premium](<https://devfeed.tech/tags/premium.md>), [production](<https://devfeed.tech/tags/production.md>), [recruitment](<https://devfeed.tech/tags/recruitment.md>), [risk](<https://devfeed.tech/tags/risk.md>)

### AI overview

Thermo Fisher Scientific built Patient Match AI on a curated AWS patient recruitment data foundation that consolidates more than 20 million profiles (about 1.6 TB) from patient, survey, recruitment, marketing, and consent sources. Users can search in natural language and refine cohorts through a COUNT -> REFINE -> TARGETED LIST -> EXPORT workflow; shared test queries ran in under six seconds, and many common feasibility workflows took minutes rather than days. Aggregate views support exploration, while identity access remains subject to privacy, governance, consent, contactability, and PII controls. Central Insights Hub provides separate dashboards for recruitment progress, site and geographic performance, and other operational measures. AWS services include S3, Glue, Step Functions, Athena, Iceberg, DocumentDB, ECS, and Lambda. AWS Countdown Premium provided architecture and readiness reviews, launch support, and post-launch stabilization; its engineering guidance helped cut estimated processing time by more than threefold. The article reports a Q2 FY26 production launch.

### Source excerpt

Learn how Thermo Fisher Scientific built an AI-powered patient-matching platform on AWS, modernizing its clinical trial recruitment data foundation and using AWS Countdown Premium to de-risk the production launch.

## Multi-statement transactions for Apache Iceberg tables in Amazon Redshift

DevFeed: [Multi-statement transactions for Apache Iceberg tables in Amazon Redshift](<https://devfeed.tech/articles/multi-statement-transactions-for-apache-iceberg-tables-in-amazon-redshift-133388.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/multi-statement-transactions-for-apache-iceberg-tables-in-amazon-redshift/>)

Author: Shiyu Xu

Published: 2026-11-13T19:32:46Z

Content type: release

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [atomic](<https://devfeed.tech/tags/atomic.md>), [foundational-100](<https://devfeed.tech/tags/foundational-100.md>), [snapshot](<https://devfeed.tech/tags/snapshot.md>), [transactions](<https://devfeed.tech/tags/transactions.md>)

### AI overview

Amazon Redshift Patch 206 and later can group multiple DML statements against one Apache Iceberg table into a single `BEGIN ... COMMIT`, publishing one atomic snapshot. The transaction can include an audit before commit; failures and rollbacks publish no changes. All writes must target the same table using the same reference name, while reads may include uncommitted writes and each table's version is fixed on its first read. Run Iceberg DDL separately. Uncommitted intermediate files can become orphans; S3 Tables removes unreferenced files by default, while Glue tables require the orphan file deletion optimizer enabled in advance. In the reported test, batching 25 small statements reduced total runtime by 14.77% for INSERT and 11.29% for DELETE; heavy data work showed no meaningful speedup. At 32 concurrent writers, four-INSERT transactions achieved 3.59 times the committed-statement rate of auto-commit. Results vary with statement size, partitioning, and contention.

### Source excerpt

Amazon Redshift now supports multi-statement transactions for Apache Iceberg tables, so you can run several DML statements against one Iceberg table inside a single BEGIN...COMMIT and publish them as one atomic Iceberg snapshot. This post explains why one commit per statement makes multi-step pipelines fragile, what you can run in a transaction, and how it performs.

## Build a self-healing ETL pipeline with Amazon Bedrock AgentCore

DevFeed: [Build a self-healing ETL pipeline with Amazon Bedrock AgentCore](<https://devfeed.tech/articles/build-a-self-healing-etl-pipeline-with-amazon-bedrock-agentcore-133386.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/build-a-self-healing-etl-pipeline-with-amazon-bedrock-agentcore/>)

Author: Kevin Tinn

Published: 2026-11-13T19:08:10Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon Bedrock AgentCore](<https://devfeed.tech/topics/amazon-bedrock-agentcore.md>), [etl](<https://devfeed.tech/topics/etl.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon Bedrock Knowledge Bases](<https://devfeed.tech/topics/amazon-bedrock-knowledge-bases.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [knowledge-engineering](<https://devfeed.tech/topics/knowledge-engineering.md>), [aws-sns](<https://devfeed.tech/topics/aws-sns.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [agentcore](<https://devfeed.tech/tags/agentcore.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-bedrock-agentcore](<https://devfeed.tech/tags/amazon-bedrock-agentcore.md>), [automatically](<https://devfeed.tech/tags/automatically.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [bedrock](<https://devfeed.tech/tags/bedrock.md>), [bedrock-agentcore](<https://devfeed.tech/tags/bedrock-agentcore.md>), [build](<https://devfeed.tech/tags/build.md>), [can](<https://devfeed.tech/tags/can.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [domain](<https://devfeed.tech/tags/domain.md>), [errors](<https://devfeed.tech/tags/errors.md>), [etl](<https://devfeed.tech/tags/etl.md>), [etl-pipeline](<https://devfeed.tech/tags/etl-pipeline.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [self-healing](<https://devfeed.tech/tags/self-healing.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

The described AWS Glue pipeline routes records that fail schema validation to a four-agent system on Amazon Bedrock AgentCore. A correction agent queries a Bedrock Knowledge Base for domain guidance and looks up precomputed strategies in S3 Vectors in parallel, selecting the higher-confidence result. At the default 0.7 threshold, it applies qualifying corrections; otherwise, the orchestrator sends an SNS alert for human review. A persistence agent writes corrected records to both corrected and transformed data lake zones. The sample annotates records with the chosen strategy rather than changing field values; direct transformations require extending its correction function. With repository credentials configured, a remediation agent can analyze the source error and open a GitHub pull request; without them, that step is skipped. The post outlines AWS CDK deployment and sample-data workflow, and notes required AWS services, model access, Python 3.10+, and a container runtime for local testing.

### Source excerpt

Build a self-healing ETL pipeline on AWS Glue and Amazon Bedrock AgentCore that automatically corrects bad records using domain knowledge, writes the fixes back to the data lake, and can open pull requests to prevent the same errors from recurring.

## Migrating from Apache Hudi-based data lake to Apache Iceberg-based data lake

DevFeed: [Migrating from Apache Hudi-based data lake to Apache Iceberg-based data lake](<https://devfeed.tech/articles/migrating-from-apache-hudi-based-data-lake-to-apache-iceberg-based-data-lake-130716.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/migrating-from-apache-hudi-based-data-lake-to-apache-iceberg-based-data-lake/>)

Author: Nitin Kumar

Published: 2026-11-12T15:50:48Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-hudi](<https://devfeed.tech/tags/apache-hudi.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [athena](<https://devfeed.tech/tags/athena.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [based](<https://devfeed.tech/tags/based.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-lakes](<https://devfeed.tech/tags/data-lakes.md>), [files](<https://devfeed.tech/tags/files.md>), [full](<https://devfeed.tech/tags/full.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [hudi](<https://devfeed.tech/tags/hudi.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [insert](<https://devfeed.tech/tags/insert.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The excerpt presents three Hudi-to-Iceberg migration paths on AWS. In-place migration reuses Parquet files, saving rewrite time and storage but retaining the source schema, partitioning, and file layout. Iceberg's `add_files` suits a one-time conversion and requires Hudi cleanup first; for MoR, compact logs and clean old versions to avoid missing updates or duplicate rows. Apache XTable translates Hudi metadata and can sync ongoing writes, but MoR logs must be compacted before each conversion. Both approaches share data files, so avoid destructive Iceberg maintenance until cutover; `add_files` and `snapshot` also lack column statistics in Iceberg 1.9.0-1.10.0, with the issue fixed in 1.10.1. Full migration with CTAS or CREATE plus INSERT rewrites data, allowing schema and partition changes, including hidden partitioning, at higher compute, time, and storage cost. For large tables, partitioned INSERT batches support parallelism and resumability. The excerpt demonstrates AWS Glue notebooks, general S3 buckets, and Amazon S3 Tables, and recommends validating against Hudi before cutover.

### Source excerpt

Learn how to migrate transactional data lakes from Apache Hudi to Apache Iceberg on AWS. This post walks through in-place migration with the Iceberg add_files procedure and Apache XTable, plus full table migration with CTAS and INSERT, using AWS Glue notebooks and Amazon S3 Tables, then validates the results in Amazon Athena.

## Build a data foundation for agentic AI on AWS: From lakehouse to MCP

DevFeed: [Build a data foundation for agentic AI on AWS: From lakehouse to MCP](<https://devfeed.tech/articles/build-a-data-foundation-for-agentic-ai-on-aws-from-lakehouse-to-mcp-129558.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/build-a-data-foundation-for-agentic-ai-on-aws-from-lakehouse-to-mcp/>)

Author: Adrien Lebret

Published: 2026-11-11T21:03:43Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AI-Ready Data](<https://devfeed.tech/topics/ai-ready-data.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Amazon Bedrock Knowledge Bases](<https://devfeed.tech/topics/amazon-bedrock-knowledge-bases.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon Bedrock](<https://devfeed.tech/topics/amazon-bedrock.md>), [AWS IAM](<https://devfeed.tech/topics/aws-iam.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [Amazon OpenSearch Service](<https://devfeed.tech/topics/amazon-opensearch-service.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-data](<https://devfeed.tech/tags/ai-data.md>), [aws](<https://devfeed.tech/tags/aws.md>), [build](<https://devfeed.tech/tags/build.md>), [building](<https://devfeed.tech/tags/building.md>), [context](<https://devfeed.tech/tags/context.md>), [context-engineering](<https://devfeed.tech/tags/context-engineering.md>), [data](<https://devfeed.tech/tags/data.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [governance](<https://devfeed.tech/tags/governance.md>), [interoperability](<https://devfeed.tech/tags/interoperability.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [like](<https://devfeed.tech/tags/like.md>), [mcp](<https://devfeed.tech/tags/mcp.md>), [model](<https://devfeed.tech/tags/model.md>), [model-context-protocol](<https://devfeed.tech/tags/model-context-protocol.md>), [uncategorized](<https://devfeed.tech/tags/uncategorized.md>)

### AI overview

The article presents a four-pillar AWS data foundation for agentic AI: a unified lakehouse, context engineering, interoperability, and governance. Amazon SageMaker Lakehouse and S3 Tables bring operational, analytical, and AI data into a governed foundation with Apache Iceberg, zero-ETL ingestion, and federated queries. Context engineering combines knowledge bases, short- and long-term memory, and tools that fetch selected data, addressing stateless models' limited context windows. For collaboration, A2A supports agent-to-agent communication over HTTP and JSON, while MCP standardizes agent access to resources, tools, and prompts; Amazon Bedrock AgentCore Gateway centralizes connections, authentication, audit logging, and permission-scoped semantic tool discovery. Amazon SageMaker Data Catalog adds dataset discovery, lineage, quality indicators, and publish-and-subscribe access controls. The article recommends establishing the lakehouse and ingestion first, developing context capabilities before scaling orchestration, and treating governance as an ongoing effort from the start.

### Source excerpt

Building a data foundation ready for agentic AI relies on a unified lakehouse and interoperability protocols like Model Context Protocol (MCP). This post walks through four pillars for building an agentic AI data foundation on AWS: a unified lakehouse, context engineering, multi-agent interoperability, and data governance.

## How Samsung Ads federated Snowflake Iceberg tables using Amazon Cognito

DevFeed: [How Samsung Ads federated Snowflake Iceberg tables using Amazon Cognito](<https://devfeed.tech/articles/how-samsung-ads-federated-snowflake-iceberg-tables-using-amazon-cognito-129562.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/how-samsung-ads-federated-snowflake-iceberg-tables-using-amazon-cognito/>)

Author: Balaji Selva Rajan

Published: 2026-11-11T19:48:02Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon Cognito](<https://devfeed.tech/topics/amazon-cognito.md>), [AWS Lake Formation](<https://devfeed.tech/topics/aws-lake-formation.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon VPC](<https://devfeed.tech/topics/amazon-vpc.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [AWS CloudShell](<https://devfeed.tech/topics/aws-cloudshell.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [ads](<https://devfeed.tech/tags/ads.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-cognito](<https://devfeed.tech/tags/amazon-cognito.md>), [amazon-route-53](<https://devfeed.tech/tags/amazon-route-53.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [amazon-sagemaker-ai](<https://devfeed.tech/tags/amazon-sagemaker-ai.md>), [amazon-vpc](<https://devfeed.tech/tags/amazon-vpc.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [authentication](<https://devfeed.tech/tags/authentication.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cli](<https://devfeed.tech/tags/aws-cli.md>), [aws-cloudformation](<https://devfeed.tech/tags/aws-cloudformation.md>), [aws-cloudshell](<https://devfeed.tech/tags/aws-cloudshell.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-lake-formation](<https://devfeed.tech/tags/aws-lake-formation.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [oauth2](<https://devfeed.tech/tags/oauth2.md>), [samsung](<https://devfeed.tech/tags/samsung.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Samsung Ads federated Snowflake-managed Iceberg tables into AWS Lake Formation so Amazon Athena can query the same S3 data without a second copy. An AWS Glue connection obtains OAuth2 client-credentials tokens from Amazon Cognito; a Pre Token Generation Lambda adds the `aud` claim Snowflake requires. The connection reaches Snowflake's Iceberg REST catalog over AWS PrivateLink, while Cognito's public token endpoint requires NAT access. Lake Formation governs access and vends scoped S3 credentials for Athena to read the Parquet files directly. Setup requires Snowflake Business-Critical or higher in the same Region, Snowflake-managed Iceberg tables, and AWS permissions for the Cognito, networking, and Lake Formation resources. The pattern does not apply Snowflake masking or row access policies to Athena reads; those controls must be recreated in Lake Formation and tested. Federate schemas containing only supported Snowflake-managed tables, and explicitly revoke Lake Formation grants when tables are removed. NAT and PrivateLink endpoints incur charges.

### Source excerpt

Learn how Samsung Ads federates Snowflake-managed Apache Iceberg tables into AWS Lake Formation and queries them from Amazon Athena using Amazon Cognito for OAuth2 machine-to-machine authentication, with no second copy of the data to maintain.

## How Intuit migrated from Apache Hive Metastore to AWS Glue Data Catalog

DevFeed: [How Intuit migrated from Apache Hive Metastore to AWS Glue Data Catalog](<https://devfeed.tech/articles/how-intuit-migrated-from-apache-hive-metastore-to-aws-glue-data-catalog-129561.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/how-intuit-migrated-from-apache-hive-metastore-to-aws-glue-data-catalog/>)

Author: Srinivasan Krishnasamy

Published: 2026-11-11T15:43:26Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [AWS Lake Formation](<https://devfeed.tech/topics/aws-lake-formation.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>)

Tags: [access-control](<https://devfeed.tech/tags/access-control.md>), [access-controls](<https://devfeed.tech/tags/access-controls.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-services](<https://devfeed.tech/tags/ai-services.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-aurora](<https://devfeed.tech/tags/amazon-aurora.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-dynamodb](<https://devfeed.tech/tags/amazon-dynamodb.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-hive](<https://devfeed.tech/tags/apache-hive.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data](<https://devfeed.tech/tags/data.md>), [data-catalog](<https://devfeed.tech/tags/data-catalog.md>), [data-discovery](<https://devfeed.tech/tags/data-discovery.md>), [hive](<https://devfeed.tech/tags/hive.md>), [integration](<https://devfeed.tech/tags/integration.md>), [intuit](<https://devfeed.tech/tags/intuit.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>)

### AI overview

Intuit migrated databases, tables, and partitions from Apache Hive Metastore (HMS) to AWS Glue Data Catalog (GDC) using a phased approach: a one-time migration, ongoing synchronization in both directions, hourly parity checks, and targeted repair syncs. A DynamoDB source-of-truth flag per table prevents sync loops. HMS changes flow through Debezium CDC and Kafka, with partition events batched over 10-second windows; GDC changes use EventBridge, Lambda, Kafka, and Flink to order events before applying them to HMS. Before migrating, Intuit recommends auditing unused objects, estimating and load-testing Glue API quotas, batching writes, and using retries with backoff to manage throttling. The migration covered databases, tables, and partitions for S3-backed data; column statistics, privileges, roles, functions, and transactions were out of scope. GDC also does not support database or table renames.

### Source excerpt

How Intuit migrated its data catalog from Apache Hive Metastore to AWS Glue Data Catalog on its journey to an open, secure lakehouse. This post covers the challenges, the phased bidirectional-sync solution, and the benefits: reduced operational overhead, improved data discovery, and tighter integration with AWS analytics services.

## How I Taught an AI to Read My Logs

DevFeed: [How I Taught an AI to Read My Logs](<https://devfeed.tech/articles/how-i-taught-an-ai-to-read-my-logs-127277.md>)

Original publisher: [Read original article](<https://medium.com/expedia-group-tech/how-i-taught-an-ai-to-read-my-logs-9c04f1cfa750?source=rss----38998a53046f---4>)

Author: Vincent DANIEL

Published: 2026-11-10T12:01:02Z

Content type: article

Language: en

Sources: [Expedia](<https://devfeed.tech/sources/expedia.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [aws](<https://devfeed.tech/tags/aws.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [execution-plan](<https://devfeed.tech/tags/execution-plan.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [spark](<https://devfeed.tech/tags/spark.md>)

### AI overview

Expedia Group describes two AI-assisted skills that diagnose Spark jobs from actual execution plans, traces, and catalog metadata instead of relying on configuration or code. The after-the-fact cost analysis correlates stage shuffle, spill and skew data with executor memory and SQL plans, identifying issues such as full-scan MERGE statements, driver or Python memory pressure, and stale Iceberg tables. A Trino MCP connection checks snapshot history and reconciles source DDL with the live catalog. A deployment skill can dry-run risky changes against a non-production table or branch, then monitor live stage progress about every ten seconds and choose recovery steps from the observed trace. The account reports over 90% lower daily compute cost on a couple of pipelines after Iceberg V3 migrations, and says plan analysis also uncovered ongoing data loss on multiple pipelines. Analysis has limits: used pre-signed dashboard URLs need a fresh request, and server-side traces are unavailable for jobs older than roughly 30 days.

### Source excerpt

Expedia Group Technology - DataAI-assisted data engineering at Expedia Group -- and less squinting at Spark execution plansPhoto by Queensland, Australia on Unsplash Anyone who has actually opened a Spark execution plan to figure out why a job is slow or expensive knows what the pain looks like. It's not one screen -- it's the Spark UI's stage DAG, then the SQL tab's physical plan with nested Exchange, BroadcastHashJoin, and SortMergeJoin operators stacked five levels deep, then a separate executors tab to check whether any of that actually correlates with memory pressure, then cross-referencing timestamps against a job's config history to rule out "did someone change something last week." Reading a plan correctly means holding all of that in your head at once, and most of it is stored in different tabs that don't talk to each other. Doing this well takes real practice, and doing it under time pressure -- a pipeline failed overnight, someone needs an answer before standup -- is exactly when people take the fastest wrong guess instead of the right slow one. This is the part of the job that AI turned out to be genuinely good at, and it's the part of the last two years worth focusing on. A brief origin point: Iceberg V3 The first place this paid off concretely was migrating tables to Apache Iceberg's V3 format after it shipped in mid-2025. The work itself was conversational -- reading a table's delete-file accumulation, reading the actual scan plan against that table, deciding whether a merge-on-read migration to deletion vectors was worth it, then checking the resulting plan again to confirm it worked. Daily compute cost on a couple of pipelines that got this treatment dropped by more than 90%. That's a good result, but the more important thing it proved was narrower: that reading a real execution plan and matching it against real table metadata, rather than trusting what the DDL or the job config claimed, was something an AI could do reliably enough to act on. (The deleti

## Automate ETL pipeline creation with Amazon Bedrock AgentCore and AWS Glue

DevFeed: [Automate ETL pipeline creation with Amazon Bedrock AgentCore and AWS Glue](<https://devfeed.tech/articles/automate-etl-pipeline-creation-with-amazon-bedrock-agentcore-and-aws-glue-126345.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/automate-etl-pipeline-creation-with-amazon-bedrock-agentcore-and-aws-glue/>)

Author: Mike Brouwers

Published: 2026-11-09T23:15:31Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AI for data automation](<https://devfeed.tech/topics/ai-for-data-automation.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Amazon Bedrock AgentCore](<https://devfeed.tech/topics/amazon-bedrock-agentcore.md>), [Amazon Bedrock](<https://devfeed.tech/topics/amazon-bedrock.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [Amazon Cognito](<https://devfeed.tech/topics/amazon-cognito.md>), [AWS CloudShell](<https://devfeed.tech/topics/aws-cloudshell.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-bedrock-agentcore](<https://devfeed.tech/tags/amazon-bedrock-agentcore.md>), [amazon-eventbridge](<https://devfeed.tech/tags/amazon-eventbridge.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [anthropic-claude](<https://devfeed.tech/tags/anthropic-claude.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [approval-gates](<https://devfeed.tech/tags/approval-gates.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [etl](<https://devfeed.tech/tags/etl.md>), [etl-pipeline](<https://devfeed.tech/tags/etl-pipeline.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Agentic Data Ingestion automates file or database-table ETL setup by proposing schema mappings, generating PySpark transformations, testing them in AWS Glue Spark, and creating and running AWS Glue jobs through a Step Functions workflow. Human approval gates review mappings and tested code; low-confidence mappings receive extra context, and a retry agent may rerun selected failed jobs while skipping cases where retries could duplicate writes or cannot fix the error. Generated code is kept reviewable with target-specific Jinja2 templates, an allowlist for the model-written transformation block, and checks that required job arguments resolve. Tests use a 100-row file sample in a matching Glue interactive session; database tests use schema-generated rows because the session cannot access databases. In the measured US West (Oregon) sample deployment, runs cost about $0.10-0.25 and idle operation about $15-20 monthly; wider sources and lower-cost models were not measured. The solution requires Claude Sonnet 4.6 access and, outside US regions, a model configuration change; production identity federation and role-based access are listed as future hardening steps.

### Source excerpt

Agentic Data Ingestion is an open-source solution that uses AI agents on Amazon Bedrock to automate data pipeline creation, from file upload to queryable tables. Point it at a file or database table, and foundation models propose the field mapping, generate and test the AWS Glue transform, and run the job behind human approval gates.

## Introducing the Workday Data Cloud integration with AWS

DevFeed: [Introducing the Workday Data Cloud integration with AWS](<https://devfeed.tech/articles/introducing-the-workday-data-cloud-integration-with-aws-126347.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/introducing-the-workday-data-cloud-integration-with-aws/>)

Author: Poornima Chand

Published: 2026-11-09T22:21:40Z

Content type: release

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [AWS Lake Formation](<https://devfeed.tech/topics/aws-lake-formation.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [ai-services](<https://devfeed.tech/tags/ai-services.md>), [amazon-sagemaker-unified-studio](<https://devfeed.tech/tags/amazon-sagemaker-unified-studio.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-integration](<https://devfeed.tech/tags/cloud-integration.md>), [data](<https://devfeed.tech/tags/data.md>), [data-cloud](<https://devfeed.tech/tags/data-cloud.md>), [developers](<https://devfeed.tech/tags/developers.md>), [integration](<https://devfeed.tech/tags/integration.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [money](<https://devfeed.tech/tags/money.md>), [people](<https://devfeed.tech/tags/people.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [services](<https://devfeed.tech/tags/services.md>), [workday](<https://devfeed.tech/tags/workday.md>), [zero-copy](<https://devfeed.tech/tags/zero-copy.md>)

### AI overview

Workday Data Cloud and AWS provide bidirectional, zero-copy access to governed data: Workday can query Iceberg tables in S3 through a federated AWS Glue Data Catalog, while AWS services can query Workday data. For near real-time operational queries, Live Data Query runs SQL within Workday's governed environment and returns results to AWS. For broader analytics, ML training, and exploration, Workday Data Connect exposes data as Apache Iceberg tables; compute runs in AWS while the data stays in Workday. Lake Formation governs AWS-side access, and Workday security controls access to Workday data and federated tables. The article says the integration is generally available and describes setup through SageMaker Unified Studio, Glue, Lake Formation, Athena, EMR, Redshift, and Workday Data Cloud.

### Source excerpt

Workday Data Cloud now integrates with AWS, giving developers bi-directional, zero-copy access to Workday's governed people and money data from the AWS analytics and AI services they already use, without brittle pipelines or data duplication.

## Oracle to S3 as Parquet: Export Tables and Replicate Changes with CDC

DevFeed: [Oracle to S3 as Parquet: Export Tables and Replicate Changes with CDC](<https://devfeed.tech/articles/oracle-to-s3-as-parquet-export-tables-and-replicate-changes-with-cdc-121706.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/oracle-to-s3>)

Author: Seán Whelan

Published: 2026-11-06T12:27:12Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [streaming data to S3 with Parquet](<https://devfeed.tech/topics/streaming-data-to-s3-with-parquet.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [Oracle Database](<https://devfeed.tech/topics/oracle-database.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Debezium Connector](<https://devfeed.tech/topics/debezium-connector.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [changes](<https://devfeed.tech/tags/changes.md>), [compare](<https://devfeed.tech/tags/compare.md>), [dms](<https://devfeed.tech/tags/dms.md>), [export](<https://devfeed.tech/tags/export.md>), [goldengate](<https://devfeed.tech/tags/goldengate.md>), [managed](<https://devfeed.tech/tags/managed.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [replicate](<https://devfeed.tech/tags/replicate.md>), [s3](<https://devfeed.tech/tags/s3.md>), [tables](<https://devfeed.tech/tags/tables.md>)

### AI overview

Oracle-to-S3 Parquet choices depend on whether you need change capture: DMS and log-based CDC tools read redo and capture deletes, while exports and scheduled query jobs see surviving rows only. All approaches must handle Oracle type mapping and immutable Parquet files, where updates and deletes become new rows; current-state queries therefore deduplicate by key and filter the latest delete. The excerpt details DMS setup, including archived redo and supplemental logging, Parquet and timestamp settings, bounded LOB sizing, and file batching. It also describes Estuary's LogMiner capture and S3 materialization: backfill runs alongside redo mining, changes append on a configurable upload interval, and deletes and key changes are represented as rows. Estuary's stated tradeoffs include skipped LOB and several other column types, string or numeric mappings that may need configuration, no hard-delete or truncate handling, and retained historical files. Alternatives include Autonomous Database exports for simple full-query exports, and Spark or Glue for custom transforms; those query-based approaches do not capture deletes.

### Source excerpt

Move Oracle to S3 as Parquet with AWS DMS, AWS Glue, GoldenGate or a managed CDC pipeline. Compare how each handles NUMBER, DATE, LOBs and deletes.

## Salesforce to Iceberg: How to Load CRM Objects and Keep Tables Current

DevFeed: [Salesforce to Iceberg: How to Load CRM Objects and Keep Tables Current](<https://devfeed.tech/articles/salesforce-to-iceberg-how-to-load-crm-objects-and-keep-tables-current-121708.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/salesforce-to-iceberg>)

Author: Seán Whelan

Published: 2026-11-06T12:26:52Z

Content type: tutorial

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [etl](<https://devfeed.tech/topics/etl.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [compare](<https://devfeed.tech/tags/compare.md>), [copy](<https://devfeed.tech/tags/copy.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [load](<https://devfeed.tech/tags/load.md>), [salesforce](<https://devfeed.tech/tags/salesforce.md>), [zero-copy](<https://devfeed.tech/tags/zero-copy.md>)

### AI overview

Keeping Salesforce data current in Iceberg requires an initial Bulk API 2.0 load followed by REST `queryAll` sweeps over `SystemModstamp` and `MERGE INTO` operations keyed on `Id`. Sweeps consume the org's shared API allocation; `IsDeleted` records need explicit treatment, purged records are not returned, and formula fields that depend on parent records need scheduled rereads. The document compares a custom Spark/dlt pipeline, AWS Glue zero-ETL for hourly AWS syncs, Snowflake Openflow (Preview), and a managed pipeline; Data 360 zero copy reads existing Iceberg tables rather than copying Salesforce objects into a catalog you control. It describes an Estuary route that polls every five minutes by default and merges on a 30-minute schedule, with daily formula refreshes and EMR Serverless Spark. Its stated limits include unsupported external and big objects, double-typed currency and percent fields, and variant columns requiring Spark 4 unless disabled. Polling and merge frequency trade freshness against API use, compute, and file maintenance.

### Source excerpt

Salesforce to Iceberg: compare 4 ways to load CRM objects into Iceberg tables, see where Data 360 zero copy fits, and handle deletes and formulas.

## Salesforce to S3: Export Objects as Parquet and Capture Changes

DevFeed: [Salesforce to S3: Export Objects as Parquet and Capture Changes](<https://devfeed.tech/articles/salesforce-to-s3-export-objects-as-parquet-and-capture-changes-121709.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/salesforce-to-s3>)

Author: Seán Whelan

Published: 2026-11-06T12:26:46Z

Content type: tutorial

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>)

Tags: [17](<https://devfeed.tech/tags/17.md>), [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [about](<https://devfeed.tech/tags/about.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [capture](<https://devfeed.tech/tags/capture.md>), [changes](<https://devfeed.tech/tags/changes.md>), [compare](<https://devfeed.tech/tags/compare.md>), [export](<https://devfeed.tech/tags/export.md>), [fields](<https://devfeed.tech/tags/fields.md>), [managed](<https://devfeed.tech/tags/managed.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [python](<https://devfeed.tech/tags/python.md>), [s3](<https://devfeed.tech/tags/s3.md>), [salesforce](<https://devfeed.tech/tags/salesforce.md>), [types](<https://devfeed.tech/tags/types.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Salesforce-to-S3 Parquet exports rely on an initial object load followed by polling `SystemModstamp` or consuming change events; updates append new rows, and deletes appear only when the method includes deleted records. The choice depends on scale and destination: AppFlow suits a few objects in AWS but has a 15 GB per-run cap and needs `preserveSourceDataTyping` enabled to retain types; Glue zero-ETL writes Iceberg tables that can be compacted but omits some objects; Python allows custom logic; managed pipelines suit many objects or destinations. Plan for shared Salesforce API limits, per-object S3 prefixes, and incremental recovery. Formula values can change without `SystemModstamp`, so they need periodic rereads; append-only files then require a query that combines the latest full row with refreshed formula values. Purged records cannot be recovered, and reloading can duplicate rows. Validate types, edits, deletes, and new fields after loading.

### Source excerpt

Move Salesforce to S3 as Parquet with Amazon AppFlow, AWS Glue, Python or a managed pipeline. Compare how each handles types, formula fields and deletes.

## Amazon S3 Tables now offer managed migration of Apache Hive tables to Apache Iceberg

DevFeed: [Amazon S3 Tables now offer managed migration of Apache Hive tables to Apache Iceberg](<https://devfeed.tech/articles/amazon-s3-tables-now-offer-managed-migration-of-apache-hive-tables-to-apache-iceberg-118925.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/aws/amazon-s3-tables-now-offer-managed-migration-of-apache-hive-tables-to-apache-iceberg/>)

Author: Channy Yun (윤석찬)

Published: 2026-11-04T23:34:52Z

Content type: release

Language: en

Sources: [AWS News Blog](<https://devfeed.tech/sources/aws-news-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>)

Tags: [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-hive](<https://devfeed.tech/tags/apache-hive.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [launch](<https://devfeed.tech/tags/launch.md>), [migration](<https://devfeed.tech/tags/migration.md>), [news](<https://devfeed.tech/tags/news.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

Amazon S3 Tables can migrate AWS Glue-registered Hive tables stored as Parquet to managed Apache Iceberg tables in the same AWS account and Region. S3 Tables checks compatibility and source objects before copying, then handles type conversions, recompression, validation, and compaction. The destination remains read-only during migration; you can sync changes repeatedly without stopping source writes, then complete cutover, which takes the source table offline for a few minutes. The service charges for data copied, with no Hive-to-Iceberg conversion fee, and leaves S3 Intelligent-Tiering objects in their current tier. The feature is available in the listed US, Asia Pacific, and European Regions.

### Source excerpt

You can now migrate existing Apache Hive datasets in Amazon S3 to fully managed Apache Iceberg tables in Amazon S3 Tables, with no pipelines to build or compute to provision, while production workloads keep running.

## MongoDB to S3: Parquet Files, Change Streams & Nested Documents Guide

DevFeed: [MongoDB to S3: Parquet Files, Change Streams & Nested Documents Guide](<https://devfeed.tech/articles/mongodb-to-s3-parquet-files-change-streams-nested-documents-guide-118922.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/mongodb-to-s3>)

Author: Seán Whelan

Published: 2026-11-04T22:41:24Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [streaming data to S3 with Parquet](<https://devfeed.tech/topics/streaming-data-to-s3-with-parquet.md>), [MongoDB](<https://devfeed.tech/topics/mongodb.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [AWS Database Migration Service](<https://devfeed.tech/topics/aws-database-migration-service.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>), [DLT](<https://devfeed.tech/topics/data-load-tool.md>), [Amazon DocumentDB](<https://devfeed.tech/topics/amazon-documentdb.md>)

Tags: [change](<https://devfeed.tech/tags/change.md>), [mongodb](<https://devfeed.tech/tags/mongodb.md>), [nested](<https://devfeed.tech/tags/nested.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [parquet-files](<https://devfeed.tech/tags/parquet-files.md>), [s3](<https://devfeed.tech/tags/s3.md>)

### AI overview

MongoDB change streams can feed append-only Parquet files in S3, but updates and deletes become new rows; readers must retain the latest row per `_id` and filter delete markers to represent current state. Change streams require a replica set or sharded cluster and an oplog window long enough to cover outages. A time-window copy or batch job misses deletes. Atlas Data Federation, AWS DMS, custom Spark/Glue/dlt jobs, and managed CDC services differ in setup and how they map nested BSON data; structs, flattened columns, and JSON text are the described options. Schema changes affect later files, and Decimal128 or digit-only strings can lose precision or leading zeros unless written as suitable decimal or string types. Estuary's described route captures changes durably and appends Parquet on a configurable interval, with nested-field depth settings; its Decimal128 and integer inference behavior may require `castToString`. The excerpt also gives S3 permissions, lifecycle, validation, and recovery requirements, including re-reading collections after an oplog history gap.

### Source excerpt

Move MongoDB to S3 as Parquet with Atlas Data Federation, AWS DMS, Spark or a managed pipeline. Compare how each handles nested documents and deletes.

## How Indegene is modernizing and scaling enterprise data quality in life sciences

DevFeed: [How Indegene is modernizing and scaling enterprise data quality in life sciences](<https://devfeed.tech/articles/how-indegene-is-modernizing-and-scaling-enterprise-data-quality-in-life-sciences-117738.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/how-indegene-is-modernizing-and-scaling-enterprise-data-quality-in-life-sciences/>)

Author: Bhagyashree Chandak

Published: 2026-11-04T16:25:57Z

Content type: article

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [AI-Ready Data](<https://devfeed.tech/topics/ai-ready-data.md>), [data-governance](<https://devfeed.tech/topics/data-governance.md>), [Amazon Bedrock Knowledge Bases](<https://devfeed.tech/topics/amazon-bedrock-knowledge-bases.md>), [Amazon Bedrock AgentCore](<https://devfeed.tech/topics/amazon-bedrock-agentcore.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [sensitive data](<https://devfeed.tech/topics/sensitive-data.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-communication](<https://devfeed.tech/tags/agent-communication.md>), [aws](<https://devfeed.tech/tags/aws.md>), [context](<https://devfeed.tech/tags/context.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data](<https://devfeed.tech/tags/data.md>), [data-platform](<https://devfeed.tech/tags/data-platform.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [enterprise-data](<https://devfeed.tech/tags/enterprise-data.md>), [governance](<https://devfeed.tech/tags/governance.md>), [human-in-the-loop](<https://devfeed.tech/tags/human-in-the-loop.md>), [life-sciences](<https://devfeed.tech/tags/life-sciences.md>), [llm](<https://devfeed.tech/tags/llm.md>), [modernizing](<https://devfeed.tech/tags/modernizing.md>), [platform](<https://devfeed.tech/tags/platform.md>), [production](<https://devfeed.tech/tags/production.md>), [python](<https://devfeed.tech/tags/python.md>), [rules](<https://devfeed.tech/tags/rules.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

Indegene's Data Quality Agent in its NEXT Unified Data Platform uses LLMs and two knowledge bases--internal enterprise materials and approved external references--to recommend technical and life sciences business rules from table metadata, profiles, anomalies, existing rules, and domain context. Users select a source such as Amazon Redshift or Snowflake, review and refine suggestions, and approve rules before the agent generates SQL, Python, or PySpark. Generated code receives syntax and schema checks and, where supported, a sample-data test. High-confidence, low-risk rules may be approved automatically; critical and business rules require SME review, with decisions recorded centrally. The system is built around Amazon Bedrock, Strands Agents, and Bedrock AgentCore, with Guardrails for input/output safeguards. Indegene reports reducing rule implementation effort from about 30 to 4 hours per dataset and table onboarding from 2-5 working days to 2-4 hours in its production deployments; these figures are estimated. The text describes current rule generation and governance, while remediation recommendations and root cause analysis are future enhancements.

### Source excerpt

This post explores how Indegene built a production-grade, LLM-powered Data Quality Agent in its NEXT Unified Data Platform on AWS. The agent generates context-aware data quality rules, validates them, and produces ready-to-run SQL and Python under human-in-the-loop governance, helping life sciences organizations scale enterprise data quality.

[Next page](<https://devfeed.tech/topics/aws-glue.md?cursor=WyIyMDI2LTExLTA0VDE2OjI1OjU3KzAwOjAwIiwgIjZmZTUwMjE1LTgzMDUtNDkzNC1hZTVmLTM5N2JhOTY0YzM1MyJd>)