# Automate ETL pipeline creation with Amazon Bedrock AgentCore and AWS Glue

DevFeed: [Automate ETL pipeline creation with Amazon Bedrock AgentCore and AWS Glue](<https://devfeed.tech/articles/automate-etl-pipeline-creation-with-amazon-bedrock-agentcore-and-aws-glue-126345.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/automate-etl-pipeline-creation-with-amazon-bedrock-agentcore-and-aws-glue/>)

Author: Mike Brouwers

Published: 2026-11-09T23:15:31Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AI for data automation](<https://devfeed.tech/topics/ai-for-data-automation.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Amazon Bedrock AgentCore](<https://devfeed.tech/topics/amazon-bedrock-agentcore.md>), [Amazon Bedrock](<https://devfeed.tech/topics/amazon-bedrock.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [Amazon Cognito](<https://devfeed.tech/topics/amazon-cognito.md>), [AWS CloudShell](<https://devfeed.tech/topics/aws-cloudshell.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-bedrock-agentcore](<https://devfeed.tech/tags/amazon-bedrock-agentcore.md>), [amazon-eventbridge](<https://devfeed.tech/tags/amazon-eventbridge.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [anthropic-claude](<https://devfeed.tech/tags/anthropic-claude.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [approval-gates](<https://devfeed.tech/tags/approval-gates.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [etl](<https://devfeed.tech/tags/etl.md>), [etl-pipeline](<https://devfeed.tech/tags/etl-pipeline.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

## AI overview

Agentic Data Ingestion automates file or database-table ETL setup by proposing schema mappings, generating PySpark transformations, testing them in AWS Glue Spark, and creating and running AWS Glue jobs through a Step Functions workflow. Human approval gates review mappings and tested code; low-confidence mappings receive extra context, and a retry agent may rerun selected failed jobs while skipping cases where retries could duplicate writes or cannot fix the error. Generated code is kept reviewable with target-specific Jinja2 templates, an allowlist for the model-written transformation block, and checks that required job arguments resolve. Tests use a 100-row file sample in a matching Glue interactive session; database tests use schema-generated rows because the session cannot access databases. In the measured US West (Oregon) sample deployment, runs cost about $0.10-0.25 and idle operation about $15-20 monthly; wider sources and lower-cost models were not measured. The solution requires Claude Sonnet 4.6 access and, outside US regions, a model configuration change; production identity federation and role-based access are listed as future hardening steps.

## Source excerpt

Agentic Data Ingestion is an open-source solution that uses AI agents on Amazon Bedrock to automate data pipeline creation, from file upload to queryable tables. Point it at a file or database table, and foundation models propose the field mapping, generate and test the AWS Glue transform, and run the job behind human approval gates.