# Amazon S3

Cloud object storage service for storing and retrieving data in Amazon Web Services.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Announcing Amazon Textract Insights: Verifiable generative AI for documents

DevFeed: [Announcing Amazon Textract Insights: Verifiable generative AI for documents](<https://devfeed.tech/articles/announcing-amazon-textract-insights-verifiable-generative-ai-for-documents-146682.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/announcing-amazon-textract-insights-verifiable-generative-ai-for-documents/>)

Author: Keith Mascarenhas

Published: 2026-11-20T20:22:00Z

Content type: release

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [Amazon Textract](<https://devfeed.tech/topics/amazon-textract.md>), [Generative AI](<https://devfeed.tech/topics/generative-ai.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [amazon-textract](<https://devfeed.tech/tags/amazon-textract.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [confidence-score](<https://devfeed.tech/tags/confidence-score.md>), [general-availability](<https://devfeed.tech/tags/general-availability.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [model-training](<https://devfeed.tech/tags/model-training.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [structured](<https://devfeed.tech/tags/structured.md>)

### AI overview

Amazon Textract Insights provides document field extraction through one asynchronous API request: define fields in a JSON Schema blueprint with natural language descriptions, then submit a document or mixed packet from S3. With splitting enabled, it separates packets of up to 3,000 pages, matches subdocuments to blueprints, and returns explicit or inferred values with confidence scores and page and bounding-box grounding. Public blueprints cover common forms; custom blueprints support business-specific documents. This offers a managed alternative to assembling an OCR, splitting, classification, model, and review pipeline, while grounding and confidence support audit and review decisions. The source recommends tuning confidence thresholds on representative documents. Inputs are PDF, PNG, JPEG, or TIFF; supported languages are English, French, German, Italian, Portuguese, and Spanish. Pricing is per page with no minimum commitment. Insights is not yet available in the newly added Tokyo and Zurich Regions; inference requests are routed to compute within the originating geography.

### Source excerpt

Announcing the general availability of Amazon Textract Insights, a new generative AI feature of Amazon Textract. Describe the fields you need in natural language, submit a document or a mixed packet, and receive structured results with a confidence score and page-level grounding for each value, with no model training required.

## Map sensitive data with Datadog Data Security Posture Management

DevFeed: [Map sensitive data with Datadog Data Security Posture Management](<https://devfeed.tech/articles/map-sensitive-data-with-datadog-data-security-posture-management-147018.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/data-security-posture-management/>)

Author: Loïc Bruneau; Kevin Hu; Alexa Levine; Candace Shamieh

Published: 2026-11-20T00:00:00Z

Content type: article

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog-the-monitor-blog.md>)

Topics: [datadog](<https://devfeed.tech/topics/datadog.md>), [sensitive data](<https://devfeed.tech/topics/sensitive-data.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Security](<https://devfeed.tech/topics/security.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>), [rdbms](<https://devfeed.tech/topics/rdbms.md>)

Tags: [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [data-observability](<https://devfeed.tech/tags/data-observability.md>), [data-security](<https://devfeed.tech/tags/data-security.md>), [data-security-posture-management](<https://devfeed.tech/tags/data-security-posture-management.md>), [database-monitoring](<https://devfeed.tech/tags/database-monitoring.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [dspm](<https://devfeed.tech/tags/dspm.md>), [financial-data](<https://devfeed.tech/tags/financial-data.md>), [map](<https://devfeed.tech/tags/map.md>), [pii](<https://devfeed.tech/tags/pii.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [preview](<https://devfeed.tech/tags/preview.md>), [secrets](<https://devfeed.tech/tags/secrets.md>), [security](<https://devfeed.tech/tags/security.md>), [sensitive-data](<https://devfeed.tech/tags/sensitive-data.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>)

### AI overview

Datadog DSPM, available in Preview, locates PII, financial data, and secrets at rest in Snowflake, PostgreSQL, and Amazon S3, showing findings in the Data Security explorer and Datadog pages engineers already use. It reuses Sensitive Data Scanner rules and reports the matched rule, location, and count without retaining or displaying sensitive values. Scanning differs by store: Snowflake sends sampled rows to a Datadog-hosted crawler for in-memory scanning; PostgreSQL scans sampled rows in the existing Datadog Agent; and Amazon S3 uses Agentless Scanning to scan sampled files in the customer's AWS account, with bucket restrictions available. PostgreSQL access may require additional table grants, which DSPM identifies; Fleet Automation can configure managed Agents automatically. Supported S3 formats include CSV, JSON, and Parquet. Planned links to lineage, query activity, and AI agent traces are future work, not current Preview capabilities.

### Source excerpt

Datadog DSPM finds PII, financial data, and secrets at rest in Snowflake, PostgreSQL, and Amazon S3, down to the exact column or file. Now in Preview.

## Amazon S3 Tables with dbt: Iceberg-based data transformations using AWS Glue

DevFeed: [Amazon S3 Tables with dbt: Iceberg-based data transformations using AWS Glue](<https://devfeed.tech/articles/amazon-s3-tables-with-dbt-iceberg-based-data-transformations-using-aws-glue-144273.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/amazon-s3-tables-with-dbt-iceberg-based-data-transformations-using-aws-glue/>)

Author: Yukiko Iwazumi

Published: 2026-11-19T17:58:50Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-quick-sight](<https://devfeed.tech/tags/amazon-quick-sight.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The walkthrough builds a dbt pipeline on Amazon S3 Tables through AWS Glue interactive sessions, with raw CSV data staged and transformed into analytics marts. It requires dbt-glue 1.12.4 or later for S3 Tables incremental models, Python 3.10-3.12, and AWS Glue 6.0 or later. In the example, `fact_orders` uses merge to retain each order's latest state, while `fact_order_history` uses append to retain changes. After source updates, the merge table has six orders, including the updated order 1004 and new order 1006; the history table grows from five to seven rows. The post describes S3 Tables support as experimental in dbt-glue 1.12.4 and recommends testing it in your environment.

### Source excerpt

Amazon S3 Tables, a capability of Amazon S3, bring built-in Apache Iceberg support to object storage. This post shows how to configure and run dbt transformations on S3 Tables with the AWS Glue dbt-glue adapter, using incremental merge and append strategies across a staging and mart architecture.

## S3 audit log streaming with OpenID Connect expands to GHE.com

DevFeed: [S3 audit log streaming with OpenID Connect expands to GHE.com](<https://devfeed.tech/articles/s3-audit-log-streaming-with-openid-connect-expands-to-ghe-com-142929.md>)

Original publisher: [Read original article](<https://github.blog/changelog/2026-11-19-s3-audit-log-streaming-with-openid-connect-expands-to-ghe-com>)

Author: Allison

Published: 2026-11-19T08:27:16Z

Content type: news

Language: en

Sources: [GitHub Changelog](<https://devfeed.tech/sources/github-changelog.md>)

Topics: [OpenID connect (OIDC)](<https://devfeed.tech/topics/oidc.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [audit](<https://devfeed.tech/topics/audit.md>)

Tags: [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [audit](<https://devfeed.tech/tags/audit.md>), [audit-log](<https://devfeed.tech/tags/audit-log.md>), [audit-logs](<https://devfeed.tech/tags/audit-logs.md>), [aws](<https://devfeed.tech/tags/aws.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [credentials](<https://devfeed.tech/tags/credentials.md>), [data](<https://devfeed.tech/tags/data.md>), [data-residency](<https://devfeed.tech/tags/data-residency.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [enterprise-cloud](<https://devfeed.tech/tags/enterprise-cloud.md>), [enterprise-management-tools](<https://devfeed.tech/tags/enterprise-management-tools.md>), [github](<https://devfeed.tech/tags/github.md>), [github-enterprise-cloud](<https://devfeed.tech/tags/github-enterprise-cloud.md>), [improvement](<https://devfeed.tech/tags/improvement.md>), [oidc](<https://devfeed.tech/tags/oidc.md>), [openid](<https://devfeed.tech/tags/openid.md>), [openid-connect](<https://devfeed.tech/tags/openid-connect.md>), [platform-governance](<https://devfeed.tech/tags/platform-governance.md>), [s3](<https://devfeed.tech/tags/s3.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

GitHub Enterprise Cloud with data residency (GHE.com) now supports streaming enterprise audit logs to Amazon S3 using OpenID Connect (OIDC), extending the existing github.com capability. This enables authentication without storing long-lived AWS credentials on GitHub. The excerpt does not describe setup steps or other requirements.

### Source excerpt

You can now stream enterprise audit logs to Amazon S3 using OpenID Connect (OIDC) on GitHub Enterprise Cloud with data residency (GHE.com). This extends the existing github.com capability, letting you... The post S3 audit log streaming with OpenID Connect expands to GHE.com appeared first on The GitHub Blog.

## Finding Amazon S3 cost savings with natural language using AWS Quick and S3 Tables

DevFeed: [Finding Amazon S3 cost savings with natural language using AWS Quick and S3 Tables](<https://devfeed.tech/articles/finding-amazon-s3-cost-savings-with-natural-language-using-aws-quick-and-s3-tables-143610.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/storage/finding-amazon-s3-cost-savings-with-natural-language-using-aws-quick-and-s3-tables/>)

Author: Santosh Setty

Published: 2026-11-18T23:59:46Z

Content type: tutorial

Language: en

Sources: [AWS Storage Blog](<https://devfeed.tech/sources/aws-storage-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [cloud cost management](<https://devfeed.tech/topics/cloud-cost-management.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [text2sql](<https://devfeed.tech/topics/text2sql.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [amazon-simple-storage-service-amazon-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-amazon-s3.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cloud-storage](<https://devfeed.tech/tags/aws-cloud-storage.md>), [aws-quick](<https://devfeed.tech/tags/aws-quick.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cost-savings](<https://devfeed.tech/tags/cost-savings.md>), [costs](<https://devfeed.tech/tags/costs.md>), [data](<https://devfeed.tech/tags/data.md>), [growth](<https://devfeed.tech/tags/growth.md>), [logs](<https://devfeed.tech/tags/logs.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [overhead](<https://devfeed.tech/tags/overhead.md>), [query](<https://devfeed.tech/tags/query.md>), [questions](<https://devfeed.tech/tags/questions.md>), [replication](<https://devfeed.tech/tags/replication.md>), [request](<https://devfeed.tech/tags/request.md>), [s3](<https://devfeed.tech/tags/s3.md>), [s3-tables](<https://devfeed.tech/tags/s3-tables.md>), [tables](<https://devfeed.tech/tags/tables.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Combining S3 Storage Lens metrics with object-level server access logs in S3 Tables lets AWS Quick answer natural-language questions about storage costs across buckets. Storage Lens provides bytes, object counts, and request metrics by storage class and prefix; access logs show object reads but not storage size. The walkthrough exports daily metrics to the aws-s3 table bucket, sends logs through CloudWatch Logs to aws-cloudwatch, then connects both to Quick. Example analyses identify cold prefixes, incomplete multipart uploads, noncurrent versions, cross-Region replication and reads, and rising GET traffic. Setup requires Quick Enterprise, and initial data may take 1-2 days to become useful; logs include sensitive requester and object details. Quick's findings are recommendations: verify the SQL or query in Athena and confirm retention needs before applying permanent expiration rules.

### Source excerpt

Across thousands of S3 buckets, the data you need to cut storage costs sits in different places. Learn how to bring Amazon S3 Storage Lens metrics and server access logs together in S3 Tables and query them with AWS Quick. Find cold data, wasted storage, replication overhead, and request growth by asking questions in natural language.

## High-throughput Transfer Manager for Amazon S3 in Rust is now generally available

DevFeed: [High-throughput Transfer Manager for Amazon S3 in Rust is now generally available](<https://devfeed.tech/articles/high-throughput-transfer-manager-for-amazon-s3-in-rust-is-now-generally-available-144014.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/developer/high-throughput-transfer-manager-for-amazon-s3-in-rust-is-now-generally-available/>)

Author: Nitin Prakash

Published: 2026-11-18T23:48:54Z

Content type: release

Language: en

Sources: [AWS Developer Tools Blog](<https://devfeed.tech/sources/aws-developer-tools-blog.md>)

Topics: [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Rust](<https://devfeed.tech/topics/rust.md>), [Tokio](<https://devfeed.tech/topics/tokio.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>)

Tags: [2025](<https://devfeed.tech/tags/2025.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-simple-storage-service-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-s3.md>), [aws-sdk](<https://devfeed.tech/tags/aws-sdk.md>), [backpressure](<https://devfeed.tech/tags/backpressure.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [buffer-pool](<https://devfeed.tech/tags/buffer-pool.md>), [caching](<https://devfeed.tech/tags/caching.md>), [cargo](<https://devfeed.tech/tags/cargo.md>), [developer-preview](<https://devfeed.tech/tags/developer-preview.md>), [developer-tools](<https://devfeed.tech/tags/developer-tools.md>), [diagnostics](<https://devfeed.tech/tags/diagnostics.md>), [directory](<https://devfeed.tech/tags/directory.md>), [dns](<https://devfeed.tech/tags/dns.md>), [download](<https://devfeed.tech/tags/download.md>), [downloads](<https://devfeed.tech/tags/downloads.md>), [engine](<https://devfeed.tech/tags/engine.md>), [launch](<https://devfeed.tech/tags/launch.md>), [library](<https://devfeed.tech/tags/library.md>), [memory](<https://devfeed.tech/tags/memory.md>), [rust](<https://devfeed.tech/tags/rust.md>), [s3](<https://devfeed.tech/tags/s3.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [uncategorized](<https://devfeed.tech/tags/uncategorized.md>)

### AI overview

The Rust Transfer Manager for Amazon S3 is generally available with a redesigned engine for high-throughput transfers. By default, it starts one worker per detected core, reuses connections across transfers, and uses a fair scheduler intended to keep large transfers from starving small ones; work can be distributed across multiple network interfaces when configured. A shared buffer pool bounds memory, defaulting to up to a quarter of available memory with a 32 GiB cap. Downloads write parts directly to their final positions, while streaming delivers parts in order with backpressure. Optional runtime diagnostics expose buffer snapshots and transfer traces through environment variables. The library also supports `wasm32-wasi*` targets. Getting started requires AWS credentials with the stated S3 permissions, a Tokio-based Rust project, and a bucket. The excerpt does not include the dependency or client setup code.

### Source excerpt

High-throughput transfers to and from Amazon Simple Storage Service (Amazon S3) often require manual tuning of threads, connections, and memory. Transfer Manager for Amazon S3, a high-level transfer library for Rust, handles this work for you. It is now generally available (GA). Since the developer preview launch in March 2025, we have redesigned the library. [...]

## Announcing the developer preview of the AWS SDK for Python v4

DevFeed: [Announcing the developer preview of the AWS SDK for Python v4](<https://devfeed.tech/articles/announcing-the-developer-preview-of-the-aws-sdk-for-python-v4-144012.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/developer/announcing-developer-preview-aws-sdk-for-python-v4/>)

Author: Jonathan Gaytan

Published: 2026-11-18T23:27:52Z

Content type: release

Language: en

Sources: [AWS Developer Tools Blog](<https://devfeed.tech/sources/aws-developer-tools-blog.md>)

Topics: [SDK](<https://devfeed.tech/topics/sdk.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>)

Tags: [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-dynamodb](<https://devfeed.tech/tags/amazon-dynamodb.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-simple-storage-service-amazon-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-amazon-s3.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [async](<https://devfeed.tech/tags/async.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [aws-sdk](<https://devfeed.tech/tags/aws-sdk.md>), [aws-sdk-for-python](<https://devfeed.tech/tags/aws-sdk-for-python.md>), [aws-sdk-python](<https://devfeed.tech/tags/aws-sdk-python.md>), [boto3](<https://devfeed.tech/tags/boto3.md>), [botocore](<https://devfeed.tech/tags/botocore.md>), [clients](<https://devfeed.tech/tags/clients.md>), [developer-preview](<https://devfeed.tech/tags/developer-preview.md>), [developer-tools](<https://devfeed.tech/tags/developer-tools.md>), [dynamodb](<https://devfeed.tech/tags/dynamodb.md>), [event-streaming](<https://devfeed.tech/tags/event-streaming.md>), [modular](<https://devfeed.tech/tags/modular.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [python](<https://devfeed.tech/tags/python.md>), [s3](<https://devfeed.tech/tags/s3.md>), [sdk](<https://devfeed.tech/tags/sdk.md>), [services](<https://devfeed.tech/tags/services.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

AWS SDK for Python v4 is in developer preview, with typed, modular service clients and both synchronous and native async APIs. The excerpt shows installing a service package such as `aws-sdk-s3`, configuring credentials and Region, and making calls with context-managed clients; async calls can run concurrently, and async clients support event streams. Python 3.12 or later, an AWS account, and configured credentials and Region are required. Packages can supply additional credential sources, while bidirectional streaming requires the AWS Common Runtime HTTP/2 client. S3 support includes streaming transfers and flexible checksums. APIs may change before general availability: pin package versions, evaluate in a test environment, and continue using Boto3 for production. Planned additions include more services and credential providers, S3 Transfer Manager, paginators, and waiters.

### Source excerpt

The AWS SDK for Python v4 is now available in developer preview. It provides typed, modular clients for a growing list of AWS services, including Amazon Simple Storage Service (Amazon S3), Amazon DynamoDB, AWS Lambda, and Amazon Bedrock. Clients are available in synchronous and asynchronous (async) versions. Some features, such as event streaming, are currently available [...]

## Long-term Amazon Redshift observability with Datadog and Amazon S3 Tables

DevFeed: [Long-term Amazon Redshift observability with Datadog and Amazon S3 Tables](<https://devfeed.tech/articles/long-term-amazon-redshift-observability-with-datadog-and-amazon-s3-tables-141880.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/long-term-amazon-redshift-observability-with-datadog-and-amazon-s3-tables/>)

Author: Raghu Kuppala

Published: 2026-11-18T22:57:27Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [datadog](<https://devfeed.tech/topics/datadog.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [elt](<https://devfeed.tech/topics/elt.md>)

Tags: [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-emr](<https://devfeed.tech/tags/amazon-emr.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [aws-cli](<https://devfeed.tech/tags/aws-cli.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-identity-and-access-management-iam](<https://devfeed.tech/tags/aws-identity-and-access-management-iam.md>), [aws-key-management-service-aws-kms](<https://devfeed.tech/tags/aws-key-management-service-aws-kms.md>), [data-observability](<https://devfeed.tech/tags/data-observability.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [history](<https://devfeed.tech/tags/history.md>), [integration](<https://devfeed.tech/tags/integration.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [observability](<https://devfeed.tech/tags/observability.md>), [performance](<https://devfeed.tech/tags/performance.md>), [query](<https://devfeed.tech/tags/query.md>)

### AI overview

Amazon Redshift system tables retain history for only 7 days. The S3 Tables integration continuously exports selected system table data as Apache Iceberg tables, letting customers set retention and query the history without collection queries consuming warehouse compute. Datadog Data Observability can read that history to build column-level lineage, normalized query profiles, and context for quality alerts across connected clusters and workgroups. It still connects to Redshift for metadata and runs configured aggregate quality monitors there. Setup requires enabling per-warehouse exports, integrating S3 Tables with the AWS Glue Data Catalog, and granting the Datadog IAM role read access; the feature is described as a preview requiring account-team access. Disabling exports creates a history gap that cannot be backfilled after Redshift's 7-day window, and customer-managed KMS encryption keys cannot be changed after table creation.

### Source excerpt

Amazon Redshift keeps only 7 days of system table history. This post pairs Datadog Data Observability with the new Amazon Redshift system table integration with Amazon S3 Tables, so you can retain query history for the long term and analyze lineage, query performance, and data quality without adding load to your warehouses.

## How to use Amazon Aurora DSQL audit logs

DevFeed: [How to use Amazon Aurora DSQL audit logs](<https://devfeed.tech/articles/how-to-use-amazon-aurora-dsql-audit-logs-141385.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/database/how-to-use-amazon-aurora-dsql-audit-logs/>)

Author: Koji Shinkubo

Published: 2026-11-18T21:33:01Z

Content type: tutorial

Language: en

Sources: [AWS Database Blog](<https://devfeed.tech/sources/aws-database-blog.md>)

Topics: [DSQL](<https://devfeed.tech/topics/dsql.md>), [audit](<https://devfeed.tech/topics/audit.md>), [Logging](<https://devfeed.tech/topics/logging.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [Tool](<https://devfeed.tech/topics/tool.md>), [Query (disambiguation)](<https://devfeed.tech/topics/query.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-aurora](<https://devfeed.tech/tags/amazon-aurora.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [athena](<https://devfeed.tech/tags/athena.md>), [audit](<https://devfeed.tech/tags/audit.md>), [audit-logging](<https://devfeed.tech/tags/audit-logging.md>), [audit-logs](<https://devfeed.tech/tags/audit-logs.md>), [configure](<https://devfeed.tech/tags/configure.md>), [database](<https://devfeed.tech/tags/database.md>), [dsql](<https://devfeed.tech/tags/dsql.md>), [field](<https://devfeed.tech/tags/field.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logs](<https://devfeed.tech/tags/logs.md>), [post](<https://devfeed.tech/tags/post.md>), [query](<https://devfeed.tech/tags/query.md>), [records](<https://devfeed.tech/tags/records.md>), [requirements](<https://devfeed.tech/tags/requirements.md>), [s3](<https://devfeed.tech/tags/s3.md>), [sql](<https://devfeed.tech/tags/sql.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Aurora DSQL audit logging records configured SQL statement classes and delivers records to S3; it is off until both a destination and recording settings are configured. The article describes role-based scope, including READ/SELECT, and a setup that separates audit administration from DBA permissions. Records can identify the IAM role and database role, transaction, statement, and optionally bound values and row counts. Athena can query the S3 logs, while CloudTrail connection events can be joined by session ID to add client IP information; transaction IDs can also link audit records to CDC changes. Limits and tradeoffs include potentially high log volume and charges, no table- or column-level scope, no client IP or success/failure field in audit records, and recording statements from rolled-back transactions. The article recommends testing scope and volume before production and protecting sensitive log content through access controls, encryption, and retention settings.

### Source excerpt

Amazon Aurora DSQL audit logging records the SQL statements that run in your database and delivers them to Amazon S3. This post shows how to configure audit logging to meet regulatory requirements, explains what each recorded field proves in an audit, and shows how to query the logs in Amazon S3 with Amazon Athena.

## Announcing the developer preview of Transfer Manager for Amazon S3 in the AWS SDK for JavaScript v3

DevFeed: [Announcing the developer preview of Transfer Manager for Amazon S3 in the AWS SDK for JavaScript v3](<https://devfeed.tech/articles/announcing-the-developer-preview-of-transfer-manager-for-amazon-s3-in-the-aws-sdk-for-javascript-v3-144013.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/developer/announcing-the-developer-preview-of-transfer-manager-for-amazon-s3-in-the-aws-sdk-for-javascript-v3/>)

Author: Abhinav Goyal

Published: 2026-11-18T21:10:06Z

Content type: release

Language: en

Sources: [AWS Developer Tools Blog](<https://devfeed.tech/sources/aws-developer-tools-blog.md>)

Topics: [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [SDK](<https://devfeed.tech/topics/sdk.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon EC2](<https://devfeed.tech/topics/amazon-ec2.md>), [Amazon Elastic Block Store (Amazon EBS)](<https://devfeed.tech/topics/amazon-elastic-block-store-amazon-ebs.md>), [npm](<https://devfeed.tech/topics/npm.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-simple-storage-service-amazon-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-amazon-s3.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-sdk](<https://devfeed.tech/tags/aws-sdk.md>), [aws-sdk-for-javascript-in-node-js](<https://devfeed.tech/tags/aws-sdk-for-javascript-in-node-js.md>), [aws-sdk-js-v3](<https://devfeed.tech/tags/aws-sdk-js-v3.md>), [checksums](<https://devfeed.tech/tags/checksums.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [developer-preview](<https://devfeed.tech/tags/developer-preview.md>), [developer-tools](<https://devfeed.tech/tags/developer-tools.md>), [developers](<https://devfeed.tech/tags/developers.md>), [downloads](<https://devfeed.tech/tags/downloads.md>), [javascript](<https://devfeed.tech/tags/javascript.md>), [large-datasets](<https://devfeed.tech/tags/large-datasets.md>), [october](<https://devfeed.tech/tags/october.md>), [operations](<https://devfeed.tech/tags/operations.md>), [preview](<https://devfeed.tech/tags/preview.md>), [s3](<https://devfeed.tech/tags/s3.md>), [sdk](<https://devfeed.tech/tags/sdk.md>), [uncategorized](<https://devfeed.tech/tags/uncategorized.md>), [v3](<https://devfeed.tech/tags/v3.md>)

### AI overview

The developer preview of `@aws-sdk/lib-transfer-manager` provides high-level concurrent S3 uploads and downloads, coordinating multipart work, checksums, and transfer controls. Node.js and browser users get object uploads, streamed downloads, configurable concurrency, cancellation, and progress events. Node.js also supports directory transfers and worker-thread parallelism for file and in-memory transfers; stream-based uploads do not yet use workers. Its download benchmark reached about 89% of configured disk bandwidth with `useODirect: true`, while RAM downloads reached about 97% of network bandwidth. Upload throughput was 8-40 times that of `@aws-sdk/lib-storage` in the measured workloads; results are not guarantees and depend on workload and settings. The package is early access and not recommended for production. Planned work includes default range-based downloads, enabling `O_DIRECT` where supported, and worker support for stream uploads.

### Source excerpt

On October 15, 2026, we launched the developer preview of Transfer Manager (@aws-sdk/lib-transfer-manager). It gives JavaScript developers a simpler way to move large datasets to and from Amazon Simple Storage Service (Amazon S3) at high throughput. Transfer Manager provides a set of high-level operations for high-throughput uploads and downloads. It coordinates multipart requests, concurrency, checksums, and [...]

## Orchestrate DuckDB analytics pipelines with Amazon MWAA

DevFeed: [Orchestrate DuckDB analytics pipelines with Amazon MWAA](<https://devfeed.tech/articles/orchestrate-duckdb-analytics-pipelines-with-amazon-mwaa-141881.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/orchestrate-duckdb-analytics-pipelines-with-amazon-mwaa/>)

Author: Vinod Jayendra

Published: 2026-11-18T20:57:09Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [parquet](<https://devfeed.tech/topics/parquet.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-managed-workflows-for-apache-airflow-amazon-mwaa](<https://devfeed.tech/tags/amazon-managed-workflows-for-apache-airflow-amazon-mwaa.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [data](<https://devfeed.tech/tags/data.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [effective](<https://devfeed.tech/tags/effective.md>), [embedded](<https://devfeed.tech/tags/embedded.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [learn-how-to](<https://devfeed.tech/tags/learn-how-to.md>), [model](<https://devfeed.tech/tags/model.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [retry](<https://devfeed.tech/tags/retry.md>), [s3](<https://devfeed.tech/tags/s3.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Amazon MWAA can orchestrate bounded SQL analytics by running DuckDB inside Airflow worker tasks, without a separate query cluster. The walkthrough reads S3 Parquet files, deduplicates events, uses ASOF LEFT JOINs to apply rates and ownership effective at each event time, and merges results into three Iceberg tables in Amazon S3 Tables. A second task validates counts, uniqueness, and allocation states. On supported MWAA Provisioned environments running Airflow 3.3.1, DuckDB and its providers plus the duckdb_default AWS-aware connection are preinstalled; the execution role supplies temporary credentials. The example's unchanged-input reruns retained 300,000 unique events and duplicate-free summaries, but the excerpt says this does not establish recovery from partial failures or concurrent external writes. Each task shares one worker's CPU, memory, and storage, so limits and concurrency need tuning with representative data. Use Glue or EMR for steps needing horizontal scaling or more capacity than one worker provides. Three table commits are not one transaction, and Iceberg update/delete support can vary by version.

### Source excerpt

Learn how to orchestrate embedded DuckDB analytics pipelines with Amazon MWAA. This post shows how to run DuckDB inside Airflow tasks to read Parquet data from Amazon S3, apply time-effective model rates, and publish Apache Iceberg tables to Amazon S3 Tables, with retry-safe publishing and worker-sizing considerations.

## How United Airlines streams database changes to Apache Iceberg with Amazon Data Firehose

DevFeed: [How United Airlines streams database changes to Apache Iceberg with Amazon Data Firehose](<https://devfeed.tech/articles/how-united-airlines-streams-database-changes-to-apache-iceberg-with-amazon-data-firehose-141918.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/how-united-airlines-streams-database-changes-to-apache-iceberg-with-amazon-data-firehose/>)

Author: Guillaume Dupont

Published: 2026-11-18T20:37:39Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Amazon Aurora](<https://devfeed.tech/topics/amazon-aurora.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [streaming data to S3 with Parquet](<https://devfeed.tech/topics/streaming-data-to-s3-with-parquet.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [Visual Studio Code](<https://devfeed.tech/topics/visual-studio-code.md>)

Tags: [airlines](<https://devfeed.tech/tags/airlines.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-aurora](<https://devfeed.tech/tags/amazon-aurora.md>), [amazon-data-firehose](<https://devfeed.tech/tags/amazon-data-firehose.md>), [amazon-kinesis](<https://devfeed.tech/tags/amazon-kinesis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cloudformation](<https://devfeed.tech/tags/aws-cloudformation.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [dms](<https://devfeed.tech/tags/dms.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [kinesis-data-streams](<https://devfeed.tech/tags/kinesis-data-streams.md>), [streams](<https://devfeed.tech/tags/streams.md>), [united-airlines](<https://devfeed.tech/tags/united-airlines.md>)

### AI overview

United Airlines describes a near-real-time CDC pipeline that moves Aurora MySQL changes through AWS DMS, Kinesis Data Streams, Lambda, and Amazon Data Firehose into Apache Iceberg tables on S3. Firehose applies inserts, updates, and deletes as Iceberg snapshots; the open tables can be queried through AWS Glue, Redshift, or Athena. United reports 5x lower average query latency and 2x shorter downstream Glue job processing time in its testing, with a low-volume test reaching about 10 seconds end-to-end. Buffer settings trade latency against cost and file size. The walkthrough uses DMS Serverless, Athena for validation, and a simplified single-table CloudFormation template; United's production setup uses provisioned DMS and primarily Redshift Serverless. The template's shard count and other demo settings are not production-tuned, and schema evolution and multi-table replication require additional handling.

### Source excerpt

United Airlines built a near-real-time change data capture pipeline that replicates reference data from Amazon Aurora MySQL to open Apache Iceberg tables using AWS DMS, Amazon Kinesis Data Streams, and Amazon Data Firehose. This post shows how to reproduce the serverless, multi-engine analytics pattern with AWS CloudFormation.

## AWS named a Leader in IDC MarketScape for public cloud object storage

DevFeed: [AWS named a Leader in IDC MarketScape for public cloud object storage](<https://devfeed.tech/articles/aws-named-a-leader-in-idc-marketscape-for-public-cloud-object-storage-143609.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/storage/aws-named-a-leader-in-idc-marketscape-for-public-cloud-object-storage/>)

Author: Sean White

Published: 2026-11-18T17:56:35Z

Content type: article

Language: en

Sources: [AWS Storage Blog](<https://devfeed.tech/sources/aws-storage-blog.md>)

Topics: [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>)

Tags: [2006](<https://devfeed.tech/tags/2006.md>), [2026](<https://devfeed.tech/tags/2026.md>), [2027](<https://devfeed.tech/tags/2027.md>), [access-control](<https://devfeed.tech/tags/access-control.md>), [access-point](<https://devfeed.tech/tags/access-point.md>), [access-points](<https://devfeed.tech/tags/access-points.md>), [account](<https://devfeed.tech/tags/account.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [amazon-simple-storage-service-amazon-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-amazon-s3.md>), [amazon-simple-storage-service-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-s3.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cloud-storage](<https://devfeed.tech/tags/aws-cloud-storage.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [data](<https://devfeed.tech/tags/data.md>), [foundational-100](<https://devfeed.tech/tags/foundational-100.md>), [idc](<https://devfeed.tech/tags/idc.md>), [leader](<https://devfeed.tech/tags/leader.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [public-cloud](<https://devfeed.tech/tags/public-cloud.md>)

### AI overview

The article presents Amazon S3 as a data foundation for workloads that read directly from storage, including analytics, AI, and agents. It describes backward-compatible APIs, automatic scaling, and strong read-after-write consistency. For durability, it cites replication across at least three Availability Zones for most storage classes, engineering reviews, automated checks, and continuous repair; it also says new objects receive checksums. S3 resources start private, with encryption enabled for new objects and controls such as Block Public Access and IAM policies for granting and reviewing access. Eight storage classes address different access patterns, from single-digit millisecond access to long-term archive. The article also describes S3 Tables for Iceberg, S3 Files for file-system access, S3 Vectors for embeddings, and metadata and annotations for discovery. It reports specific performance and scale figures, but provides no independent comparison with other vendors or assessment details beyond the IDC report citation.

### Source excerpt

AWS has been named a Leader in the IDC MarketScape for North America Public Cloud Object Storage 2026-2027 Vendor Assessment (November 2026). IDC found that object storage is no longer just a place to keep data. Analytics engines, AI pipelines, and agents now read straight from the bucket, so the storage layer affects how those [...]

## S3 vs. ScyllaDB for tiny telemetry: What Red Canary learned the hard way

DevFeed: [S3 vs. ScyllaDB for tiny telemetry: What Red Canary learned the hard way](<https://devfeed.tech/articles/s3-vs-scylladb-for-tiny-telemetry-what-red-canary-learned-the-hard-way-140898.md>)

Original publisher: [Read original article](<https://thenewstack.io/scylladb-s3-tiny-telemetry/>)

Author: Cynthia Dunlop

Published: 2026-11-18T15:00:00Z

Content type: article

Language: en

Sources: [The New Stack](<https://devfeed.tech/sources/the-new-stack.md>)

Topics: [scylladb](<https://devfeed.tech/topics/scylladb.md>), [Endpoint Security & XDR](<https://devfeed.tech/topics/endpoint-security-xdr.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [experience report](<https://devfeed.tech/topics/experience-report.md>)

Tags: [account](<https://devfeed.tech/tags/account.md>), [aws](<https://devfeed.tech/tags/aws.md>), [batch](<https://devfeed.tech/tags/batch.md>), [batching](<https://devfeed.tech/tags/batching.md>), [buffer](<https://devfeed.tech/tags/buffer.md>), [canary](<https://devfeed.tech/tags/canary.md>), [data](<https://devfeed.tech/tags/data.md>), [databases](<https://devfeed.tech/tags/databases.md>), [detection](<https://devfeed.tech/tags/detection.md>), [detection-and-response](<https://devfeed.tech/tags/detection-and-response.md>), [edr](<https://devfeed.tech/tags/edr.md>), [endpoint-detection-and-response](<https://devfeed.tech/tags/endpoint-detection-and-response.md>), [finops](<https://devfeed.tech/tags/finops.md>), [large](<https://devfeed.tech/tags/large.md>), [linux](<https://devfeed.tech/tags/linux.md>), [post-contributed](<https://devfeed.tech/tags/post-contributed.md>), [red](<https://devfeed.tech/tags/red.md>), [s3](<https://devfeed.tech/tags/s3.md>), [scylladb](<https://devfeed.tech/tags/scylladb.md>), [small](<https://devfeed.tech/tags/small.md>), [sponsor-scylladb](<https://devfeed.tech/tags/sponsor-scylladb.md>), [sponsored-post-contributed](<https://devfeed.tech/tags/sponsored-post-contributed.md>), [storage](<https://devfeed.tech/tags/storage.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>), [tiny](<https://devfeed.tech/tags/tiny.md>), [vs](<https://devfeed.tech/tags/vs.md>)

### AI overview

Red Canary reduced costs and improved endpoint data visibility by moving Linux EDR health and telemetry storage from S3 and Postgres to ScyllaDB Cloud. Buffering uploads through Firehose and Kinesis cut S3 PUT costs, but imposed a one-megabyte-per-shard limit, delayed visibility and made endpoint debugging harder; Postgres also struggled with write spikes. In ScyllaDB, the team used short TTLs and faster tombstone cleanup to improve reads, and split large customers across numbered buckets to avoid hot partitions. For telemetry, it partitioned by sensor ID and moved large process-memory arrays into an auxiliary table. A fork of the Rust driver enabled token lookup and execution without serializing statements twice, while separating CPU work from I/O; file processing fell from about 150 ms to 50 ms in this use case. Red Canary reports about 20x fewer writes than inserted rows and endpoint retrieval in seconds rather than minutes for its largest customers. The team is exploring further uses, including detection data; the excerpt does not establish results for those future plans.

### Source excerpt

At Red Canary, storing large amounts of small data is a big deal. Their Linux Endpoint Detection and Response (EDR) The post S3 vs. ScyllaDB for tiny telemetry: What Red Canary learned the hard way appeared first on The New Stack.

## How Mycom optimized its SaaS data fabric with Amazon S3 Access Points and Amazon FSx for NetApp ONTAP

DevFeed: [How Mycom optimized its SaaS data fabric with Amazon S3 Access Points and Amazon FSx for NetApp ONTAP](<https://devfeed.tech/articles/how-mycom-optimized-its-saas-data-fabric-with-amazon-s3-access-points-and-amazon-fsx-for-netapp-ontap-140642.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/storage/how-mycom-optimized-its-saas-data-fabric-with-amazon-s3-access-points-and-amazon-fsx-for-netapp-ontap/>)

Author: Zhiying Lee

Published: 2026-11-17T23:05:19Z

Content type: article

Language: en

Sources: [AWS Storage Blog](<https://devfeed.tech/sources/aws-storage-blog.md>)

Topics: [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Authorization](<https://devfeed.tech/topics/authorization.md>)

Tags: [access-points](<https://devfeed.tech/tags/access-points.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-and-machine-learning](<https://devfeed.tech/tags/ai-and-machine-learning.md>), [ai-applications](<https://devfeed.tech/tags/ai-applications.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-eks](<https://devfeed.tech/tags/amazon-eks.md>), [amazon-elastic-kubernetes-service-amazon-eks](<https://devfeed.tech/tags/amazon-elastic-kubernetes-service-amazon-eks.md>), [amazon-fsx-for-netapp-ontap](<https://devfeed.tech/tags/amazon-fsx-for-netapp-ontap.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-simple-storage-service-amazon-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-amazon-s3.md>), [amazon-simple-storage-service-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-s3.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-pipeline](<https://devfeed.tech/tags/analytics-pipeline.md>), [api](<https://devfeed.tech/tags/api.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [carbon](<https://devfeed.tech/tags/carbon.md>), [cost](<https://devfeed.tech/tags/cost.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data-fabric](<https://devfeed.tech/tags/data-fabric.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [nfs](<https://devfeed.tech/tags/nfs.md>), [saas](<https://devfeed.tech/tags/saas.md>), [telco](<https://devfeed.tech/tags/telco.md>)

### AI overview

Mycom replaced a duplicate S3 analytics copy with Amazon S3 Access Points attached to its Amazon FSx for NetApp ONTAP volume. Operational services continue to use NFS files, while AWS Glue, Athena, and other consumers access the same data through the S3 API, avoiding copy synchronization and application changes. Mycom reported a 20% reduction in analytics storage footprint, with associated cost and carbon reductions, and tighter consistency between operational and analytics views. The design uses existing ONTAP file permissions alongside scoped IAM access point policies; access points are VPC-restricted. Deployment requires two provisioning passes, and existing bucket data is migrated by streaming it to an NFS-mounted volume, followed by a brief producer write pause for the final sync. A configuration switch lets tenants use either the access point or native bucket. A single-AZ volume is an availability, cost, and sustainability tradeoff evaluated per customer and dataset.

### Source excerpt

Mycom collapsed two copies of its telco assurance data (one file store, one object store) into a single copy on Amazon FSx for NetApp ONTAP, served through both NFS and the Amazon S3 API using S3 Access Points. Learn how this removed duplicate storage, reduced cost and carbon, and kept analytics and operational apps consistent.

## Add your organization's runbooks and policies using Retrieval Augmented Generation (RAG) on Amazon Aurora PostgreSQL

DevFeed: [Add your organization's runbooks and policies using Retrieval Augmented Generation (RAG) on Amazon Aurora PostgreSQL](<https://devfeed.tech/articles/add-your-organization-s-runbooks-and-policies-using-retrieval-augmented-generation-rag-on-amazon-aurora-postgresql-138650.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/database/add-your-organizations-runbooks-and-policies-using-retrieval-augmented-generation-rag-on-amazon-aurora-postgresql/>)

Author: Narendra Tawar

Published: 2026-11-17T19:43:40Z

Content type: tutorial

Language: en

Sources: [AWS Database Blog](<https://devfeed.tech/sources/aws-database-blog.md>)

Topics: [Amazon Bedrock Knowledge Bases](<https://devfeed.tech/topics/amazon-bedrock-knowledge-bases.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [Amazon Aurora](<https://devfeed.tech/topics/amazon-aurora.md>), [Vectors & LLMs](<https://devfeed.tech/topics/vectors-llms.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Access Control](<https://devfeed.tech/topics/access-control.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-aurora](<https://devfeed.tech/tags/amazon-aurora.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-bedrock-knowledge-bases](<https://devfeed.tech/tags/amazon-bedrock-knowledge-bases.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [kiro](<https://devfeed.tech/tags/kiro.md>), [pgvector](<https://devfeed.tech/tags/pgvector.md>), [policies](<https://devfeed.tech/tags/policies.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [rag](<https://devfeed.tech/tags/rag.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [retrieval-augmented-generation-rag](<https://devfeed.tech/tags/retrieval-augmented-generation-rag.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The walkthrough connects a Kiro DBA assistant to team runbooks and policies using Amazon Bedrock Knowledge Bases, Amazon S3, and Aurora PostgreSQL with pgvector. Bedrock chunks documents, creates Titan Text Embeddings V2 vectors, and stores them with text in Aurora; at query time, the Retrieval MCP server returns relevant chunks for the assistant to ground its recommendations. Setup requires Aurora PostgreSQL 15.4 or later, pgvector, the RDS Data API, and appropriate Bedrock, S3, and Secrets Manager access. The Knowledge Base needs the `mcp-multirag-kb=true` tag for automatic MCP discovery. For structured policies, section-based custom chunking may improve retrieval precision over default chunking. In the example, retrieval surfaces required peer review, CAB approval, and a manual snapshot before schema changes, while suggesting reader-endpoint traffic redirection as an immediate mitigation. The article also stresses that steering files are soft guardrails; read-only database permissions and IAM deny policies provide additional write protection. Re-sync documents as they change, and restrict `bedrock:Retrieve` access to protect sensitive runbooks.

### Source excerpt

In Part 2 of this series, you connect your Kiro DBA assistant to your team's runbooks and policies. Store runbooks in Amazon S3, index them with Amazon Bedrock Knowledge Bases on Amazon Aurora PostgreSQL using pgvector, and wire up retrieval so the assistant grounds its recommendations in your own documentation.

## Built on object storage, served from cache

DevFeed: [Built on object storage, served from cache](<https://devfeed.tech/articles/built-on-object-storage-served-from-cache-138350.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/built-object-storage-served-cache>)

Author: Carlota Soto

Published: 2026-11-17T13:54:05Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Cache](<https://devfeed.tech/topics/cache.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [branch](<https://devfeed.tech/tags/branch.md>), [cache](<https://devfeed.tech/tags/cache.md>), [compute](<https://devfeed.tech/tags/compute.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [databases](<https://devfeed.tech/tags/databases.md>), [delta](<https://devfeed.tech/tags/delta.md>), [durability](<https://devfeed.tech/tags/durability.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [failover](<https://devfeed.tech/tags/failover.md>), [history](<https://devfeed.tech/tags/history.md>), [image](<https://devfeed.tech/tags/image.md>), [lakebase](<https://devfeed.tech/tags/lakebase.md>), [latency](<https://devfeed.tech/tags/latency.md>), [layer](<https://devfeed.tech/tags/layer.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

Lakebase Postgres uses object storage for durable history without putting it on the routine query read path. Reads check the compute node's local cache first, then pageserver NVMe, and reach object storage only on a pageserver miss; each tier caches fetched data. In the reported production sample, compute caches handled about 99.46% of page accesses, pageserver NVMe about 0.54%, and object storage about 39 requests per second--roughly one in 28 million accesses. Object-store fetches mainly served old branches or restores and databases waking after long idle periods; those initial reads become cached. Pageservers reconstruct requested page versions from image and delta layers, while eagerly creating base images reduced the reported p99 replay work to around six deltas. The excerpt also describes quorum-replicated WAL, asynchronous persistence of materialized history, and branching or restores as pointers to an LSN.

### Source excerpt

The common argument against running Postgres on object storage goes like this:Postgres...

## Deliver petabytes of Apache Kafka data to Amazon S3 at up to 60% lower cost with Amazon MSK Express brokers

DevFeed: [Deliver petabytes of Apache Kafka data to Amazon S3 at up to 60% lower cost with Amazon MSK Express brokers](<https://devfeed.tech/articles/deliver-petabytes-of-apache-kafka-data-to-amazon-s3-at-up-to-60-lower-cost-with-amazon-msk-express-brokers-136179.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/deliver-petabytes-of-apache-kafka-data-to-amazon-s3-at-up-to-60-lower-cost-with-amazon-msk-express-brokers/>)

Author: Diksha Dubey

Published: 2026-11-16T20:42:44Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>)

Tags: [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-managed-streaming-for-apache-kafka-amazon-msk](<https://devfeed.tech/tags/amazon-managed-streaming-for-apache-kafka-amazon-msk.md>), [amazon-msk](<https://devfeed.tech/tags/amazon-msk.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-simple-storage-service-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-s3.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [channel](<https://devfeed.tech/tags/channel.md>), [connect](<https://devfeed.tech/tags/connect.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [delivered](<https://devfeed.tech/tags/delivered.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [express](<https://devfeed.tech/tags/express.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [one](<https://devfeed.tech/tags/one.md>), [post](<https://devfeed.tech/tags/post.md>), [run](<https://devfeed.tech/tags/run.md>), [s3](<https://devfeed.tech/tags/s3.md>), [up](<https://devfeed.tech/tags/up.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Amazon MSK Express brokers deliver Kafka topic records to general purpose Amazon S3 buckets through a channel, without Kafka Connect. The article says delivery is billed per byte, adds no broker egress load, and can cost up to 60% less than self-managed alternatives. Brokers automatically scale storage and rebalance partitions; MSK manages recovery and updates. Channels batch records into S3 objects, preserving producer formats, with configurable compression, storage class, and object key layout. JSON and String records are validated; failures go to an S3 dead-letter queue while delivery continues. The article also describes CloudWatch metrics and logs, setup through the console, APIs, CLI, or an Agent Skill, and recommends overlapping a new channel with an existing connector until delivered data is confirmed. Review supported configurations and prerequisites before production use.

### Source excerpt

Amazon MSK Express brokers now deliver Apache Kafka topic data to Amazon S3 at up to 60% lower cost, with no Kafka Connect to run. You create a channel on a topic and pay only for the data delivered. In this post, we cover the two challenges of delivering Kafka data to Amazon S3 today and how Express brokers and data delivery to Amazon S3 address each one.

## Analyze Anthropic usage, spend, and access with AWS Glue, Amazon S3 Tables, and AWS Quick

DevFeed: [Analyze Anthropic usage, spend, and access with AWS Glue, Amazon S3 Tables, and AWS Quick](<https://devfeed.tech/articles/analyze-anthropic-usage-spend-and-access-with-aws-glue-amazon-s3-tables-and-aws-quick-136178.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/analyze-anthropic-usage-spend-and-access-with-aws-glue-amazon-s3-tables-and-aws-quick/>)

Author: Daniyal Alam

Published: 2026-11-16T20:09:24Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Claude](<https://devfeed.tech/topics/claude.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [anthropic](<https://devfeed.tech/topics/anthropic.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [JOIN](<https://devfeed.tech/topics/join.md>), [data ownership](<https://devfeed.tech/topics/data-ownership.md>), [AWS IAM](<https://devfeed.tech/topics/aws-iam.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [api](<https://devfeed.tech/tags/api.md>), [api-key](<https://devfeed.tech/tags/api-key.md>), [athena](<https://devfeed.tech/tags/athena.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-quick](<https://devfeed.tech/tags/aws-quick.md>), [s3](<https://devfeed.tech/tags/s3.md>), [s3-tables](<https://devfeed.tech/tags/s3-tables.md>), [tables](<https://devfeed.tech/tags/tables.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

AWS Glue's native Anthropic connector can load usage, cost, users, workspaces, and API key data into an AWS lakehouse without custom integration code. The walkthrough stores API keys in Secrets Manager, uses separate Admin and workspace connections for their different data scopes, and writes raw and curated Apache Iceberg tables to Amazon S3 Tables. AWS Glue materialized views join that data with business unit, outcome, and employee reference data; Athena queries the results, and AWS Quick can build dashboards directly from the views. The example surfaces spending by business unit, cost per outcome, and recently used keys created by former employees for ownership review--not automatic evidence of misuse. The article notes API rate limits, recommends bounded incremental reads, and specifies Glue materialized view requirements including Glue 5.1 or later, iam:PassRole, deterministic definitions, and inline subqueries because WITH-based views failed to refresh in its testing.

### Source excerpt

The new AWS Glue connector for Anthropic reads Claude usage, cost, and API key data into AWS without custom code. This post shows how to land it in Amazon S3 Tables as Apache Iceberg tables, join it with your business data, query it with Amazon Athena, and visualize AI spend and governance in AWS Quick.

## Introducing enhanced snapshots for Amazon Managed Service for Apache Flink

DevFeed: [Introducing enhanced snapshots for Amazon Managed Service for Apache Flink](<https://devfeed.tech/articles/introducing-enhanced-snapshots-for-amazon-managed-service-for-apache-flink-136180.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/introducing-enhanced-snapshots-for-amazon-managed-service-for-apache-flink/>)

Author: Francisco Morillo

Published: 2026-11-16T18:17:18Z

Content type: release

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Deployment Strategies](<https://devfeed.tech/topics/deployment-strategies.md>), [zero downtime deployment](<https://devfeed.tech/topics/zero-downtime-deployment.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-managed-service-for-apache-flink](<https://devfeed.tech/tags/amazon-managed-service-for-apache-flink.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [blue-green-deployment](<https://devfeed.tech/tags/blue-green-deployment.md>), [disaster-recovery](<https://devfeed.tech/tags/disaster-recovery.md>), [migrations](<https://devfeed.tech/tags/migrations.md>), [snapshots](<https://devfeed.tech/tags/snapshots.md>)

### AI overview

Enhanced snapshots are standalone AWS resources that let Amazon Managed Service for Apache Flink applications reuse state: start another application from a snapshot, export it to S3, or import a Flink savepoint. The text describes blue/green deployments, second-Region recovery, and migration from self-managed Flink. Blue/green cutover requires accounting for overlapping output and possible duplicates; both applications incur compute while running. Cross-Region recovery uses scheduled exports and S3 replication, trading more frequent exports for lower recovery work and higher operation, storage, and replication costs. Migration and restore require compatible Flink versions and matching operator UIDs; restores also require compatible state types and maxParallelism, and imported savepoints have validation prerequisites. Enhanced snapshots require Flink 1.20 or later, have separate IAM permissions, and persist until deleted, accruing storage charges. The text says snapshot storage is priced like application snapshots, with no added export/import charge; S3 charges still apply.

### Source excerpt

Enhanced snapshots for Amazon Managed Service for Apache Flink are now generally available. An enhanced snapshot is an AWS resource with its own ARN, decoupled from the application that created it, so you can start another application from it, export it to Amazon S3, and import savepoints for blue/green deployments, disaster recovery, and migrations.

## Production MLOps at GS Caltex with Amazon SageMaker Canvas, part 1

DevFeed: [Production MLOps at GS Caltex with Amazon SageMaker Canvas, part 1](<https://devfeed.tech/articles/production-mlops-at-gs-caltex-with-amazon-sagemaker-canvas-part-1-133425.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/production-mlops-at-gs-caltex-with-amazon-sagemaker-canvas-part-1/>)

Author: WonGu Rho

Published: 2026-11-13T23:22:09Z

Content type: article

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [MLOps](<https://devfeed.tech/topics/mlops.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [amazon-sagemaker-canvas](<https://devfeed.tech/tags/amazon-sagemaker-canvas.md>), [batch](<https://devfeed.tech/tags/batch.md>), [canvas](<https://devfeed.tech/tags/canvas.md>), [code](<https://devfeed.tech/tags/code.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [engineers](<https://devfeed.tech/tags/engineers.md>), [inference](<https://devfeed.tech/tags/inference.md>), [minutes](<https://devfeed.tech/tags/minutes.md>), [ml](<https://devfeed.tech/tags/ml.md>), [mlops](<https://devfeed.tech/tags/mlops.md>), [models](<https://devfeed.tech/tags/models.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [post](<https://devfeed.tech/tags/post.md>), [process](<https://devfeed.tech/tags/process.md>), [production](<https://devfeed.tech/tags/production.md>), [reuse](<https://devfeed.tech/tags/reuse.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

GS Caltex connects process engineers' validated SageMaker Canvas models and Data Wrangler flows to SageMaker Pipelines, where ML engineers configure preprocessing, model approval, Batch Transform inference, and scheduling. The handoff reuses validated assets while ML engineers supply production paths, execution order, and deployment configuration. An EventBridge rule can start batch runs every 30 minutes; unlike a continuously running real-time endpoint, Batch Transform releases compute after each job, though full workflow costs and runtime still need measurement. The article highlights practical constraints: inference data may lack a training target column, so a placeholder-and-drop approach is suitable only if the target does not affect features; fixed flow input paths can cause overlapping runs to conflict; and assembled predictions require row-order and count checks. GS Caltex reports deployment time fell from about a week to a few hours, with lower inference costs.

### Source excerpt

GS Caltex connects Amazon SageMaker Canvas to Amazon SageMaker Pipelines so process engineers' no-code models run in production. This post shows how ML engineers reuse validated preprocessing flows and models, cutting deployment from about a week to a few hours and running batch inference every 30 minutes.

## OpenFold3 on Amazon SageMaker JumpStart: Predict biomolecular structures

DevFeed: [OpenFold3 on Amazon SageMaker JumpStart: Predict biomolecular structures](<https://devfeed.tech/articles/openfold3-on-amazon-sagemaker-jumpstart-predict-biomolecular-structures-133424.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/openfold3-on-amazon-sagemaker-jumpstart-predict-biomolecular-structures/>)

Author: Malav Shastri

Published: 2026-11-13T21:53:40Z

Content type: tutorial

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [Amazon SageMaker JumpStart](<https://devfeed.tech/topics/amazon-sagemaker-jumpstart.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [aws-sns](<https://devfeed.tech/topics/aws-sns.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [account](<https://devfeed.tech/tags/account.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [amazon-sagemaker-ai](<https://devfeed.tech/tags/amazon-sagemaker-ai.md>), [amazon-sagemaker-jumpstart](<https://devfeed.tech/tags/amazon-sagemaker-jumpstart.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [aws](<https://devfeed.tech/tags/aws.md>), [cost](<https://devfeed.tech/tags/cost.md>), [deploy](<https://devfeed.tech/tags/deploy.md>), [dna](<https://devfeed.tech/tags/dna.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [jumpstart](<https://devfeed.tech/tags/jumpstart.md>), [multi-gpu](<https://devfeed.tech/tags/multi-gpu.md>), [post](<https://devfeed.tech/tags/post.md>), [preview](<https://devfeed.tech/tags/preview.md>), [proteins](<https://devfeed.tech/tags/proteins.md>), [reduce](<https://devfeed.tech/tags/reduce.md>), [rna](<https://devfeed.tech/tags/rna.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

OpenFold3-preview on SageMaker JumpStart serves protein, DNA, RNA, ligand, and complex structure predictions through asynchronous endpoints, which queue jobs and store results or errors in S3. The article outlines deployment prerequisites, request formats, confidence files, and multi-GPU use: a prediction uses one GPU, while a four-GPU ml.g6.12xlarge can process four requests concurrently. Network isolation prevents calls to external MSA servers; users can supply precomputed alignments generated on infrastructure they control. In the article's carbonic anhydrase II example, an MSA recovered the known binding mode while sequence-only prediction misfolded, but the authors caution this well-studied complex is not a novel-target benchmark. Scale-to-zero can reduce idle instance charges, though restarting adds delay; S3 and other charges still apply. The model is described as a research preview, and the post advises meeting organizational security and compliance requirements before using proprietary or regulated data.

### Source excerpt

Deploy OpenFold3-preview from Amazon SageMaker JumpStart to predict 3D structures of proteins, DNA, RNA, and ligand complexes in your own AWS account. The post covers asynchronous inference, multi-GPU parallelism, bringing your own MSAs for accuracy and privacy, and scaling to zero to reduce cost.

## Monitoring global traffic with AWS Cloud WAN Flow Logs

DevFeed: [Monitoring global traffic with AWS Cloud WAN Flow Logs](<https://devfeed.tech/articles/monitoring-global-traffic-with-aws-cloud-wan-flow-logs-134113.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/networking-and-content-delivery/monitoring-global-traffic-with-aws-cloud-wan-flow-logs/>)

Author: Thaddeus Worsnop

Published: 2026-11-13T21:02:02Z

Content type: tutorial

Language: en

Sources: [Networking & Content Delivery](<https://devfeed.tech/sources/networking-content-delivery.md>)

Topics: [Cloud Networking](<https://devfeed.tech/topics/cloud-networking.md>), [Amazon CloudWatch Logs](<https://devfeed.tech/topics/amazon-cloudwatch-logs.md>), [VPC Flow Logs](<https://devfeed.tech/topics/vpc-flow-logs.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [NAT and SNAT](<https://devfeed.tech/topics/nat-and-snat.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-cloudwatch-logs](<https://devfeed.tech/tags/amazon-cloudwatch-logs.md>), [amazon-data-firehose](<https://devfeed.tech/tags/amazon-data-firehose.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cloud-wan](<https://devfeed.tech/tags/aws-cloud-wan.md>), [cloudwatch](<https://devfeed.tech/tags/cloudwatch.md>), [cross](<https://devfeed.tech/tags/cross.md>), [edge](<https://devfeed.tech/tags/edge.md>), [global-network](<https://devfeed.tech/tags/global-network.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [logs](<https://devfeed.tech/tags/logs.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [network](<https://devfeed.tech/tags/network.md>), [query](<https://devfeed.tech/tags/query.md>), [region](<https://devfeed.tech/tags/region.md>), [s3](<https://devfeed.tech/tags/s3.md>), [scopes](<https://devfeed.tech/tags/scopes.md>)

### AI overview

AWS Cloud WAN Flow Logs provide a core-network view of traffic across VPC and hybrid attachments, including Region, segment, attachment, NAT, and packet-drop context. Logs can be created at core-network, edge, or attachment scope and delivered to CloudWatch Logs, S3, or Data Firehose; Athena and CloudWatch Logs Insights support analysis. Records capture traffic at each in-scope attachment, so a flow may appear more than once; filtering for ingress or a single attachment can avoid double counting. Setup requires the core-network owner account, IAM permissions, active traffic, and AWS CLI 2.40.1 or later. Initial core-network or edge log creation briefly locks the core network in UPDATING; first attachment logs briefly update that attachment. Logs use a fixed one-minute interval and are immutable. Drops inside an inspection VPC are not marked as Cloud WAN drops and require firewall or appliance-interface logs to investigate.

### Source excerpt

AWS Cloud WAN Flow Logs capture flow-level traffic across your global network. Learn how to create flow logs at the core network, edge, and attachment scopes, publish them to Amazon CloudWatch Logs, Amazon S3, or Amazon Data Firehose, and query them with Amazon Athena and CloudWatch Logs Insights to analyze cross-segment and cross-Region traffic and drops.

## How IGNYTE runs candidate search with Amazon S3 Vectors

DevFeed: [How IGNYTE runs candidate search with Amazon S3 Vectors](<https://devfeed.tech/articles/how-ignyte-runs-candidate-search-with-amazon-s3-vectors-133435.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/storage/how-ignyte-runs-candidate-search-with-amazon-s3-vectors/>)

Author: Saurabh Gupta

Published: 2026-11-13T19:52:38Z

Content type: article

Language: en

Sources: [AWS Storage Blog](<https://devfeed.tech/sources/aws-storage-blog.md>)

Topics: [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon OpenSearch Service](<https://devfeed.tech/topics/amazon-opensearch-service.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [vector embeddings in AI](<https://devfeed.tech/topics/vector-embeddings-in-ai.md>), [Amazon Bedrock AgentCore](<https://devfeed.tech/topics/amazon-bedrock-agentcore.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>)

Tags: [agentcore](<https://devfeed.tech/tags/agentcore.md>), [amazon-opensearch-service](<https://devfeed.tech/tags/amazon-opensearch-service.md>), [amazon-sagemaker-ai](<https://devfeed.tech/tags/amazon-sagemaker-ai.md>), [amazon-simple-storage-service-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-s3.md>), [approximate-nearest-neighbor-search](<https://devfeed.tech/tags/approximate-nearest-neighbor-search.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [layout](<https://devfeed.tech/tags/layout.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [semantic](<https://devfeed.tech/tags/semantic.md>)

### AI overview

IGNYTE moved embeddings for more than 1 billion contacts from OpenSearch Service to Amazon S3 Vectors; more than 800 million contacts are currently embedded. OpenSearch Service retains full profiles and handles structured filters. A scheduled batch pipeline loads company records first, then enriches contact records, generates embeddings, and writes profiles and vectors to their respective stores. Vectors are partitioned by continent, with minimal metadata for query filters. Searches embed the recruiter's query, retrieve the 100 closest contacts from the matching S3 Vectors index, then apply skills, experience, and other structured filters in OpenSearch Service. The article reports that the full agent matching workflow averages under 30 seconds after migration, compared with 3-4 minutes before; this is an end-to-end result, not a single vector query.

### Source excerpt

Ignyte moved the embeddings for more than 1 billion contacts from Amazon OpenSearch Service to Amazon S3 Vectors, keeping OpenSearch Service for structured filtering. See the per-continent index layout, the minimal-metadata design, and the batch pipeline behind its two-hop semantic candidate search.

[Next page](<https://devfeed.tech/topics/amazon-s3.md?cursor=WyIyMDI2LTExLTEzVDE5OjUyOjM4KzAwOjAwIiwgIjU1M2U0NWE1LTRlZTctNDQ2OC1iMzA0LWFlNWIzODQ4ZmU3MSJd>)