# Building event-driven pipelines with SQS and S3

Published articles for Building event-driven pipelines with SQS and S3.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Building event-driven pipelines with SQS and S3

DevFeed: [Building event-driven pipelines with SQS and S3](<https://devfeed.tech/articles/building-event-driven-pipelines-with-sqs-and-s3-12679.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/building-event-driven-pipelines-sqs-s3>)

Author: Chandler Mayo

Published: 2025-08-21T00:00:00Z

Content type: tutorial

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>), [event driven](<https://devfeed.tech/topics/event-driven.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [amazon-sqs-s3-integration](<https://devfeed.tech/tags/amazon-sqs-s3-integration.md>), [automate-data-ingestion-s3](<https://devfeed.tech/tags/automate-data-ingestion-s3.md>), [building-data-pipelines-with-sqs-and-s3](<https://devfeed.tech/tags/building-data-pipelines-with-sqs-and-s3.md>), [building-event-driven-pipelines-with-sqs-and-s3](<https://devfeed.tech/tags/building-event-driven-pipelines-with-sqs-and-s3.md>), [data-pipeline-automation-with-sqs](<https://devfeed.tech/tags/data-pipeline-automation-with-sqs.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [event-driven-data-processing](<https://devfeed.tech/tags/event-driven-data-processing.md>), [event-driven-pipelines-sqs-s3](<https://devfeed.tech/tags/event-driven-pipelines-sqs-s3.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [real-time-data-pipelines-s3](<https://devfeed.tech/tags/real-time-data-pipelines-s3.md>), [s3-to-redpanda-streaming](<https://devfeed.tech/tags/s3-to-redpanda-streaming.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [set-up-event-driven-architecture-aws](<https://devfeed.tech/tags/set-up-event-driven-architecture-aws.md>), [sqs-s3-data-pipeline-tutorial](<https://devfeed.tech/tags/sqs-s3-data-pipeline-tutorial.md>), [sqs-s3-notifications-for-data-pipelines](<https://devfeed.tech/tags/sqs-s3-notifications-for-data-pipelines.md>), [streaming-data-to-redpanda-from-s3](<https://devfeed.tech/tags/streaming-data-to-redpanda-from-s3.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>)

### AI overview

This tutorial explains how to use S3 Event Notifications and Amazon SQS to trigger Redpanda Connect processing whenever new objects arrive in an S3 bucket. It demonstrates an automated, continuously running pipeline that ingests object-storage data into real-time streams.

### Source excerpt

Learn how to build real-time, event-driven data pipelines directly from your object storage with Amazon SQS and S3.

## Exporting data from Redpanda to S3 in batched JSON

DevFeed: [Exporting data from Redpanda to S3 in batched JSON](<https://devfeed.tech/articles/exporting-data-from-redpanda-to-s3-in-batched-json-12698.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/exporting-data-from-redpanda-to-s3-in-batched-json-arrays>)

Author: Chandler Mayo

Published: 2025-08-07T00:00:00Z

Content type: tutorial

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [JSON](<https://devfeed.tech/topics/json.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Compression](<https://devfeed.tech/topics/compression.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>)

Tags: [batch-json-arrays-to-s3](<https://devfeed.tech/tags/batch-json-arrays-to-s3.md>), [batching](<https://devfeed.tech/tags/batching.md>), [batching-messages-in-s3](<https://devfeed.tech/tags/batching-messages-in-s3.md>), [building-event-driven-pipelines-with-sqs-and-s3](<https://devfeed.tech/tags/building-event-driven-pipelines-with-sqs-and-s3.md>), [data](<https://devfeed.tech/tags/data.md>), [exporting-data-from-redpanda-to-s3](<https://devfeed.tech/tags/exporting-data-from-redpanda-to-s3.md>), [exporting-parquet-files-to-s3](<https://devfeed.tech/tags/exporting-parquet-files-to-s3.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [json](<https://devfeed.tech/tags/json.md>), [optimize-redpanda-s3-pipeline](<https://devfeed.tech/tags/optimize-redpanda-s3-pipeline.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [redpanda-connect](<https://devfeed.tech/tags/redpanda-connect.md>), [redpanda-s3-integration](<https://devfeed.tech/tags/redpanda-s3-integration.md>), [redpanda-to-s3-json-export](<https://devfeed.tech/tags/redpanda-to-s3-json-export.md>), [s3](<https://devfeed.tech/tags/s3.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [setting-up-redpanda-observability-in-datadog](<https://devfeed.tech/tags/setting-up-redpanda-observability-in-datadog.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [streaming-data-to-s3-with-redpanda](<https://devfeed.tech/tags/streaming-data-to-s3-with-redpanda.md>), [streaming-optimized-data-to-s3-for-analytics](<https://devfeed.tech/tags/streaming-optimized-data-to-s3-for-analytics.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>)

### AI overview

A tutorial showing how to configure a Redpanda Connect pipeline to batch multiple Redpanda messages into JSON files stored in Amazon S3. It explains the performance benefits of batching, deployment considerations for Redpanda Serverless, and production tuning options, then points to Parquet for larger analytical workloads.

### Source excerpt

Learn how to write messages from Redpanda to its own JSON file in S3.