# automating data pipelines in S3

Published articles for automating data pipelines in S3.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Streaming optimized data to S3 for analytics with Parquet

DevFeed: [Streaming optimized data to S3 for analytics with Parquet](<https://devfeed.tech/articles/streaming-optimized-data-to-s3-for-analytics-with-parquet-12776.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/streaming-data-s3-analytics-parquet>)

Author: Chandler Mayo

Published: 2025-08-13T00:00:00Z

Content type: tutorial

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [parquet](<https://devfeed.tech/topics/parquet.md>), [Redpanda-Connect](<https://devfeed.tech/topics/redpanda-connect.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data analytics](<https://devfeed.tech/topics/data-analytics.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [pandas](<https://devfeed.tech/topics/pandas.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-systems-data-feed](<https://devfeed.tech/tags/analytics-systems-data-feed.md>), [apache-spark-and-athena-data-queries](<https://devfeed.tech/tags/apache-spark-and-athena-data-queries.md>), [athena](<https://devfeed.tech/tags/athena.md>), [automating-data-pipelines-in-s3](<https://devfeed.tech/tags/automating-data-pipelines-in-s3.md>), [building-dashboards-with-s3-data](<https://devfeed.tech/tags/building-dashboards-with-s3-data.md>), [compressing-data-with-parquet](<https://devfeed.tech/tags/compressing-data-with-parquet.md>), [data](<https://devfeed.tech/tags/data.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [data-pipeline-for-analytics](<https://devfeed.tech/tags/data-pipeline-for-analytics.md>), [event-driven-pipelines-with-s3](<https://devfeed.tech/tags/event-driven-pipelines-with-s3.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [learn](<https://devfeed.tech/tags/learn.md>), [pandas](<https://devfeed.tech/tags/pandas.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [parquet-files-in-amazon-s3](<https://devfeed.tech/tags/parquet-files-in-amazon-s3.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [real-time-data-streaming](<https://devfeed.tech/tags/real-time-data-streaming.md>), [redpanda-connect](<https://devfeed.tech/tags/redpanda-connect.md>), [redpanda-data-streaming](<https://devfeed.tech/tags/redpanda-data-streaming.md>), [s3](<https://devfeed.tech/tags/s3.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [setting-up-redpanda-observability](<https://devfeed.tech/tags/setting-up-redpanda-observability.md>), [spark](<https://devfeed.tech/tags/spark.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [streaming-data-to-s3-with-parquet](<https://devfeed.tech/tags/streaming-data-to-s3-with-parquet.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>)

### AI overview

This tutorial explains how to use Redpanda Connect to continuously batch streaming data and write compressed Apache Parquet files to Amazon S3 for analytical workloads. It covers the benefits of Parquet and querying the resulting files with tools such as Pandas, Apache Spark, and Athena.

### Source excerpt

Learn how to build a powerful data pipeline that feeds analytics systems from Redpanda using clean, compressed Parquet files in Amazon S3.