# Redpanda-Connect

Redpanda Connect is a declarative data-integration framework with high-performance connectors for building data pipelines.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Drag, drop, done: a visual composer for Redpanda Connect

DevFeed: [Drag, drop, done: a visual composer for Redpanda Connect](<https://devfeed.tech/articles/drag-drop-done-a-visual-composer-for-redpanda-connect-12749.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/redpanda-connect-pipeline-builder>)

Author: Prakhar Garg

Published: 2026-07-23T00:00:00Z

Content type: article

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [Redpanda-Connect](<https://devfeed.tech/topics/redpanda-connect.md>), [Canvas](<https://devfeed.tech/topics/canvas.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Caching](<https://devfeed.tech/topics/caching.md>), [Routing (disambiguation)](<https://devfeed.tech/topics/routing.md>), [HTTP](<https://devfeed.tech/topics/http.md>), [n8n](<https://devfeed.tech/topics/n8n.md>), [workflow automation](<https://devfeed.tech/topics/workflow-automation.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [automation](<https://devfeed.tech/tags/automation.md>), [building](<https://devfeed.tech/tags/building.md>), [cache](<https://devfeed.tech/tags/cache.md>), [caching](<https://devfeed.tech/tags/caching.md>), [events](<https://devfeed.tech/tags/events.md>), [front-end](<https://devfeed.tech/tags/front-end.md>), [http](<https://devfeed.tech/tags/http.md>), [n8n](<https://devfeed.tech/tags/n8n.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [redpanda-connect](<https://devfeed.tech/tags/redpanda-connect.md>), [routing](<https://devfeed.tech/tags/routing.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>), [workflow](<https://devfeed.tech/tags/workflow.md>)

### AI overview

Redpanda introduces a visual Pipeline Builder for Redpanda Connect, using a drag-and-drop canvas to compose pipelines from inputs, processors, and outputs. The builder generates YAML while making routing, enrichment, caching, error handling, and retries easier to understand and configure. The article contrasts it with n8n, emphasizing Redpanda Connect's suitability for high-volume event streaming with ordering, backpressure, and exactly-once semantics.

### Source excerpt

A first look at the new visual Pipeline Builder for Redpanda Connect (preview), plus a roundup of recent CDC and connector updates.

## Stream Oracle changes to ClickHouse in real time

DevFeed: [Stream Oracle changes to ClickHouse in real time](<https://devfeed.tech/articles/stream-oracle-changes-to-clickhouse-in-real-time-12774.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/stream-oracle-changes-to-clickhouse-real-time-cdc>)

Author: Prakhar Garg

Published: 2026-07-07T00:00:00Z

Content type: tutorial

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [clickhouse](<https://devfeed.tech/topics/clickhouse.md>), [Redpanda-Connect](<https://devfeed.tech/topics/redpanda-connect.md>), [Oracle Database](<https://devfeed.tech/topics/oracle-database.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [data](<https://devfeed.tech/topics/data.md>), [Database](<https://devfeed.tech/topics/database.md>), [YAML](<https://devfeed.tech/topics/yaml.md>)

Tags: [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [event](<https://devfeed.tech/tags/event.md>), [integration](<https://devfeed.tech/tags/integration.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [redpanda-connect](<https://devfeed.tech/tags/redpanda-connect.md>), [schema](<https://devfeed.tech/tags/schema.md>), [yaml](<https://devfeed.tech/tags/yaml.md>)

### AI overview

A hands-on walkthrough shows how to stream Oracle change data capture events through Redpanda Connect into ClickHouse in real time. It uses Oracle LogMiner and YAML-configured pipelines, with a Redpanda topic serving as an intermediate store before data is delivered to ClickHouse or other destinations.

### Source excerpt

A hands-on walkthrough of the new Oracle input in Redpanda Connect. No Debezium, no Kafka Connect runtime, no JVM.

## Introducing Iceberg output for Redpanda Connect

DevFeed: [Introducing Iceberg output for Redpanda Connect](<https://devfeed.tech/articles/introducing-iceberg-output-for-redpanda-connect-12746.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/redpanda-connect-apache-iceberg-output>)

Author: Prakhar Garg

Published: 2026-03-05T00:00:00Z

Content type: release

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Redpanda-Connect](<https://devfeed.tech/topics/redpanda-connect.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Routing (disambiguation)](<https://devfeed.tech/topics/routing.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [YAML](<https://devfeed.tech/topics/yaml.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [integration](<https://devfeed.tech/tags/integration.md>), [pii](<https://devfeed.tech/tags/pii.md>), [redpanda-connect](<https://devfeed.tech/tags/redpanda-connect.md>), [routing](<https://devfeed.tech/tags/routing.md>), [schema-evolution](<https://devfeed.tech/tags/schema-evolution.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [yaml](<https://devfeed.tech/tags/yaml.md>)

### AI overview

Redpanda introduces an Iceberg output for Redpanda Connect that writes streaming data directly to Apache Iceberg tables through declarative YAML pipelines. The component supports transformations, enrichment, PII removal, and content-based routing across Redpanda Connect inputs and processors, enabling data from sources such as HTTP webhooks, Postgres CDC streams, and GCP Pub/Sub to reach lakehouse tables with less infrastructure and configuration overhead.

### Source excerpt

Land any data into Apache Iceberg with Redpanda Connect. Maximize infra density, automate schema evolution, and scale 1-to-many routing without the toil.

## Aloha, Redpanda FY27 company kick-off

DevFeed: [Aloha, Redpanda FY27 company kick-off](<https://devfeed.tech/articles/aloha-redpanda-fy27-company-kick-off-12744.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/redpanda-company-kickoff-fy27>)

Author: Jenny Medeiros

Published: 2026-02-24T00:00:00Z

Content type: article

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Redpanda-Connect](<https://devfeed.tech/topics/redpanda-connect.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Disaster Recovery](<https://devfeed.tech/topics/disaster-recovery.md>), [MCP Server](<https://devfeed.tech/topics/mcp-server.md>), [Graphs](<https://devfeed.tech/topics/graphs.md>), [Google](<https://devfeed.tech/topics/google.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [aws](<https://devfeed.tech/tags/aws.md>), [disaster-recovery](<https://devfeed.tech/tags/disaster-recovery.md>), [google](<https://devfeed.tech/tags/google.md>), [graphs](<https://devfeed.tech/tags/graphs.md>), [mcp-server](<https://devfeed.tech/tags/mcp-server.md>), [redpanda-connect](<https://devfeed.tech/tags/redpanda-connect.md>), [serverless](<https://devfeed.tech/tags/serverless.md>)

### AI overview

Redpanda's FY27 company kick-off brought more than 160 employees together in San Diego to reflect on the company's progress and direction. The article highlights the Agentic Data Plane, Redpanda Cloud, Redpanda Serverless on AWS, disaster recovery in Redpanda 25.3, MCP server development, and new Redpanda Connect connectors, alongside talks about company culture and using AI to reshape work.

### Source excerpt

From AI hackathons to hakas, here's what happened at our fourth company kick-off.

## Building with Redpanda Connect: Bloblang and Claude Plugin

DevFeed: [Building with Redpanda Connect: Bloblang and Claude Plugin](<https://devfeed.tech/articles/building-with-redpanda-connect-bloblang-and-claude-plugin-12748.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/redpanda-connect-bloblang-claude-workshop>)

Author: Prakhar Garg

Published: 2026-02-24T00:00:00Z

Content type: tutorial

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [Redpanda-Connect](<https://devfeed.tech/topics/redpanda-connect.md>), [PostgreSQL](<https://devfeed.tech/topics/postgresql.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>), [configuration](<https://devfeed.tech/topics/configuration.md>), [Testing](<https://devfeed.tech/topics/testing.md>), [Code](<https://devfeed.tech/topics/code.md>)

Tags: [claude](<https://devfeed.tech/tags/claude.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [redpanda-connect](<https://devfeed.tech/tags/redpanda-connect.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

A practical workshop on building a production-ready Postgres-to-Redpanda streaming pipeline with Redpanda Connect. It covers Connect's input, processor, and output architecture; PostgreSQL change data capture; Bloblang payload transformations; and the Claude plugin for generating configurations, explaining mappings, and debugging workflows.

### Source excerpt

Join our workshop to learn how to build a production-ready Postgres-to-Redpanda pipeline and see what that simplicity looks like in practice.

## Redpanda Terraform supports managing Redpanda Connect pipelines, private network

DevFeed: [Redpanda Terraform supports managing Redpanda Connect pipelines, private network](<https://devfeed.tech/articles/redpanda-terraform-supports-managing-redpanda-connect-pipelines-private-network-12778.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/terraform-provider-redpanda-cloud>)

Author: David Yu

Published: 2026-02-05T00:00:00Z

Content type: article

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [Terraform](<https://devfeed.tech/topics/terraform.md>), [Redpanda-Connect](<https://devfeed.tech/topics/redpanda-connect.md>), [Provisioning](<https://devfeed.tech/topics/provisioning.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [configuration](<https://devfeed.tech/topics/configuration.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [VPC](<https://devfeed.tech/topics/vpc.md>), [API](<https://devfeed.tech/topics/api.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Command-line interface](<https://devfeed.tech/topics/cli.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [automation](<https://devfeed.tech/tags/automation.md>), [aws](<https://devfeed.tech/tags/aws.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [governance](<https://devfeed.tech/tags/governance.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [provisioning](<https://devfeed.tech/tags/provisioning.md>), [redpanda-connect](<https://devfeed.tech/tags/redpanda-connect.md>), [terraform](<https://devfeed.tech/tags/terraform.md>), [vpc](<https://devfeed.tech/tags/vpc.md>)

### AI overview

This article describes recent additions to the Redpanda Terraform provider for managing Redpanda Cloud clusters and Apache Kafka resources. It covers provisioning, configuration, and management of clusters, topics, schemas, roles, and pipelines, including BYOC and BYOVPC deployments supported on AWS and GCP.

### Source excerpt

Automate Redpanda Cloud with Terraform. Manage Connect pipelines, PrivateLink, and BYOVPC to reduce operational toil and enforce enterprise governance.

## Build production-grade agentic AI with Redpanda Connect and broker auditing

DevFeed: [Build production-grade agentic AI with Redpanda Connect and broker auditing](<https://devfeed.tech/articles/build-production-grade-agentic-ai-with-redpanda-connect-and-broker-auditing-12669.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/agentic-ai-redpanda-connect-audit-logging>)

Author: Redpanda

Published: 2025-12-16T00:00:00Z

Content type: tutorial

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Redpanda-Connect](<https://devfeed.tech/topics/redpanda-connect.md>), [audit trail](<https://devfeed.tech/topics/audit-trail.md>), [observability](<https://devfeed.tech/topics/observability.md>)

Tags: [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [audit](<https://devfeed.tech/tags/audit.md>), [redpanda-connect](<https://devfeed.tech/tags/redpanda-connect.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>)

### AI overview

A guide to building secure and observable agentic AI pipelines on Redpanda, using Connect for AI tools and data access and broker audit logs to record agent actions.

### Source excerpt

Learn how to build secure, observable agentic AI on Redpanda using Connect for AI tools and data access, plus broker audit logs to capture every agent action.

## Streaming IoT and event data into Snowflake and ClickHouse

DevFeed: [Streaming IoT and event data into Snowflake and ClickHouse](<https://devfeed.tech/articles/streaming-iot-and-event-data-into-snowflake-and-clickhouse-12773.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/stream-iot-snowflake-clickhouse>)

Author: Mdu Sibisi

Published: 2025-12-09T00:00:00Z

Content type: tutorial

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [Internet of things](<https://devfeed.tech/topics/iot.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Redpanda-Connect](<https://devfeed.tech/topics/redpanda-connect.md>), [clickhouse](<https://devfeed.tech/topics/clickhouse.md>), [data observability](<https://devfeed.tech/topics/data-observability.md>), [Compression](<https://devfeed.tech/topics/compression.md>), [data-governance](<https://devfeed.tech/topics/data-governance.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [Database](<https://devfeed.tech/topics/database.md>)

Tags: [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [cloud-storage](<https://devfeed.tech/tags/cloud-storage.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [data](<https://devfeed.tech/tags/data.md>), [data-governance](<https://devfeed.tech/tags/data-governance.md>), [event](<https://devfeed.tech/tags/event.md>), [iot](<https://devfeed.tech/tags/iot.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [redpanda-connect](<https://devfeed.tech/tags/redpanda-connect.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

A guide to streaming IoT and event data through Redpanda and Redpanda Connect into Snowflake and ClickHouse. It compares ClickHouse for real-time analysis with Snowflake for scalable cloud storage, historical reporting, and querying, while discussing governance, compression, data freshness, and robust pipeline design.

### Source excerpt

Learn how to stream IoT and event data into Snowflake and ClickHouse using Redpanda

## What's new in Redpanda Cloud: A simpler Serverless now on GCP

DevFeed: [What's new in Redpanda Cloud: A simpler Serverless now on GCP](<https://devfeed.tech/articles/what-s-new-in-redpanda-cloud-a-simpler-serverless-now-on-gcp-12754.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/redpanda-serverless-gcp>)

Author: Towfiqa Yasmeen

Published: 2025-11-25T00:00:00Z

Content type: article

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [Redpanda-Connect](<https://devfeed.tech/topics/redpanda-connect.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [User experience (UX)](<https://devfeed.tech/topics/ux.md>)

Tags: [announcement](<https://devfeed.tech/tags/announcement.md>), [gcp](<https://devfeed.tech/tags/gcp.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [redpanda-connect](<https://devfeed.tech/tags/redpanda-connect.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [usability](<https://devfeed.tech/tags/usability.md>), [ux](<https://devfeed.tech/tags/ux.md>)

### AI overview

Redpanda Serverless introduces a guided onboarding experience, availability on Google Cloud Platform (GCP), and a simpler path to deploying Kafka-compatible streaming. The update targets developers and platform leaders with managed, usage-based streaming that removes broker, cluster, scaling, and infrastructure-management concerns.

### Source excerpt

Redpanda Serverless gives you the shortest path from idea to impact with a new onboarding UX, availability on GCP, and more. Here's what's new.

## Redpanda open-sources top 16 AI connectors

DevFeed: [Redpanda open-sources top 16 AI connectors](<https://devfeed.tech/articles/redpanda-open-sources-top-16-ai-connectors-12764.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/redpanda-top-16-ai-connectors-open-source>)

Author: Mike Broberg

Published: 2025-08-18T00:00:00Z

Content type: article

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [Redpanda-Connect](<https://devfeed.tech/topics/redpanda-connect.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [OpenAI](<https://devfeed.tech/topics/openai.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Amazon Bedrock](<https://devfeed.tech/topics/amazon-bedrock.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [cohere](<https://devfeed.tech/topics/cohere.md>), [Ollama](<https://devfeed.tech/topics/ollama.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-connector-platforms](<https://devfeed.tech/tags/ai-connector-platforms.md>), [ai-connectors-for-commercial-products](<https://devfeed.tech/tags/ai-connectors-for-commercial-products.md>), [ai-connectors-open-source](<https://devfeed.tech/tags/ai-connectors-open-source.md>), [ai-data-streaming](<https://devfeed.tech/tags/ai-data-streaming.md>), [ai-integration-tools](<https://devfeed.tech/tags/ai-integration-tools.md>), [announcement](<https://devfeed.tech/tags/announcement.md>), [bedrock](<https://devfeed.tech/tags/bedrock.md>), [cloud-native-ai-services](<https://devfeed.tech/tags/cloud-native-ai-services.md>), [cohere](<https://devfeed.tech/tags/cohere.md>), [embeddable-ai-capabilities](<https://devfeed.tech/tags/embeddable-ai-capabilities.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [ollama](<https://devfeed.tech/tags/ollama.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [open-source-ai-for-business](<https://devfeed.tech/tags/open-source-ai-for-business.md>), [openai](<https://devfeed.tech/tags/openai.md>), [openai-connectors](<https://devfeed.tech/tags/openai-connectors.md>), [rag](<https://devfeed.tech/tags/rag.md>), [real-time-ai-streaming](<https://devfeed.tech/tags/real-time-ai-streaming.md>), [redpanda-ai-connectors](<https://devfeed.tech/tags/redpanda-ai-connectors.md>), [redpanda-connect](<https://devfeed.tech/tags/redpanda-connect.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [streaming-ai-processors](<https://devfeed.tech/tags/streaming-ai-processors.md>)

### AI overview

Redpanda announces the open-source release of its top AI connectors under the Apache 2.0 license. The connectors integrate Redpanda Connect with destinations and models including OpenAI, Cohere, Amazon Bedrock, Ollama, and Google Cloud Vertex AI, supporting streaming pipelines and use cases such as generation, summarization, classification, translation, and text embeddings for RAG.

### Source excerpt

Redpanda open-sources top AI connectors to the most used destinations, including OpenAI, Cohere, Bedrock, Ollama, and Vertex AI. Learn more.

## Streaming optimized data to S3 for analytics with Parquet

DevFeed: [Streaming optimized data to S3 for analytics with Parquet](<https://devfeed.tech/articles/streaming-optimized-data-to-s3-for-analytics-with-parquet-12776.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/streaming-data-s3-analytics-parquet>)

Author: Chandler Mayo

Published: 2025-08-13T00:00:00Z

Content type: tutorial

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [parquet](<https://devfeed.tech/topics/parquet.md>), [Redpanda-Connect](<https://devfeed.tech/topics/redpanda-connect.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data analytics](<https://devfeed.tech/topics/data-analytics.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [pandas](<https://devfeed.tech/topics/pandas.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-systems-data-feed](<https://devfeed.tech/tags/analytics-systems-data-feed.md>), [apache-spark-and-athena-data-queries](<https://devfeed.tech/tags/apache-spark-and-athena-data-queries.md>), [athena](<https://devfeed.tech/tags/athena.md>), [automating-data-pipelines-in-s3](<https://devfeed.tech/tags/automating-data-pipelines-in-s3.md>), [building-dashboards-with-s3-data](<https://devfeed.tech/tags/building-dashboards-with-s3-data.md>), [compressing-data-with-parquet](<https://devfeed.tech/tags/compressing-data-with-parquet.md>), [data](<https://devfeed.tech/tags/data.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [data-pipeline-for-analytics](<https://devfeed.tech/tags/data-pipeline-for-analytics.md>), [event-driven-pipelines-with-s3](<https://devfeed.tech/tags/event-driven-pipelines-with-s3.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [learn](<https://devfeed.tech/tags/learn.md>), [pandas](<https://devfeed.tech/tags/pandas.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [parquet-files-in-amazon-s3](<https://devfeed.tech/tags/parquet-files-in-amazon-s3.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [real-time-data-streaming](<https://devfeed.tech/tags/real-time-data-streaming.md>), [redpanda-connect](<https://devfeed.tech/tags/redpanda-connect.md>), [redpanda-data-streaming](<https://devfeed.tech/tags/redpanda-data-streaming.md>), [s3](<https://devfeed.tech/tags/s3.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [setting-up-redpanda-observability](<https://devfeed.tech/tags/setting-up-redpanda-observability.md>), [spark](<https://devfeed.tech/tags/spark.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [streaming-data-to-s3-with-parquet](<https://devfeed.tech/tags/streaming-data-to-s3-with-parquet.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>)

### AI overview

This tutorial explains how to use Redpanda Connect to continuously batch streaming data and write compressed Apache Parquet files to Amazon S3 for analytical workloads. It covers the benefits of Parquet and querying the resulting files with tools such as Pandas, Apache Spark, and Athena.

### Source excerpt

Learn how to build a powerful data pipeline that feeds analytics systems from Redpanda using clean, compressed Parquet files in Amazon S3.

## Writing data from Redpanda to S3

DevFeed: [Writing data from Redpanda to S3](<https://devfeed.tech/articles/writing-data-from-redpanda-to-s3-12782.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/writing-data-redpanda-amazon-s3>)

Author: Chandler Mayo

Published: 2025-07-29T00:00:00Z

Content type: tutorial

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [Redpanda-Connect](<https://devfeed.tech/topics/redpanda-connect.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [JSON](<https://devfeed.tech/topics/json.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [configuration](<https://devfeed.tech/topics/configuration.md>), [data observability](<https://devfeed.tech/topics/data-observability.md>), [debug](<https://devfeed.tech/topics/debug.md>)

Tags: [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [batch-messages-from-redpanda-to-s3](<https://devfeed.tech/tags/batch-messages-from-redpanda-to-s3.md>), [event-driven-pipelines-with-redpanda-and-s3](<https://devfeed.tech/tags/event-driven-pipelines-with-redpanda-and-s3.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [json](<https://devfeed.tech/tags/json.md>), [optimizing-redpanda-data-for-s3](<https://devfeed.tech/tags/optimizing-redpanda-data-for-s3.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [redpanda-community-and-s3-integration](<https://devfeed.tech/tags/redpanda-community-and-s3-integration.md>), [redpanda-connect](<https://devfeed.tech/tags/redpanda-connect.md>), [redpanda-json-files-to-s3](<https://devfeed.tech/tags/redpanda-json-files-to-s3.md>), [redpanda-s3-bucket-configuration](<https://devfeed.tech/tags/redpanda-s3-bucket-configuration.md>), [redpanda-to-amazon-s3-data-streaming](<https://devfeed.tech/tags/redpanda-to-amazon-s3-data-streaming.md>), [redpanda-to-s3-integration](<https://devfeed.tech/tags/redpanda-to-s3-integration.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [setting-up-redpanda-with-amazon-s3](<https://devfeed.tech/tags/setting-up-redpanda-with-amazon-s3.md>), [stream-data-to-amazon-s3-from-redpanda](<https://devfeed.tech/tags/stream-data-to-amazon-s3-from-redpanda.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>), [writing-data-from-redpanda-to-s3](<https://devfeed.tech/tags/writing-data-from-redpanda-to-s3.md>)

### AI overview

This tutorial shows how to use Redpanda Connect to stream messages from a Redpanda topic into Amazon S3, writing each message as an individual JSON file. It covers deploying a YAML pipeline, configuring its input and output, and notes that the pattern is suited to small datasets, debugging, and archives rather than production-scale throughput.

### Source excerpt

Learn the easiest way to stream data into Amazon S3 from Redpanda.

## IoT for fun and Prophet: Scaling IoT and predicting the future

DevFeed: [IoT for fun and Prophet: Scaling IoT and predicting the future](<https://devfeed.tech/articles/iot-for-fun-and-prophet-scaling-iot-and-predicting-the-future-12768.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/scaling-iot-iceberg-prophet>)

Author: Bryan Wood

Published: 2025-07-22T00:00:00Z

Content type: tutorial

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Internet of things](<https://devfeed.tech/topics/iot.md>), [Redpanda-Connect](<https://devfeed.tech/topics/redpanda-connect.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Time Series](<https://devfeed.tech/topics/time-series.md>), [ESP32](<https://devfeed.tech/topics/esp32.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [event driven](<https://devfeed.tech/topics/event-driven.md>), [Hardware](<https://devfeed.tech/topics/hardware.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-for-scalable-storage](<https://devfeed.tech/tags/apache-iceberg-for-scalable-storage.md>), [aws](<https://devfeed.tech/tags/aws.md>), [core](<https://devfeed.tech/tags/core.md>), [devices](<https://devfeed.tech/tags/devices.md>), [esp32-mqtt-iot-example](<https://devfeed.tech/tags/esp32-mqtt-iot-example.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [integrating-iceberg-and-redpanda](<https://devfeed.tech/tags/integrating-iceberg-and-redpanda.md>), [iot](<https://devfeed.tech/tags/iot.md>), [iot-data-pipeline-architecture](<https://devfeed.tech/tags/iot-data-pipeline-architecture.md>), [iot-predictive-analytics](<https://devfeed.tech/tags/iot-predictive-analytics.md>), [iot-sensor-data-forecasting](<https://devfeed.tech/tags/iot-sensor-data-forecasting.md>), [predictive-analysis-with-iot](<https://devfeed.tech/tags/predictive-analysis-with-iot.md>), [prophet-forecasting-for-iot](<https://devfeed.tech/tags/prophet-forecasting-for-iot.md>), [real-time-iot-data-streaming](<https://devfeed.tech/tags/real-time-iot-data-streaming.md>), [redpanda-connect](<https://devfeed.tech/tags/redpanda-connect.md>), [redpanda-connect-for-iot](<https://devfeed.tech/tags/redpanda-connect-for-iot.md>), [s3](<https://devfeed.tech/tags/s3.md>), [scaling-iot-with-prophet](<https://devfeed.tech/tags/scaling-iot-with-prophet.md>), [schema-evolution](<https://devfeed.tech/tags/schema-evolution.md>), [schema-management-in-iot](<https://devfeed.tech/tags/schema-management-in-iot.md>), [sensor](<https://devfeed.tech/tags/sensor.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [thought-leadership](<https://devfeed.tech/tags/thought-leadership.md>)

### AI overview

This tutorial presents a scalable IoT data pipeline for real-time streaming and forecasting. It combines Redpanda and Redpanda Connect for ingestion and messaging, Apache Iceberg for scalable time-series data storage, AWS IoT for device-to-cloud communication, and ESP32 hardware for affordable connected-device experiments and deployments.

### Source excerpt

Check out this real-world example of scaling IoT for predictive analysis with Redpanda, Iceberg, and Prophet--without high costs or complexity.

## Stream financial data to Snowflake for real-time analytics | Redpanda

DevFeed: [Stream financial data to Snowflake for real-time analytics | Redpanda](<https://devfeed.tech/articles/stream-financial-data-to-snowflake-for-real-time-analytics-redpanda-12772.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/stream-financial-data-snowflake-real-time-analytics>)

Author: Rajkumar Venkatasamy

Published: 2025-06-03T00:00:00Z

Content type: tutorial

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [Redpanda-Connect](<https://devfeed.tech/topics/redpanda-connect.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [data](<https://devfeed.tech/topics/data.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [Python](<https://devfeed.tech/topics/python.md>), [make](<https://devfeed.tech/topics/make.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [financial-data-streaming-solutions](<https://devfeed.tech/tags/financial-data-streaming-solutions.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [no-code](<https://devfeed.tech/tags/no-code.md>), [no-code-snowflake-data-streaming](<https://devfeed.tech/tags/no-code-snowflake-data-streaming.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [portfolio-management-real-time-data](<https://devfeed.tech/tags/portfolio-management-real-time-data.md>), [python](<https://devfeed.tech/tags/python.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [real-time-analytics-with-snowflake](<https://devfeed.tech/tags/real-time-analytics-with-snowflake.md>), [redpanda-connect](<https://devfeed.tech/tags/redpanda-connect.md>), [redpanda-snowflake-integration](<https://devfeed.tech/tags/redpanda-snowflake-integration.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [snowflake-connector-for-financial-data](<https://devfeed.tech/tags/snowflake-connector-for-financial-data.md>), [snowflake-for-compliance-monitoring](<https://devfeed.tech/tags/snowflake-for-compliance-monitoring.md>), [snowflake-real-time-data-processing](<https://devfeed.tech/tags/snowflake-real-time-data-processing.md>), [snowflake-streaming-data-architecture](<https://devfeed.tech/tags/snowflake-streaming-data-architecture.md>), [stream-financial-data-to-snowflake](<https://devfeed.tech/tags/stream-financial-data-to-snowflake.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [streaming-financial-data-analytics](<https://devfeed.tech/tags/streaming-financial-data-analytics.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>)

### AI overview

A tutorial showing how to capture financial transactions and stream them to Snowflake for real-time analysis using Redpanda Serverless and Redpanda Connect. It uses a Python producer application and a connector to ingest stock-market data into a Snowflake table.

### Source excerpt

Learn how to set up a real-time data stream into Snowflake for real-time insights using Redpanda's blazing-fast Snowflake connector.