# Redpanda-Connect Published articles for Redpanda-Connect. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Streamhouse Working Group proposes a vendor-neutral architecture for real-time data infrastructure DevFeed: [Streamhouse Working Group proposes a vendor-neutral architecture for real-time data infrastructure]() Original publisher: [Read original article]() Author: Alexander Gallego Published: 2026-09-15T00:00:00Z Content type: opinion Language: en Sources: [Redpanda]() Topics: [Data Infrastructure](), [real-time](), [Streaming](), [Artificial Intelligence](), [Redpanda-Connect]() Tags: [ai](), [announcement](), [data-infrastructure](), [real-time](), [redpanda-connect](), [streaming]() ### AI overview Redpanda says it has joined Aiven, Confluent, StreamNative, and Ververica to form the Streamhouse Working Group, which proposes a vendor-neutral architecture for real-time operational data infrastructure. The article distinguishes Streamhouse from the lakehouse by focusing on continuously processing current data for production applications and AI agents. ### Source excerpt Redpanda has joined Aiven, Confluent, StreamNative, and Ververica to form the Streamhouse Working Group. Here's what that means for the future of real-time data infrastructure. ## Drag, drop, done: a visual composer for Redpanda Connect DevFeed: [Drag, drop, done: a visual composer for Redpanda Connect]() Original publisher: [Read original article]() Author: Prakhar Garg Published: 2026-07-23T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Redpanda-Connect](), [Canvas](), [stream-processing](), [Caching](), [Routing (disambiguation)](), [HTTP](), [n8n](), [workflow automation]() Tags: [api](), [automation](), [building](), [cache](), [caching](), [events](), [front-end](), [http](), [n8n](), [pipeline](), [redpanda-connect](), [routing](), [stream-processing](), [workflow]() ### AI overview Redpanda introduces a visual Pipeline Builder for Redpanda Connect, using a drag-and-drop canvas to compose pipelines from inputs, processors, and outputs. The builder generates YAML while making routing, enrichment, caching, error handling, and retries easier to understand and configure. The article contrasts it with n8n, emphasizing Redpanda Connect's suitability for high-volume event streaming with ordering, backpressure, and exactly-once semantics. ### Source excerpt A first look at the new visual Pipeline Builder for Redpanda Connect (preview), plus a roundup of recent CDC and connector updates. ## Full CDC semantics land in the Iceberg output for Redpanda Connect DevFeed: [Full CDC semantics land in the Iceberg output for Redpanda Connect]() Original publisher: [Read original article]() Author: Prakhar Garg Published: 2026-07-14T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Apache Iceberg](), [clickhouse](), [Database](), [Kafka]() Tags: [analytics](), [clickhouse](), [data](), [database](), [engineering](), [product](), [real-time](), [redpanda-connect](), [streams](), [sync]() ### AI overview This hands-on article explains how Redpanda Connect v4.99.0 applies full change data capture semantics to Apache Iceberg tables. Its output supports row-level inserts, upserts, and deletes, allowing a lakehouse table to stay synchronized with a source database in real time through a single pipeline definition. ### Source excerpt Full CDC semantics in Iceberg mean the lakehouse reflects what the source database looks like right now, not what it looked like during last night's batch window. Get to know the latest addition to Redpanda Connect. ## Stream Oracle changes to ClickHouse in real time DevFeed: [Stream Oracle changes to ClickHouse in real time]() Original publisher: [Read original article]() Author: Prakhar Garg Published: 2026-07-07T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [clickhouse](), [Redpanda-Connect](), [Oracle Database](), [Kafka](), [real-time](), [data](), [Database](), [YAML]() Tags: [clickhouse](), [connectors](), [data](), [database](), [event](), [integration](), [kafka](), [oracle](), [pipeline](), [real-time](), [redpanda-connect](), [schema](), [yaml]() ### AI overview A hands-on walkthrough shows how to stream Oracle change data capture events through Redpanda Connect into ClickHouse in real time. It uses Oracle LogMiner and YAML-configured pipelines, with a Redpanda topic serving as an intermediate store before data is delivered to ClickHouse or other destinations. ### Source excerpt A hands-on walkthrough of the new Oracle input in Redpanda Connect. No Debezium, no Kafka Connect runtime, no JVM. ## Oracle CDC now available in Redpanda Connect DevFeed: [Oracle CDC now available in Redpanda Connect]() Original publisher: [Read original article]() Author: Prakhar Garg Published: 2026-04-09T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Databases](), [Streaming](), [Structured-data](), [Transactions](), [data](), [systems]() Tags: [announcement](), [data](), [debezium](), [event](), [jvm](), [oracle](), [redpanda-connect](), [schema](), [streaming](), [transactions]() ### AI overview Redpanda Connect v4.83.0 introduces a native Oracle CDC connector that captures inserts, updates, and deletes without a JVM, Kafka Connect cluster, or separate workers. It can snapshot existing tables, stream subsequent changes from Oracle redo logs, resume from checkpoints, and route schema-tracked Avro events to per-table Redpanda topics. ### Source excerpt Native CDC for Oracle enables real-time access without rearchitecting. No JVM, middleware, or operational overhead. ## Introducing Iceberg output for Redpanda Connect DevFeed: [Introducing Iceberg output for Redpanda Connect]() Original publisher: [Read original article]() Author: Prakhar Garg Published: 2026-03-05T00:00:00Z Content type: release Language: en Sources: [Redpanda]() Topics: [Apache Iceberg](), [Redpanda-Connect](), [Streaming](), [Routing (disambiguation)](), [schema-evolution](), [YAML](), [data]() Tags: [apache-iceberg](), [connectors](), [integration](), [pii](), [redpanda-connect](), [routing](), [schema-evolution](), [streaming](), [yaml]() ### AI overview Redpanda introduces an Iceberg output for Redpanda Connect that writes streaming data directly to Apache Iceberg tables through declarative YAML pipelines. The component supports transformations, enrichment, PII removal, and content-based routing across Redpanda Connect inputs and processors, enabling data from sources such as HTTP webhooks, Postgres CDC streams, and GCP Pub/Sub to reach lakehouse tables with less infrastructure and configuration overhead. ### Source excerpt Land any data into Apache Iceberg with Redpanda Connect. Maximize infra density, automate schema evolution, and scale 1-to-many routing without the toil. ## Aloha, Redpanda FY27 company kick-off DevFeed: [Aloha, Redpanda FY27 company kick-off]() Original publisher: [Read original article]() Author: Jenny Medeiros Published: 2026-02-24T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [AI Infrastructure](), [Redpanda-Connect](), [Serverless](), [Amazon Web Services](), [Disaster Recovery](), [MCP Server](), [Graphs](), [Google]() Tags: [ai](), [ai-infrastructure](), [aws](), [disaster-recovery](), [google](), [graphs](), [mcp-server](), [redpanda-connect](), [serverless]() ### AI overview Redpanda's FY27 company kick-off brought more than 160 employees together in San Diego to reflect on the company's progress and direction. The article highlights the Agentic Data Plane, Redpanda Cloud, Redpanda Serverless on AWS, disaster recovery in Redpanda 25.3, MCP server development, and new Redpanda Connect connectors, alongside talks about company culture and using AI to reshape work. ### Source excerpt From AI hackathons to hakas, here's what happened at our fourth company kick-off. ## Building with Redpanda Connect: Bloblang and Claude Plugin DevFeed: [Building with Redpanda Connect: Bloblang and Claude Plugin]() Original publisher: [Read original article]() Author: Prakhar Garg Published: 2026-02-24T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [Redpanda-Connect](), [PostgreSQL](), [Streaming](), [Claude Code](), [configuration](), [Testing](), [Code]() Tags: [claude](), [claude-code](), [how-to](), [pipeline](), [postgres](), [postgresql](), [redpanda-connect](), [streaming](), [testing]() ### AI overview A practical workshop on building a production-ready Postgres-to-Redpanda streaming pipeline with Redpanda Connect. It covers Connect's input, processor, and output architecture; PostgreSQL change data capture; Bloblang payload transformations; and the Claude plugin for generating configurations, explaining mappings, and debugging workflows. ### Source excerpt Join our workshop to learn how to build a production-ready Postgres-to-Redpanda pipeline and see what that simplicity looks like in practice. ## Redpanda Terraform supports managing Redpanda Connect pipelines, private network DevFeed: [Redpanda Terraform supports managing Redpanda Connect pipelines, private network]() Original publisher: [Read original article]() Author: David Yu Published: 2026-02-05T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Terraform](), [Redpanda-Connect](), [Provisioning](), [Cloud](), [configuration](), [Kafka](), [VPC](), [API](), [Amazon Web Services](), [Command-line interface]() Tags: [api](), [automation](), [aws](), [cloud](), [configuration](), [governance](), [infrastructure](), [pipelines](), [provisioning](), [redpanda-connect](), [terraform](), [vpc]() ### AI overview This article describes recent additions to the Redpanda Terraform provider for managing Redpanda Cloud clusters and Apache Kafka resources. It covers provisioning, configuration, and management of clusters, topics, schemas, roles, and pipelines, including BYOC and BYOVPC deployments supported on AWS and GCP. ### Source excerpt Automate Redpanda Cloud with Terraform. Manage Connect pipelines, PrivateLink, and BYOVPC to reduce operational toil and enforce enterprise governance. ## Build production-grade agentic AI with Redpanda Connect and broker auditing DevFeed: [Build production-grade agentic AI with Redpanda Connect and broker auditing]() Original publisher: [Read original article]() Author: Redpanda Published: 2025-12-16T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [AI Agent](), [Redpanda-Connect](), [audit trail](), [observability]() Tags: [agentic-ai](), [ai-agent](), [audit](), [redpanda-connect](), [tutorial]() ### AI overview A guide to building secure and observable agentic AI pipelines on Redpanda, using Connect for AI tools and data access and broker audit logs to record agent actions. ### Source excerpt Learn how to build secure, observable agentic AI on Redpanda using Connect for AI tools and data access, plus broker audit logs to capture every agent action. ## Streaming IoT and event data into Snowflake and ClickHouse DevFeed: [Streaming IoT and event data into Snowflake and ClickHouse]() Original publisher: [Read original article]() Author: Mdu Sibisi Published: 2025-12-09T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [Internet of things](), [Streaming](), [Redpanda-Connect](), [clickhouse](), [data observability](), [Compression](), [data-governance](), [DataOps](), [Database]() Tags: [clickhouse](), [cloud-storage](), [compliance](), [data](), [data-governance](), [event](), [iot](), [pipelines](), [redpanda-connect](), [streaming]() ### AI overview A guide to streaming IoT and event data through Redpanda and Redpanda Connect into Snowflake and ClickHouse. It compares ClickHouse for real-time analysis with Snowflake for scalable cloud storage, historical reporting, and querying, while discussing governance, compression, data freshness, and robust pipeline design. ### Source excerpt Learn how to stream IoT and event data into Snowflake and ClickHouse using Redpanda ## Operationalize Redpanda Connect with GitOps DevFeed: [Operationalize Redpanda Connect with GitOps]() Original publisher: [Read original article]() Author: Ben Barkhouse Published: 2025-12-02T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [GitOps](), [Kubernetes](), [CI/CD](), [internal developer portal](), [Command-line interface](), [Git]() Tags: [argo](), [argo-cd](), [auditability](), [cli](), [deployment](), [developer-experience](), [github](), [gitops](), [kubernetes](), [redpanda-connect](), [security](), [yaml]() ### AI overview This tutorial explains how to operationalize Redpanda Connect with GitOps on Kubernetes. It covers declarative management through Git and Argo CD, deployment workflows, and the differences between standalone mode and Streams mode for running pipelines. ### Source excerpt Here's how to operationalize Redpanda Connect with GitOps for less manual ops, predictable deployments, and a standard workflow. ## What's new in Redpanda Cloud: A simpler Serverless now on GCP DevFeed: [What's new in Redpanda Cloud: A simpler Serverless now on GCP]() Original publisher: [Read original article]() Author: Towfiqa Yasmeen Published: 2025-11-25T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Redpanda-Connect](), [Serverless](), [Google Cloud Platform (GCP)](), [Streaming](), [Kafka](), [User experience (UX)]() Tags: [announcement](), [gcp](), [kafka](), [redpanda-connect](), [serverless](), [stream-processing](), [streaming](), [usability](), [ux]() ### AI overview Redpanda Serverless introduces a guided onboarding experience, availability on Google Cloud Platform (GCP), and a simpler path to deploying Kafka-compatible streaming. The update targets developers and platform leaders with managed, usage-based streaming that removes broker, cluster, scaling, and infrastructure-management concerns. ### Source excerpt Redpanda Serverless gives you the shortest path from idea to impact with a new onboarding UX, availability on GCP, and more. Here's what's new. ## Build an e-commerce streaming data pipeline with Snowflake and Redpanda DevFeed: [Build an e-commerce streaming data pipeline with Snowflake and Redpanda]() Original publisher: [Read original article]() Author: Rajkumar Venkatasamy Published: 2025-10-21T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [Streaming](), [real-time](), [data](), [App](), [dashboards](), [Security](), [TLS (Transport Layer Security)](), [Python](), [GitHub]() Tags: [analytics](), [dashboards](), [e-commerce](), [github](), [how-to](), [kafka](), [learn](), [pipelines](), [product](), [python](), [real-time](), [redpanda-connect](), [security](), [streaming](), [tls]() ### AI overview A tutorial showing how to build a live e-commerce order data pipeline from a Redpanda topic into a Snowflake table using Redpanda Connect. It covers generating simulated order data, configuring a topic and cluster, and querying streamed data for real-time analytics, dashboards, and fraud detection. ### Source excerpt Learn how to set up a live data pipeline streaming e-commerce order data from Redpanda into Snowflake for real-time insights. ## Build a real-time equipment monitoring pipeline with Snowflake and MQTT DevFeed: [Build a real-time equipment monitoring pipeline with Snowflake and MQTT]() Original publisher: [Read original article]() Author: Sesethu Mhlana Published: 2025-09-16T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [Monitoring](), [MQTT](), [Internet of things](), [dashboards](), [Databases]() Tags: [connectors](), [data](), [database](), [heat](), [how-to](), [iot](), [manufacturing](), [metrics](), [monitoring](), [mqtt](), [operational](), [real-time](), [redpanda-connect](), [sensors](), [tutorial]() ### AI overview A tutorial showing how to build a real-time equipment monitoring pipeline for machine temperature data using IoT sensors, MQTT, Redpanda Connect, and Snowflake. It covers streaming sensor data, storing it in a database, and visualizing it in charts to support proactive maintenance, operational efficiency, safety, and compliance. ### Source excerpt Learn how to build an end-to-end, real-time equipment monitoring solution using MQTT, Redpanda Connect, and Snowflake. ## Setting up Redpanda observability in Datadog DevFeed: [Setting up Redpanda observability in Datadog]() Original publisher: [Read original article]() Author: Kavya Shivashankar Published: 2025-08-27T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [Setting up Redpanda observability in Datadog](), [observability](), [datadog agent](), [dashboards](), [Docker Compose](), [Streaming](), [Raft](), [Cloud]() Tags: [anomaly-detection](), [configure-redpanda-on-datadog](), [dashboards](), [datadog](), [datadog-agent](), [datadog-agent-for-redpanda](), [datadog-anomaly-detection-redpanda](), [datadog-dashboards-for-redpanda](), [docker](), [integration](), [monitor-redpanda-with-datadog](), [observability](), [prometheus](), [raft](), [redpanda-cluster-monitoring](), [redpanda-connect](), [redpanda-datadog-integration](), [redpanda-docker-setup](), [redpanda-logging-with-datadog](), [redpanda-metrics-in-datadog](), [redpanda-performance-monitoring](), [setting-up-redpanda-observability](), [setting-up-redpanda-observability-in-datadog](), [streaming]() ### AI overview This guide explains how to integrate Datadog with Redpanda to collect, visualize, and analyze cluster metrics and broker logs. It covers Datadog dashboards, alerts, anomaly detection, the Datadog Agent, and a Docker-based single-node setup. ### Source excerpt A comprehensive guide to integrating the Datadog Agent with Redpanda for data collection, metric scraping, and tapping into Datadog's powerful dashboards. ## Redpanda Cloud Summer 2025: Connect everywhere, Iceberg GA in cloud, and more DevFeed: [Redpanda Cloud Summer 2025: Connect everywhere, Iceberg GA in cloud, and more]() Original publisher: [Read original article]() Author: Mike Broberg Published: 2025-08-19T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Cloud](), [Streaming](), [Apache Iceberg](), [Serverless](), [Kafka](), [Cache](), [Google Cloud Platform (GCP)](), [HTTP](), [BigQuery](), [tbox](), [Slack]() Tags: [2026](), [announcement](), [apache-iceberg-integration](), [bigquery](), [byoc-on-azure](), [cloud](), [cloud-event-streaming-architecture](), [http](), [iceberg-ga-in-cloud](), [iceberg-topics-byoc](), [kafka](), [multi-cloud-solution-azure-aws-gcp](), [real-time-data-ingestion-lakehouse](), [redpanda-cloud-summer-2025](), [redpanda-connect](), [redpanda-connect-deployment](), [redpanda-serverless-features](), [serverless](), [serverless-capabilities-expansion](), [slack](), [stream-processing](), [streaming-pipelines-scalability](), [vertex-ai]() ### AI overview Redpanda Cloud's June-July 2025 summer update introduces general availability for Redpanda Connect across deployment types and supported cloud providers, expands Serverless capabilities, and makes Iceberg Topics generally available for AWS and GCP BYOC clusters. It also adds a Spanner change data capture connector, a write-optimized cache, Slack reaction output, and an HTTP gateway input for triggering streaming pipelines. ### Source excerpt Our summer update is sizzling with expanded Serverless capabilities, Iceberg topics on across all major clouds, and Redpanda Connect on all deployments. Read on. ## Redpanda open-sources top 16 AI connectors DevFeed: [Redpanda open-sources top 16 AI connectors]() Original publisher: [Read original article]() Author: Mike Broberg Published: 2025-08-18T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Redpanda-Connect](), [Open Source](), [Artificial Intelligence](), [stream-processing](), [OpenAI](), [Streaming](), [Amazon Bedrock](), [Embeddings](), [Retrieval Augmented Generation (RAG)](), [cohere](), [Ollama]() Tags: [ai](), [ai-connector-platforms](), [ai-connectors-for-commercial-products](), [ai-connectors-open-source](), [ai-data-streaming](), [ai-integration-tools](), [announcement](), [bedrock](), [cloud-native-ai-services](), [cohere](), [embeddable-ai-capabilities](), [embeddings](), [ollama](), [open-source](), [open-source-ai-for-business](), [openai](), [openai-connectors](), [rag](), [real-time-ai-streaming](), [redpanda-ai-connectors](), [redpanda-connect](), [stream-processing](), [streaming](), [streaming-ai-processors]() ### AI overview Redpanda announces the open-source release of its top AI connectors under the Apache 2.0 license. The connectors integrate Redpanda Connect with destinations and models including OpenAI, Cohere, Amazon Bedrock, Ollama, and Google Cloud Vertex AI, supporting streaming pipelines and use cases such as generation, summarization, classification, translation, and text embeddings for RAG. ### Source excerpt Redpanda open-sources top AI connectors to the most used destinations, including OpenAI, Cohere, Bedrock, Ollama, and Vertex AI. Learn more. ## Streaming optimized data to S3 for analytics with Parquet DevFeed: [Streaming optimized data to S3 for analytics with Parquet]() Original publisher: [Read original article]() Author: Chandler Mayo Published: 2025-08-13T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [parquet](), [Redpanda-Connect](), [Amazon S3](), [data analytics](), [Streaming](), [Apache Spark](), [pandas](), [Serverless]() Tags: [analytics](), [analytics-systems-data-feed](), [apache-spark-and-athena-data-queries](), [athena](), [automating-data-pipelines-in-s3](), [building-dashboards-with-s3-data](), [compressing-data-with-parquet](), [data](), [data-pipeline](), [data-pipeline-for-analytics](), [event-driven-pipelines-with-s3](), [how-to](), [learn](), [pandas](), [parquet](), [parquet-files-in-amazon-s3](), [pipeline](), [real-time-data-streaming](), [redpanda-connect](), [redpanda-data-streaming](), [s3](), [serverless](), [setting-up-redpanda-observability](), [spark](), [streaming](), [streaming-data-to-s3-with-parquet](), [tutorial]() ### AI overview This tutorial explains how to use Redpanda Connect to continuously batch streaming data and write compressed Apache Parquet files to Amazon S3 for analytical workloads. It covers the benefits of Parquet and querying the resulting files with tools such as Pandas, Apache Spark, and Athena. ### Source excerpt Learn how to build a powerful data pipeline that feeds analytics systems from Redpanda using clean, compressed Parquet files in Amazon S3. ## Real-time CDC for inventory tracking with PostgreSQL DevFeed: [Real-time CDC for inventory tracking with PostgreSQL]() Original publisher: [Read original article]() Author: Aykut Bulgu Published: 2025-08-12T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [PostgreSQL](), [event driven](), [Replication](), [Streaming](), [Serverless](), [Cloud](), [dashboards]() Tags: [cdc-for-retail-inventory-management](), [cdc-inventory-tracking](), [command-line](), [configure-postgresql-for-cdc](), [how-to](), [implement-real-time-cdc-postgresql](), [postgresql](), [postgresql-cdc-tutorial](), [postgresql-to-redpanda-streaming](), [real-time](), [real-time-cdc-postgresql](), [real-time-inventory-tracking-system](), [redpanda-connect](), [redpanda-connect-cdc](), [redpanda-serverless-cdc](), [replication](), [serverless](), [setup-cdc-with-redpanda](), [stream-changes-postgresql-redpanda](), [streaming](), [tutorial]() ### AI overview A tutorial on implementing real-time change data capture from PostgreSQL with Redpanda Connect for inventory tracking. It covers the architecture, setup, streaming database changes to Redpanda topics, and accessing a Redpanda Serverless cluster. ### Source excerpt Learn how to easily implement real-time CDC using PostgreSQL and Redpanda for stress-free inventory tracking. ## Exporting data from Redpanda to S3 in batched JSON DevFeed: [Exporting data from Redpanda to S3 in batched JSON]() Original publisher: [Read original article]() Author: Chandler Mayo Published: 2025-08-07T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [Amazon S3](), [JSON](), [Streaming](), [Serverless](), [parquet](), [Compression](), [schema-evolution]() Tags: [batch-json-arrays-to-s3](), [batching](), [batching-messages-in-s3](), [building-event-driven-pipelines-with-sqs-and-s3](), [data](), [exporting-data-from-redpanda-to-s3](), [exporting-parquet-files-to-s3](), [how-to](), [json](), [optimize-redpanda-s3-pipeline](), [performance](), [pipelines](), [redpanda-connect](), [redpanda-s3-integration](), [redpanda-to-s3-json-export](), [s3](), [serverless](), [setting-up-redpanda-observability-in-datadog](), [streaming](), [streaming-data-to-s3-with-redpanda](), [streaming-optimized-data-to-s3-for-analytics](), [tutorial]() ### AI overview A tutorial showing how to configure a Redpanda Connect pipeline to batch multiple Redpanda messages into JSON files stored in Amazon S3. It explains the performance benefits of batching, deployment considerations for Redpanda Serverless, and production tuning options, then points to Parquet for larger analytical workloads. ### Source excerpt Learn how to write messages from Redpanda to its own JSON file in S3. ## Writing data from Redpanda to S3 DevFeed: [Writing data from Redpanda to S3]() Original publisher: [Read original article]() Author: Chandler Mayo Published: 2025-07-29T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [Redpanda-Connect](), [Amazon S3](), [Streaming](), [JSON](), [Serverless](), [configuration](), [data observability](), [debug]() Tags: [amazon-s3](), [batch-messages-from-redpanda-to-s3](), [event-driven-pipelines-with-redpanda-and-s3](), [how-to](), [json](), [optimizing-redpanda-data-for-s3](), [pipeline](), [redpanda-community-and-s3-integration](), [redpanda-connect](), [redpanda-json-files-to-s3](), [redpanda-s3-bucket-configuration](), [redpanda-to-amazon-s3-data-streaming](), [redpanda-to-s3-integration](), [serverless](), [setting-up-redpanda-with-amazon-s3](), [stream-data-to-amazon-s3-from-redpanda](), [streaming](), [tutorial](), [writing-data-from-redpanda-to-s3]() ### AI overview This tutorial shows how to use Redpanda Connect to stream messages from a Redpanda topic into Amazon S3, writing each message as an individual JSON file. It covers deploying a YAML pipeline, configuring its input and output, and notes that the pattern is suited to small datasets, debugging, and archives rather than production-scale throughput. ### Source excerpt Learn the easiest way to stream data into Amazon S3 from Redpanda. ## IoT for fun and Prophet: Scaling IoT and predicting the future DevFeed: [IoT for fun and Prophet: Scaling IoT and predicting the future]() Original publisher: [Read original article]() Author: Bryan Wood Published: 2025-07-22T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [Apache Iceberg](), [Internet of things](), [Redpanda-Connect](), [Streaming](), [Time Series](), [ESP32](), [Amazon Web Services](), [Amazon S3](), [schema-evolution](), [event driven](), [Hardware](), [Kafka]() Tags: [apache-iceberg](), [apache-iceberg-for-scalable-storage](), [aws](), [core](), [devices](), [esp32-mqtt-iot-example](), [event-driven](), [hardware](), [integrating-iceberg-and-redpanda](), [iot](), [iot-data-pipeline-architecture](), [iot-predictive-analytics](), [iot-sensor-data-forecasting](), [predictive-analysis-with-iot](), [prophet-forecasting-for-iot](), [real-time-iot-data-streaming](), [redpanda-connect](), [redpanda-connect-for-iot](), [s3](), [scaling-iot-with-prophet](), [schema-evolution](), [schema-management-in-iot](), [sensor](), [streaming](), [thought-leadership]() ### AI overview This tutorial presents a scalable IoT data pipeline for real-time streaming and forecasting. It combines Redpanda and Redpanda Connect for ingestion and messaging, Apache Iceberg for scalable time-series data storage, AWS IoT for device-to-cloud communication, and ESP32 hardware for affordable connected-device experiments and deployments. ### Source excerpt Check out this real-world example of scaling IoT for predictive analysis with Redpanda, Iceberg, and Prophet--without high costs or complexity. ## Introducing multi-language dynamic plugins for Redpanda Connect DevFeed: [Introducing multi-language dynamic plugins for Redpanda Connect]() Original publisher: [Read original article]() Author: James Kinley Published: 2025-06-17T00:00:00Z Content type: release Language: en Sources: [Redpanda]() Topics: [gRPC](), [SDKs](), [Go Language](), [Python](), [Processes](), [streaming-data-processing](), [Unix]() Tags: [ai-ml-capabilities-in-streaming-data](), [architecture](), [configuration](), [creating-plugins-in-golang-and-python](), [dynamic-vs-compiled-plugins](), [framework](), [go](), [grpc-plugin-system](), [integration](), [interfaces](), [ipc](), [language-agnostic-plugin-system](), [multi-language-plugin-development](), [pipelines](), [plugin-development](), [plugins](), [process](), [processes](), [product](), [protocol](), [python](), [python-sdk-for-streaming-data](), [redpanda-connect](), [redpanda-connect-dynamic-plugins](), [redpanda-streaming-infrastructure](), [runtime-loaded-plugins-grpc](), [sdk](), [sdks](), [stream-processing](), [streaming-data-pipeline-plugins]() ### AI overview Redpanda Connect introduces Apache 2.0-licensed dynamic plugins in Beta with version 4.56.0. Plugins can be loaded at runtime as external executables that communicate with the main process through gRPC over Unix sockets, enabling plugin development in Go and Python. Official SDKs support both languages, while native Go plugins remain the preferred option for performance-critical workloads. ### Source excerpt Redpanda Connect dynamic plugins framework allows you to create and load plugins at runtime, opening up a world of new integration possibilities beyond Go. [Next page]()