# Amazon MSK

Amazon MSK is a fully managed streaming data service that operates Apache Kafka infrastructure for building and running streaming-data applications.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Amazon MSK Express brokers available on AWS Graviton4-based M8g instances

DevFeed: [Amazon MSK Express brokers available on AWS Graviton4-based M8g instances](<https://devfeed.tech/articles/amazon-msk-express-brokers-available-on-aws-graviton4-based-m8g-instances-140165.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/amazon-msk-express-brokers-available-on-aws-graviton4-based-m8g-instances/>)

Author: Charishma Makineni

Published: 2026-11-17T22:07:48Z

Content type: release

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Graviton](<https://devfeed.tech/topics/graviton.md>), [Amazon EC2](<https://devfeed.tech/topics/amazon-ec2.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-managed-streaming-for-apache-kafka-amazon-msk](<https://devfeed.tech/tags/amazon-managed-streaming-for-apache-kafka-amazon-msk.md>), [amazon-msk](<https://devfeed.tech/tags/amazon-msk.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [aws](<https://devfeed.tech/tags/aws.md>), [based](<https://devfeed.tech/tags/based.md>), [broker](<https://devfeed.tech/tags/broker.md>), [client](<https://devfeed.tech/tags/client.md>), [consumer](<https://devfeed.tech/tags/consumer.md>), [cross](<https://devfeed.tech/tags/cross.md>), [express](<https://devfeed.tech/tags/express.md>), [faster](<https://devfeed.tech/tags/faster.md>), [foundational-100](<https://devfeed.tech/tags/foundational-100.md>), [graviton](<https://devfeed.tech/tags/graviton.md>), [graviton4](<https://devfeed.tech/tags/graviton4.md>), [latency](<https://devfeed.tech/tags/latency.md>), [p99](<https://devfeed.tech/tags/p99.md>), [pricing](<https://devfeed.tech/tags/pricing.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [size](<https://devfeed.tech/tags/size.md>), [traffic](<https://devfeed.tech/tags/traffic.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Amazon MSK Express brokers on Graviton4-based M8g instances combine lower costs with better results than M7g in the described tests: broker-hour pricing is 15% lower, broker-side cross-AZ client traffic is uncharged (cutting total cross-AZ transfer cost by 50%), and P99 produce latency was 21-54% lower at the same tested load. Brokers used 9-11 percentage points less CPU, and consumers cleared up to three times more backlog in 30 minutes. Results depend on workload settings; the catch-up test used large brokers only. M8g adds a 24xlarge option, while shared sizes retain the same per-broker storage and network throughput. Teams can create an M8g cluster, update an M7g Express cluster in place one broker at a time, or migrate from Standard or self-managed Kafka using MSK Replicator. Rack-aware consumers can reduce cross-AZ charges further, though reading from follower replicas may add some end-to-end latency.

### Source excerpt

Amazon MSK Express brokers are now available on AWS Graviton4-based M8g instances. Compared with Express on M7g, M8g delivers 15% lower broker-hour pricing, lower P99 produce latency, faster consumer recovery, a new 24xlarge size, and no broker-side charge for cross-AZ client traffic.

## Deliver petabytes of Apache Kafka data to Amazon S3 at up to 60% lower cost with Amazon MSK Express brokers

DevFeed: [Deliver petabytes of Apache Kafka data to Amazon S3 at up to 60% lower cost with Amazon MSK Express brokers](<https://devfeed.tech/articles/deliver-petabytes-of-apache-kafka-data-to-amazon-s3-at-up-to-60-lower-cost-with-amazon-msk-express-brokers-136179.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/deliver-petabytes-of-apache-kafka-data-to-amazon-s3-at-up-to-60-lower-cost-with-amazon-msk-express-brokers/>)

Author: Diksha Dubey

Published: 2026-11-16T20:42:44Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>)

Tags: [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-managed-streaming-for-apache-kafka-amazon-msk](<https://devfeed.tech/tags/amazon-managed-streaming-for-apache-kafka-amazon-msk.md>), [amazon-msk](<https://devfeed.tech/tags/amazon-msk.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-simple-storage-service-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-s3.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [channel](<https://devfeed.tech/tags/channel.md>), [connect](<https://devfeed.tech/tags/connect.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [delivered](<https://devfeed.tech/tags/delivered.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [express](<https://devfeed.tech/tags/express.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [one](<https://devfeed.tech/tags/one.md>), [post](<https://devfeed.tech/tags/post.md>), [run](<https://devfeed.tech/tags/run.md>), [s3](<https://devfeed.tech/tags/s3.md>), [up](<https://devfeed.tech/tags/up.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Amazon MSK Express brokers deliver Kafka topic records to general purpose Amazon S3 buckets through a channel, without Kafka Connect. The article says delivery is billed per byte, adds no broker egress load, and can cost up to 60% less than self-managed alternatives. Brokers automatically scale storage and rebalance partitions; MSK manages recovery and updates. Channels batch records into S3 objects, preserving producer formats, with configurable compression, storage class, and object key layout. JSON and String records are validated; failures go to an S3 dead-letter queue while delivery continues. The article also describes CloudWatch metrics and logs, setup through the console, APIs, CLI, or an Agent Skill, and recommends overlapping a new channel with an existing connector until delivered data is confirmed. Review supported configurations and prerequisites before production use.

### Source excerpt

Amazon MSK Express brokers now deliver Apache Kafka topic data to Amazon S3 at up to 60% lower cost, with no Kafka Connect to run. You create a channel on a topic and pay only for the data delivered. In this post, we cover the two challenges of delivering Kafka data to Amazon S3 today and how Express brokers and data delivery to Amazon S3 address each one.

## Allocate shared Amazon MSK costs to Kafka topics for cost visibility

DevFeed: [Allocate shared Amazon MSK costs to Kafka topics for cost visibility](<https://devfeed.tech/articles/allocate-shared-amazon-msk-costs-to-kafka-topics-for-cost-visibility-129557.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/allocate-shared-amazon-msk-costs-to-kafka-topics-for-cost-visibility/>)

Author: Durai Krishnan

Published: 2026-11-11T19:46:04Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [cloud cost management](<https://devfeed.tech/topics/cloud-cost-management.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [Amazon Managed Service for Prometheus](<https://devfeed.tech/topics/amazon-managed-service-for-prometheus.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Prometheus](<https://devfeed.tech/topics/prometheus.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [aws-sns](<https://devfeed.tech/topics/aws-sns.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-eventbridge](<https://devfeed.tech/tags/amazon-eventbridge.md>), [amazon-managed-service-for-prometheus](<https://devfeed.tech/tags/amazon-managed-service-for-prometheus.md>), [amazon-managed-streaming-for-apache-kafka-amazon-msk](<https://devfeed.tech/tags/amazon-managed-streaming-for-apache-kafka-amazon-msk.md>), [amazon-msk](<https://devfeed.tech/tags/amazon-msk.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-simple-storage-service-amazon-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-amazon-s3.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cost-and-usage-report](<https://devfeed.tech/tags/aws-cost-and-usage-report.md>), [billing](<https://devfeed.tech/tags/billing.md>), [cloudwatch](<https://devfeed.tech/tags/cloudwatch.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cost-and-usage-report](<https://devfeed.tech/tags/cost-and-usage-report.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [prometheus](<https://devfeed.tech/tags/prometheus.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [topic](<https://devfeed.tech/tags/topic.md>), [visibility](<https://devfeed.tech/tags/visibility.md>)

### AI overview

The pipeline estimates each Kafka topic's share of an Amazon MSK Provisioned cluster's daily cost by joining cluster-level CUR charges with per-topic ingress metrics, mapping cluster names to ARNs, then allocating cost in proportion to bytes in. A daily Lambda writes Parquet results to S3 for Athena and Quick Sight. Enhanced Monitoring publishes per-topic CloudWatch metrics with little setup, but metric charges grow with topic-broker pairs. Open Monitoring exposes broker metrics at no additional charge; a Fargate scraper sends them to Amazon Managed Service for Prometheus, whose ingestion costs grow with samples. Both options use the same allocation logic. Bytes in is only a proxy: it misses retention, replication, compression, and consumer fan-out. Idle topics receive no row, so active topics may absorb the full cost; if no topics report traffic, the function splits cost evenly across configured topics. CUR can lag up to 24 hours, and metrics exist only after they are enabled. The method applies to Provisioned clusters; Serverless is out of scope.

### Source excerpt

Amazon MSK bills at the cluster level, with no per-topic breakdown. This post shows how to allocate shared MSK cluster costs to individual Kafka topics by combining AWS Cost and Usage Report billing with per-topic CloudWatch or Prometheus metrics, giving each team a usage-based view of its spend.

## How Intuit migrated from Apache Hive Metastore to AWS Glue Data Catalog

DevFeed: [How Intuit migrated from Apache Hive Metastore to AWS Glue Data Catalog](<https://devfeed.tech/articles/how-intuit-migrated-from-apache-hive-metastore-to-aws-glue-data-catalog-129561.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/how-intuit-migrated-from-apache-hive-metastore-to-aws-glue-data-catalog/>)

Author: Srinivasan Krishnasamy

Published: 2026-11-11T15:43:26Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>), [etl](<https://devfeed.tech/topics/etl.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [AWS Lake Formation](<https://devfeed.tech/topics/aws-lake-formation.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>)

Tags: [access-control](<https://devfeed.tech/tags/access-control.md>), [access-controls](<https://devfeed.tech/tags/access-controls.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-services](<https://devfeed.tech/tags/ai-services.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-aurora](<https://devfeed.tech/tags/amazon-aurora.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-dynamodb](<https://devfeed.tech/tags/amazon-dynamodb.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-hive](<https://devfeed.tech/tags/apache-hive.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data](<https://devfeed.tech/tags/data.md>), [data-catalog](<https://devfeed.tech/tags/data-catalog.md>), [data-discovery](<https://devfeed.tech/tags/data-discovery.md>), [hive](<https://devfeed.tech/tags/hive.md>), [integration](<https://devfeed.tech/tags/integration.md>), [intuit](<https://devfeed.tech/tags/intuit.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>)

### AI overview

Intuit migrated databases, tables, and partitions from Apache Hive Metastore (HMS) to AWS Glue Data Catalog (GDC) using a phased approach: a one-time migration, ongoing synchronization in both directions, hourly parity checks, and targeted repair syncs. A DynamoDB source-of-truth flag per table prevents sync loops. HMS changes flow through Debezium CDC and Kafka, with partition events batched over 10-second windows; GDC changes use EventBridge, Lambda, Kafka, and Flink to order events before applying them to HMS. Before migrating, Intuit recommends auditing unused objects, estimating and load-testing Glue API quotas, batching writes, and using retries with backoff to manage throttling. The migration covered databases, tables, and partitions for S3-backed data; column statistics, privileges, roles, functions, and transactions were out of scope. GDC also does not support database or table renames.

### Source excerpt

How Intuit migrated its data catalog from Apache Hive Metastore to AWS Glue Data Catalog on its journey to an open, secure lakehouse. This post covers the challenges, the phased bidirectional-sync solution, and the benefits: reduced operational overhead, improved data discovery, and tighter integration with AWS analytics services.

## S3 to Kafka: How to Turn New S3 Files into Kafka Messages

DevFeed: [S3 to Kafka: How to Turn New S3 Files into Kafka Messages](<https://devfeed.tech/articles/s3-to-kafka-how-to-turn-new-s3-files-into-kafka-messages-121707.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/s3-to-kafka>)

Author: Seán Whelan

Published: 2026-11-06T12:27:02Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [parquet](<https://devfeed.tech/topics/parquet.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [compare](<https://devfeed.tech/tags/compare.md>), [files](<https://devfeed.tech/tags/files.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [managed](<https://devfeed.tech/tags/managed.md>), [messages](<https://devfeed.tech/tags/messages.md>), [ordering](<https://devfeed.tech/tags/ordering.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [s3](<https://devfeed.tech/tags/s3.md>), [setup](<https://devfeed.tech/tags/setup.md>), [source](<https://devfeed.tech/tags/source.md>), [spark](<https://devfeed.tech/tags/spark.md>)

### AI overview

S3 files can become Kafka messages through Kafka Connect, S3 event notifications with Lambda, Spark or Flink, or a managed pipeline. The main differences are how each discovers and tracks files, supported parsing and transformation, and setup; most approaches deliver at least once, so consumers may need to handle duplicates. Listing tools can miss late-arriving keys when they track only the last key, while re-reading modified files can produce duplicates. The guide details S3 and Kafka permissions, format and message-size constraints, and operational tradeoffs. Estuary's described route lists a bucket prefix, parses new files into a durable collection, then materializes records to Kafka; it runs every five minutes by default, delivers at least once, and has specific Parquet size, field truncation, schema-subject, authentication, and deletion limitations. Event-triggered Lambda is positioned for a modest number of uploads needing quick reaction, and Spark or Flink for files requiring transformation.

### Source excerpt

Move S3 data into Kafka with an S3 source connector, AWS Lambda, Spark or a managed pipeline. Compare file ordering, formats, duplicates and setup.

## Kafka to ClickHouse: Streaming Ingestion, Duplicates & Tombstones Guide

DevFeed: [Kafka to ClickHouse: Streaming Ingestion, Duplicates & Tombstones Guide](<https://devfeed.tech/articles/kafka-to-clickhouse-streaming-ingestion-duplicates-tombstones-guide-121703.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/kafka-to-clickhouse>)

Author: Seán Whelan

Published: 2026-11-06T12:26:41Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [self-hosted databases vs managed services](<https://devfeed.tech/topics/self-hosted-databases-vs-managed-services.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>)

Tags: [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [compare](<https://devfeed.tech/tags/compare.md>), [connect](<https://devfeed.tech/tags/connect.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [engine](<https://devfeed.tech/tags/engine.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [guide](<https://devfeed.tech/tags/guide.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [stream](<https://devfeed.tech/tags/stream.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [table](<https://devfeed.tech/tags/table.md>)

### AI overview

Kafka ingestion into ClickHouse is generally at least once, so replayed messages can create duplicates; reliable designs need batching and tables suited to event or keyed data. The article compares the Kafka table engine, ClickPipes, the Kafka Connect sink, and managed pipelines: the engine runs consumers on ClickHouse servers, ClickPipes is recommended for ClickHouse Cloud, and the sink fits teams already running Kafka Connect. For event streams, store partition and offset in a ReplacingMergeTree sorting key; keyed topics with tombstones need ReplacingMergeTree keyed by message key and queries with FINAL. Tombstone support differs: ClickPipes can delete rows in exactly-once mode or mark them deleted, while the Connect sink lists deletes as unsupported and the table engine's documentation does not describe null-value messages. Retention limits recovery, and compaction can remove tombstones after the configured retention period. Large, infrequent inserts reduce part pressure; monitor lag and active parts. The article also describes Estuary capture and materialization, which keeps Kafka progress in its own checkpoints and writes to ClickHouse on a configurable schedule.

### Source excerpt

Stream Kafka to ClickHouse with the Kafka table engine, ClickPipes or Kafka Connect. Compare 4 methods, delivery guarantees and duplicate handling.

## Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg

DevFeed: [Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg](<https://devfeed.tech/articles/migrate-google-bigquery-to-an-open-lakehouse-on-aws-with-apache-iceberg-114202.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/migrate-google-bigquery-to-an-open-lakehouse-on-aws-with-apache-iceberg/>)

Author: Saman Irfan

Published: 2026-11-02T17:21:17Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [AWS Lake Formation](<https://devfeed.tech/topics/aws-lake-formation.md>), [Amazon Bedrock Knowledge Bases](<https://devfeed.tech/topics/amazon-bedrock-knowledge-bases.md>), [AWS Step Functions](<https://devfeed.tech/topics/aws-step-functions.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [data](<https://devfeed.tech/tags/data.md>), [design](<https://devfeed.tech/tags/design.md>), [future](<https://devfeed.tech/tags/future.md>), [google](<https://devfeed.tech/tags/google.md>), [google-bigquery](<https://devfeed.tech/tags/google-bigquery.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [migrate](<https://devfeed.tech/tags/migrate.md>), [migration](<https://devfeed.tech/tags/migration.md>), [open](<https://devfeed.tech/tags/open.md>), [optimize](<https://devfeed.tech/tags/optimize.md>), [over](<https://devfeed.tech/tags/over.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

The article presents a phased path from BigQuery to an AWS lakehouse using Apache Iceberg tables on Amazon S3, with governance through AWS Glue Data Catalog and Lake Formation and engines selected by workload. It recommends assessing tables, query complexity, consumers, volumes, and access controls; designing storage, partitioning, ingestion, and governance; then converting a representative slice and validating results, performance, and cost. For bulk transfer, export BigQuery tables to Parquet in Google Cloud Storage, move them to S3 with AWS DataSync, create or register Iceberg tables, and synchronize incremental changes during transition. Cutover includes parallel validation and deciding whether each consumer will reconnect, be rebuilt, or relay data. The article distinguishes managed S3 Tables, which handle maintenance automatically, from self-managed Iceberg on general-purpose S3, which requires tuning. It also cautions that BigQuery scan-based and Redshift compute-based pricing are not directly comparable, recommending workload-based benchmarks and monthly TCO estimates. Post-migration steps include query and cost optimization.

### Source excerpt

Learn how to migrate Google BigQuery workloads to an open lakehouse on AWS built on Apache Iceberg and Amazon S3. This post explains why an open lakehouse future-proofs your data, presents the target architecture on AWS, and walks through a step-by-step migration: assess, design, convert, migrate, cut over, and optimize.

## How UTS established a modern curriculum data services foundation on AWS

DevFeed: [How UTS established a modern curriculum data services foundation on AWS](<https://devfeed.tech/articles/how-uts-established-a-modern-curriculum-data-services-foundation-on-aws-87252.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/how-uts-established-a-modern-curriculum-data-services-foundation-on-aws/>)

Author: Oleg Polianski

Published: 2026-10-14T14:29:10Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [AWS Amplify](<https://devfeed.tech/topics/aws-amplify.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [Amazon OpenSearch Service](<https://devfeed.tech/topics/amazon-opensearch-service.md>), [Redis](<https://devfeed.tech/topics/redis.md>), [Authorization](<https://devfeed.tech/topics/authorization.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>)

Tags: [amazon-api-gateway](<https://devfeed.tech/tags/amazon-api-gateway.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-eventbridge](<https://devfeed.tech/tags/amazon-eventbridge.md>), [amazon-msk](<https://devfeed.tech/tags/amazon-msk.md>), [amazon-neptune](<https://devfeed.tech/tags/amazon-neptune.md>), [amazon-opensearch-service](<https://devfeed.tech/tags/amazon-opensearch-service.md>), [amazon-route-53](<https://devfeed.tech/tags/amazon-route-53.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-appsync](<https://devfeed.tech/tags/aws-appsync.md>), [aws-certificate-manager](<https://devfeed.tech/tags/aws-certificate-manager.md>), [aws-identity-and-access-management-iam](<https://devfeed.tech/tags/aws-identity-and-access-management-iam.md>), [aws-key-management-service](<https://devfeed.tech/tags/aws-key-management-service.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [aws-secrets-manager](<https://devfeed.tech/tags/aws-secrets-manager.md>), [aws-waf](<https://devfeed.tech/tags/aws-waf.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data](<https://devfeed.tech/tags/data.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [semantic-search](<https://devfeed.tech/tags/semantic-search.md>)

### AI overview

The University of Technology Sydney built an AWS-based curriculum data service that keeps its Curriculum Management System as the system of record while exposing governed APIs, graph traversal, semantic search, and event delivery. Since production launch, the service has recorded zero incidents, averages 1.6-second delivery of changes, and is operated by one engineer.

### Source excerpt

The University of Technology Sydney built a modern curriculum data services foundation on AWS that treats curriculum data as a governed, reusable product, available through APIs, graph traversal, semantic search, and event delivery, while leaving the source Curriculum Management System authoritative. One engineer now operates the domain.

## Optimize consumer rebalancing on Amazon MSK with next generation protocol

DevFeed: [Optimize consumer rebalancing on Amazon MSK with next generation protocol](<https://devfeed.tech/articles/optimize-consumer-rebalancing-on-amazon-msk-with-next-generation-protocol-63190.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/optimize-consumer-rebalancing-on-amazon-msk-with-next-generation-protocol/>)

Author: Yashika Jain

Published: 2026-10-01T16:43:54Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-managed-streaming-for-apache-kafka-amazon-msk](<https://devfeed.tech/tags/amazon-managed-streaming-for-apache-kafka-amazon-msk.md>), [amazon-msk](<https://devfeed.tech/tags/amazon-msk.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [ci-cd](<https://devfeed.tech/tags/ci-cd.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The article explains how Apache Kafka's KIP-848 consumer protocol changes group rebalancing by moving coordination to the broker and incrementally reassigning affected partitions. It covers enabling the protocol on Amazon MSK, compatible client versions, configuration changes, troubleshooting slow rebalances, and monitoring results.

### Source excerpt

The KIP-848 consumer protocol in Apache Kafka 4.0 redesigns consumer group rebalancing to eliminate stop-the-world pauses. This post explains how the consumer protocol works on Amazon MSK, how to enable it on MSK Standard and Express brokers, and how to diagnose and resolve slow rebalancing issues.

## Best practices for scaling large consumer groups on Amazon MSK

DevFeed: [Best practices for scaling large consumer groups on Amazon MSK](<https://devfeed.tech/articles/best-practices-for-scaling-large-consumer-groups-on-amazon-msk-57877.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/best-practices-for-scaling-large-consumer-groups-on-amazon-msk/>)

Author: Pallavi Jha

Published: 2026-09-22T15:16:06Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [configuration](<https://devfeed.tech/topics/configuration.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [Command-line interface](<https://devfeed.tech/topics/cli.md>)

Tags: [amazon-managed-streaming-for-apache-kafka-amazon-msk](<https://devfeed.tech/tags/amazon-managed-streaming-for-apache-kafka-amazon-msk.md>), [amazon-msk](<https://devfeed.tech/tags/amazon-msk.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [capacity-planning](<https://devfeed.tech/tags/capacity-planning.md>), [cli](<https://devfeed.tech/tags/cli.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [expert-400](<https://devfeed.tech/tags/expert-400.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [scaling](<https://devfeed.tech/tags/scaling.md>)

### AI overview

A practical guide to scaling large Amazon MSK consumer groups whose Kafka metadata records can exceed the 1 MB topic limit. It explains how to estimate metadata size, increase the topic-level limit, split groups, right-size partitions, optimize naming, plan capacity, and monitor the system.

### Source excerpt

As consumer groups on Amazon MSK scale to thousands of members, the metadata record Kafka writes during rebalances can exceed the 1 MB limit and stall the group. Learn how to estimate metadata size, raise the topic-level limit safely, plan capacity, and apply complementary strategies for scaling large consumer groups.

## Five Years of Kafka at Razorpay's UPI Switch

DevFeed: [Five Years of Kafka at Razorpay's UPI Switch](<https://devfeed.tech/articles/five-years-of-kafka-at-razorpay-s-upi-switch-24044.md>)

Original publisher: [Read original article](<https://engineering.razorpay.com/tryst-with-kafka-2f5cef766c45?source=rss----6407ad2e59af---4>)

Author: Kshitij Nawandar

Published: 2026-09-07T09:09:58Z

Content type: article

Language: en

Sources: [Razorpay](<https://devfeed.tech/sources/razorpay-engineering-medium.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [razorpay](<https://devfeed.tech/topics/razorpay.md>), [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>), [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [aws-sns](<https://devfeed.tech/topics/aws-sns.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-sns](<https://devfeed.tech/tags/aws-sns.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [event](<https://devfeed.tech/tags/event.md>), [growth](<https://devfeed.tech/tags/growth.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [messaging](<https://devfeed.tech/tags/messaging.md>), [microservices](<https://devfeed.tech/tags/microservices.md>), [monolith](<https://devfeed.tech/tags/monolith.md>), [ordering](<https://devfeed.tech/tags/ordering.md>), [payment](<https://devfeed.tech/tags/payment.md>), [payment-processing](<https://devfeed.tech/tags/payment-processing.md>), [performance](<https://devfeed.tech/tags/performance.md>), [platform](<https://devfeed.tech/tags/platform.md>), [queue](<https://devfeed.tech/tags/queue.md>), [razorpay](<https://devfeed.tech/tags/razorpay.md>), [switch](<https://devfeed.tech/tags/switch.md>), [upi](<https://devfeed.tech/tags/upi.md>)

### AI overview

Razorpay's UPI Switch grew from a monolith using two AWS SQS queues (handling a 400 TPS peak) into an event-driven system whose Kafka infrastructure became constrained by partition growth, connector routing failures, broker connection imbalance, stale DNS after MSK Serverless restarts, and an IAM-auth connection cap. The team replaced GoCloud Pub/Sub with the Kafka-native franz-go client, adding DNS recovery, connection controls, batching and metrics; it reports lower connection counts and publish latency, but marks those figures for verification. It also consolidated event topics by domain, parallelized consumer processing through worker pools while preserving per-partition ordering, isolated latency-sensitive and financial flows into separate gateway components, and moved to provisioned MSK with SASL/SCRAM. The excerpt says mixed-load benchmarks were run but gives no results. Remaining work includes consumer idempotency, restoring atomicity between database writes and Kafka publishing, and consistent retry/DLQ policies.

### Source excerpt

Preface The UPI Switch at Razorpay has evolved significantly in the five years since we started building it. The Switch is the platform that enables real-time payment processing with NPCI. When the team began, it was little more than an idea. Today it powers more than 70% of Razorpay's total UPI volume. Because UPI is inherently asynchronous, a messaging system sits at the heart of the Switch and has a direct impact on performance, reliability, and scale. What began as a straightforward queue became the core of the system, shaping how every new feature was designed and delivered. This post covers that evolution: the decisions that enabled growth, the ones that slowed us down, the operational issues that forced us to rethink our assumptions, and the optimizations that ultimately stabilized our Kafka-based infrastructure. This is the story of what we got right, what we got wrong, and how we eventually built something stable enough to grow on. The First Version: Monolith and SQS When we began building the UPI Switch, we weren't thinking about massive scale, distributed systems, or elegant event routing. So we built Switch v1 as a monolith. No microservices, no distributed orchestration: just one solid block of code doing everything. That was the right call. We needed to move fast, experiment, and learn, and we followed the Keep It Simple, Stupid (KISS) principle deliberately. For messaging, we picked AWS SQS: reliable, managed, and low on cognitive load. We didn't need ordering guarantees at the time, so a standard queue worked fine. We started with just two queues, and this setup held its ground. It handled a peak of 400 TPS during the IPL. The limitations showed up as the ecosystem grew. A single event, like a successful payment, needed to fan out into multiple workflows: Update NPCI with an API call Send callbacks to merchants about payment status Push structured data into our warehouse (AWS Redshift) To handle this, we started bolting on AWS SNS plus SQS for fan-ou

## Best Kafka Alternatives in 2026: Compared by Use Case

DevFeed: [Best Kafka Alternatives in 2026: Compared by Use Case](<https://devfeed.tech/articles/best-kafka-alternatives-in-2026-compared-by-use-case-82181.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/kafka-alternatives>)

Author: Emily Lucek

Published: 2025-02-05T17:39:20Z

Content type: comparison

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Redpanda](<https://devfeed.tech/topics/redpanda-data-streaming.md>), [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [alternatives](<https://devfeed.tech/tags/alternatives.md>), [amazon-kinesis](<https://devfeed.tech/tags/amazon-kinesis.md>), [amqp](<https://devfeed.tech/tags/amqp.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [aws](<https://devfeed.tech/tags/aws.md>), [azure](<https://devfeed.tech/tags/azure.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [compare](<https://devfeed.tech/tags/compare.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [kinesis](<https://devfeed.tech/tags/kinesis.md>), [pricing](<https://devfeed.tech/tags/pricing.md>)

### AI overview

This comparison matches Kafka alternatives to different workloads, including Kafka-compatible platforms, cloud-managed streaming services, message brokers, and CDC pipelines. It discusses trade-offs in operational effort, cost, compatibility, latency, retention, and cloud integration, and advises teams to choose based on what they use Kafka for.

### Source excerpt

Kafka too complex for your use case? Compare 9 alternatives including Redpanda, WarpStream, Confluent, Kinesis, and more. Honest pros, cons, and pricing included.

## Getting Started with Ruby and an AWS Managed Kafka Server

DevFeed: [Getting Started with Ruby and an AWS Managed Kafka Server](<https://devfeed.tech/articles/getting-started-with-ruby-and-an-aws-managed-kafka-server-28123.md>)

Original publisher: [Read original article](<http://fuzzyblog.io/blog/aws/2020/02/12/getting-started-with-ruby-and-an-aws-managed-kafka-server.html>)

Author: Fuzzygroup

Published: 2020-02-12T00:00:00Z

Content type: tutorial

Language: en

Sources: [Scott Johnson](<https://devfeed.tech/sources/scott-johnson.md>)

Topics: [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Ruby](<https://devfeed.tech/topics/ruby.md>), [X (Twitter)](<https://devfeed.tech/topics/twitter.md>), [Amazon VPC](<https://devfeed.tech/topics/amazon-vpc.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [ec2](<https://devfeed.tech/tags/ec2.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [networking](<https://devfeed.tech/tags/networking.md>), [pricing](<https://devfeed.tech/tags/pricing.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [sqs](<https://devfeed.tech/tags/sqs.md>)

### AI overview

The article walks through connecting Ruby 2.7 to an AWS Managed Kafka cluster in the same VPC, using ruby-kafka to list topics and send a message. It estimates a kafka.m5.large broker at $0.21/hour ($151.20/month), excluding storage, and says the total may rise with multiple availability zones and per-broker storage; the author is unsure how those costs are counted. Setup advice includes choosing the client's VPC and needed availability zones, retaining default 1 TB storage (which cannot later be reduced), and selecting a security group. After connection timeouts, the author fixed access by adding an inbound rule allowing all traffic from the client instance's security group to the default VPC security group. The article shows direct delivery and buffered producer delivery, noting the latter is intended for higher performance but that the author had not established their practical differences.

### Source excerpt

I am in the position of building a data pipeline that looks like this: Kafka as the core repository using an AWS Managed Kafka Server A data feed from Twitter feeding into Kafka using the ruby-kafka gem from Zendesk Python Machine Learning routines processing the data in Kafka and outputting it to SQS In this blog post I will walk through creating the Kafka server and then show you the initial configuration to get Ruby to talk to it. Core Assumptions Here are the core assumptions: AWS Managed Kafka Instance in the same VPC as an EC2 Instance Ruby 2.7 Minimal ruby script; just enough to connect and put one thing into a topic AWS Kafka Pricing The baseline pricing for an AWS kafka.m5.large cluster is: 0.21 / hour without factoring in storage. 0.21 / hour means nothing to me so here's the calculation: 0.21 * 24 = $5.04 / day 0.21 * 24 * 7 = $35.28 / week 0.21 * 24 * 30 = $151.20 / month 0.21 * 24 * 365 = $1839.60 / year Whether or not this is a good deal depends on your budget and application but what I can tell you is that the AWS Managed Kafka is phenomenally easy to setup and get started with. Note: Read thru the creation process below because I suspect it has pricing implications. Creating the Kafka Instance The new Managed Kafka service from AWS makes creating a Kafka instance absolutely trivial. You do need to know the VPC id of your instance to make things easier. You want to start here to create a Kafka cluster. Here are screenshots of the entire creation flow along with notes on anything I thought relevant. Naming your Kafka cluster is pretty simple and unless you are an expert I'd go with the standard configuration. AWS sometimes lags behind on their versions of open source stuff so keep that in mind. The standard configuration options are generally pretty good. I might argue that auto.create.topics.enable should be enabled by default but that's a debatable point and really one tied to application needs. The networking options are where you likely will get in