# Allocate shared Amazon MSK costs to Kafka topics for cost visibility

DevFeed: [Allocate shared Amazon MSK costs to Kafka topics for cost visibility](<https://devfeed.tech/articles/allocate-shared-amazon-msk-costs-to-kafka-topics-for-cost-visibility-129557.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/allocate-shared-amazon-msk-costs-to-kafka-topics-for-cost-visibility/>)

Author: Durai Krishnan

Published: 2026-11-11T19:46:04Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [cloud cost management](<https://devfeed.tech/topics/cloud-cost-management.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [Amazon Managed Service for Prometheus](<https://devfeed.tech/topics/amazon-managed-service-for-prometheus.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Prometheus](<https://devfeed.tech/topics/prometheus.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [aws-sns](<https://devfeed.tech/topics/aws-sns.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-eventbridge](<https://devfeed.tech/tags/amazon-eventbridge.md>), [amazon-managed-service-for-prometheus](<https://devfeed.tech/tags/amazon-managed-service-for-prometheus.md>), [amazon-managed-streaming-for-apache-kafka-amazon-msk](<https://devfeed.tech/tags/amazon-managed-streaming-for-apache-kafka-amazon-msk.md>), [amazon-msk](<https://devfeed.tech/tags/amazon-msk.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-simple-storage-service-amazon-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-amazon-s3.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cost-and-usage-report](<https://devfeed.tech/tags/aws-cost-and-usage-report.md>), [billing](<https://devfeed.tech/tags/billing.md>), [cloudwatch](<https://devfeed.tech/tags/cloudwatch.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cost-and-usage-report](<https://devfeed.tech/tags/cost-and-usage-report.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [prometheus](<https://devfeed.tech/tags/prometheus.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [topic](<https://devfeed.tech/tags/topic.md>), [visibility](<https://devfeed.tech/tags/visibility.md>)

## AI overview

The pipeline estimates each Kafka topic's share of an Amazon MSK Provisioned cluster's daily cost by joining cluster-level CUR charges with per-topic ingress metrics, mapping cluster names to ARNs, then allocating cost in proportion to bytes in. A daily Lambda writes Parquet results to S3 for Athena and Quick Sight. Enhanced Monitoring publishes per-topic CloudWatch metrics with little setup, but metric charges grow with topic-broker pairs. Open Monitoring exposes broker metrics at no additional charge; a Fargate scraper sends them to Amazon Managed Service for Prometheus, whose ingestion costs grow with samples. Both options use the same allocation logic. Bytes in is only a proxy: it misses retention, replication, compression, and consumer fan-out. Idle topics receive no row, so active topics may absorb the full cost; if no topics report traffic, the function splits cost evenly across configured topics. CUR can lag up to 24 hours, and metrics exist only after they are enabled. The method applies to Provisioned clusters; Serverless is out of scope.

## Source excerpt

Amazon MSK bills at the cluster level, with no per-topic breakdown. This post shows how to allocate shared MSK cluster costs to individual Kafka topics by combining AWS Cost and Usage Report billing with per-topic CloudWatch or Prometheus metrics, giving each team a usage-based view of its spend.