# How We Built Automated Capacity Testing for Kafka Consumers

DevFeed: [How We Built Automated Capacity Testing for Kafka Consumers](<https://devfeed.tech/articles/how-we-built-automated-capacity-testing-for-kafka-consumers-23723.md>)

Original publisher: [Read original article](<https://medium.com/booking-com-development/how-we-built-automated-capacity-testing-for-kafka-consumers-1853623bce78?source=rss----1c36c35f9c76---4>)

Author: Kaan Karakaya

Published: 2026-09-14T09:46:34Z

Content type: tutorial

Language: en

Sources: [Booking.com Development - Medium](<https://devfeed.tech/sources/booking-com-development-medium.md>)

Topics: [Kafka](<https://devfeed.tech/topics/kafka.md>), [Testing](<https://devfeed.tech/topics/testing.md>), [HTTP](<https://devfeed.tech/topics/http.md>)

Tags: [automated](<https://devfeed.tech/tags/automated.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [health-checks](<https://devfeed.tech/tags/health-checks.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [load](<https://devfeed.tech/tags/load.md>), [load-balancer](<https://devfeed.tech/tags/load-balancer.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>), [partition](<https://devfeed.tech/tags/partition.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [scale](<https://devfeed.tech/tags/scale.md>), [site-reliability-engineer](<https://devfeed.tech/tags/site-reliability-engineer.md>), [sre](<https://devfeed.tech/tags/sre.md>), [testing](<https://devfeed.tech/tags/testing.md>)

## AI overview

This article describes Booking.com's extension of an existing capacity-testing platform for Kafka consumers. It explains how changing partition assignment can provide a controlled, measurable way to test consumer throughput and whether remaining consumers can absorb reassigned work after an instance or failure domain disappears.

## Source excerpt

Photo by GuerrillaBuzz on Unsplash Kafka makes it easy to distribute work across consumer instances. It is much harder to prove, safely and repeatedly, how those instances behave when the distribution changes and one of them has to carry more than its usual share. For teams that run Kafka at scale, this is a practical reliability question: how much load can a consumer instance actually handle? We had automated capacity testing for HTTP services, but Kafka consumers were still tested with manual drills. Those drills could tell us something, but they were disruptive, difficult to reproduce, and risky precisely when the system was close to its limit. We wanted a controlled way to answer three questions: What is the maximum sustainable throughput of a consumer instance? If an instance or failure domain disappears, can the remaining consumers absorb the reassigned work? Are we overprovisioning resources because we do not know the real limit? The result was an extension to our capacity-testing platform that turns Kafka partition assignment into a safe, measurable load-control mechanism. Why HTTP capacity testing did not translate Our existing platform was designed for request-response services behind a load balancer. A scheduled test selects one instance, routes an increasing share of traffic to it, runs health checks after each step, and records the highest ratio the instance can sustain. After the test, traffic returns to its normal distribution and the result is reported to the service owner. Kafka has no equivalent traffic knob. Consumers pull records, and the unit of parallelism is the partition. Within a consumer group, each partition is owned by one consumer at a time. If a topic has 12 partitions and four equally loaded instances, each instance owns about three. When one instance disappears, a rebalance gives the survivors more partitions -- and the extra work arrives as a step change, not as a smooth increase from a load balancer. The key translation: for an HTTP