# kafka-consumer

Published articles for kafka-consumer.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## How We Cut Kafka Consumer Deployment Costs by 83%

DevFeed: [How We Cut Kafka Consumer Deployment Costs by 83%](<https://devfeed.tech/articles/how-we-cut-kafka-consumer-deployment-costs-by-83-28062.md>)

Original publisher: [Read original article](<https://tech.trivago.com/post/2026-06-12-how-we-cut-kafka-consumer-deployment-costs-by-83/>)

Author: ZhongLi Shen He creates fantastic software as a backend engineer GitHub profile Linkedin profile

Published: 2026-06-12T00:00:00Z

Content type: article

Language: en

Sources: [Trivago](<https://devfeed.tech/sources/trivago.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [cost-optimization](<https://devfeed.tech/topics/cost-optimization.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [Deployment](<https://devfeed.tech/topics/deployment.md>), [Post Mortem](<https://devfeed.tech/topics/post-mortem.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>)

Tags: [backend](<https://devfeed.tech/tags/backend.md>), [backpressure](<https://devfeed.tech/tags/backpressure.md>), [bottlenecks](<https://devfeed.tech/tags/bottlenecks.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [consumer](<https://devfeed.tech/tags/consumer.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [debugging](<https://devfeed.tech/tags/debugging.md>), [delay](<https://devfeed.tech/tags/delay.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [deployments](<https://devfeed.tech/tags/deployments.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [grpc](<https://devfeed.tech/tags/grpc.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [investigation](<https://devfeed.tech/tags/investigation.md>), [java](<https://devfeed.tech/tags/java.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [kafka-consumer](<https://devfeed.tech/tags/kafka-consumer.md>), [performance](<https://devfeed.tech/tags/performance.md>)

### AI overview

PSE-kafka's high lag and slow deployments came from several interacting bottlenecks, so isolated fixes had little effect. Its custom pull-based Kafka consumer delayed poll() and commits when processing slowed; a 75-second in-memory delay capped throughput, while a small gRPC thread pool also constrained consumption. Switching to spring-kafka removed the lag in testing but would have sacrificed backpressure, and reactor-kafka alone did not fix staging performance. The team combined reactor-kafka with higher concurrency and a larger gRPC pool to reach 8,000 messages per second in testing, then removed the delay after adding an IsFinished flag and tuned the pool for production. Reported results were pods reduced from 60 to 6, lag alerts ended, startup time fell from about 60 to 10 seconds, and business metrics remained unchanged.

### Source excerpt

This post walks through a layered performance investigation that cut PSE-kafka's infrastructure costs by 83% and ended a run of 19 P1 incidents.PSE-kafka (price-search-engine-kafka) i...

## How QA Validates Productivity Analytics Data Across Environments

DevFeed: [How QA Validates Productivity Analytics Data Across Environments](<https://devfeed.tech/articles/the-hidden-layer-of-analytics-how-qa-builds-trust-in-data-30520.md>)

Original publisher: [Read original article](<https://medium.com/helpshift-engineering/the-hidden-layer-of-analytics-how-qa-builds-trust-in-data-f2dcca8adf56?source=rss----3229f31ca4f4---4>)

Author: Gayatri Panganti

Published: 2025-11-26T13:55:02Z

Content type: article

Language: en

Sources: [Helpshift](<https://devfeed.tech/sources/helpshift.md>)

Topics: [Software Testing](<https://devfeed.tech/topics/software-testing.md>), [data](<https://devfeed.tech/topics/data.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [dashboards](<https://devfeed.tech/topics/dashboards.md>), [JSON](<https://devfeed.tech/topics/json.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [data-validation-testing](<https://devfeed.tech/tags/data-validation-testing.md>), [json](<https://devfeed.tech/tags/json.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [kafka-consumer](<https://devfeed.tech/tags/kafka-consumer.md>), [logging](<https://devfeed.tech/tags/logging.md>), [metric](<https://devfeed.tech/tags/metric.md>), [qa](<https://devfeed.tech/tags/qa.md>), [testing](<https://devfeed.tech/tags/testing.md>), [time](<https://devfeed.tech/tags/time.md>)

### AI overview

The article describes a primarily manual QA process for a productivity analytics framework. It uses simulations, event validation through Kafka, metric checks with Metabase, timestamp recalculations, and testing across sandbox, staging, and production environments.

### Source excerpt

Every accurate metric is backed by countless validations, events checks and integrity tests in the background.Introduction Quality Assurance in the data-driven systems extends beyond UI validation and backend verification. Such systems rely heavily on data precision and accuracy. A recent QA focused on validating a productivity analytics framework, ensuring that every event, metric and data flow accurately represented real-world user behaviour. The process was primarily manual, involving live simulations, event validation and detailed metric verification across environment which emphasised logic and data accuracy over automation. Simulating Real-World Scenarios Extensive simulations were conducted for multiple user roles as Agent, Supervisor, Admin and Super admin. Each governed by specific dashboard permissions and access rules. Actions tested included : Logging in and logging out from the dashboard Marking presence states such as Available, Online, Or Away Switching between workzones Navigating through dashboards These test conditions generated diverse event streams used to verify how accurately the system captured and processed state transitions. Parallel Event Validation Through Kafka Real-time validation was a key aspect of this QA process. Event streams were observed directly through kafka consoles, enabling verification of generated events and their payloads, data structures, and JSON fields as actions were executed. Each dashboard or SDK action was simulated while the corresponding kafka stream was monitored in parallel, confirming that events triggered correctly and carried accurate data information. Testing covered sandbox, staging, and production environments, ensuring reliability and consistency across all setups. Data Verification with Metabase and Calculations After events were processed by the analytics pipeline, metabase queries were used to validate computed metrics against expected outcomes from simulations. To cross-check results, timestamps (ex.

## Using Clojure channels to increase throughput

DevFeed: [Using Clojure channels to increase throughput](<https://devfeed.tech/articles/using-clojure-channels-to-increase-throughput-30521.md>)

Original publisher: [Read original article](<https://medium.com/helpshift-engineering/using-clojure-channels-to-increase-throughput-c051cc7f9893?source=rss----3229f31ca4f4---4>)

Author: Abhinav Dubey

Published: 2025-05-28T10:07:12Z

Content type: article

Language: en

Sources: [Helpshift](<https://devfeed.tech/sources/helpshift.md>)

Topics: [Clojure](<https://devfeed.tech/topics/clojure.md>), [multithreading](<https://devfeed.tech/topics/multithreading.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [threadpools](<https://devfeed.tech/topics/threadpools.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>)

Tags: [channel](<https://devfeed.tech/tags/channel.md>), [clojure](<https://devfeed.tech/tags/clojure.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [core-async](<https://devfeed.tech/tags/core-async.md>), [grafana](<https://devfeed.tech/tags/grafana.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [kafka-consumer](<https://devfeed.tech/tags/kafka-consumer.md>), [multithreading](<https://devfeed.tech/tags/multithreading.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>), [performance](<https://devfeed.tech/tags/performance.md>), [single-threaded](<https://devfeed.tech/tags/single-threaded.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [worker](<https://devfeed.tech/tags/worker.md>)

### AI overview

A Clojure Kafka consumer can process messages faster by placing them on a `core.async` channel and having worker threads take and process them; the example uses blocking `>!!` and `<!!` operations and `future`. With 1,000 events, processing time fell from 16m 30s with one worker to 5m 42s with three and 5m with ten. For 2,500 events, it fell from 40m 28s to 13m 39s with three workers and 5m 6s with ten. More workers increased throughput but also raised CPU use and system load, so the article recommends tuning thread count to system capacity. Benchmarks ran on one machine, which the author says is sufficient to gauge the impact of adding workers but may not represent production.

### Source excerpt

When building systems that process large volumes of messages synchronously, performance bottlenecks can quickly become a challenge specially with single-threaded designs. In this post, we'll look at how leveraging worker threads in a Clojure-based Kafka consumer can significantly boost throughput & reduce total processing time. Using simple concurrency primitives, it's possible to achieve parallelism & scale gracefully, all while keeping the codebase clean & maintainable. We'll start with a baseline, introduce worker threads using Clojure's core.async & measure the impact. Setup & Context Kafka & Zookeeper For observability: Grafana Kafka producer: A simple script that sends messages to a Kafka topic at a configurable rate (messages per minute) for a fixed duration. After each event is pushed, a counter metric is emitted Kafka consumer: A simple script that listens to a topic & consumes messages & simulates processing time finding square-root of a number (henceforth, assume that it takes ~1 second to find the square root) . A counter metric is emitted after processing each message The Baseline: Single-Threaded Consumer If each message takes t seconds to process & there are n messages, total processing time becomes n x t seconds. This provides a clean baseline to evaluate the impact of using channel moving forward. Adding workers with core.asyncValues are conveyed on queue-like channels. By default channels require producer and consumer to rendezvous for the transfer of a value through the channel https://clojuredocs.org/clojure.core.async To improve throughput, we introduce parallelism using Clojure's core.async channels. Messages from Kafka are fed into a channel, & multiple worker threads read from this channel to process messages concurrently Here, we used >!! (blocking put) & <!! (blocking take) to communicate via channels & future to execute the business-logic on a separate thread Who gets blocked & when : The thread putting message into the channel will get bl

## Kafka Consumer Lag: Causes, Impacts, and Solutions

DevFeed: [Kafka Consumer Lag: Causes, Impacts, and Solutions](<https://devfeed.tech/articles/kafka-consumer-lag-causes-impacts-and-solutions-80023.md>)

Original publisher: [Read original article](<https://www.groundcover.com/blog/kafka-slow-consumer>)

Author: Shahar Azulay

Published: 2023-06-22T00:00:00Z

Content type: tutorial

Language: en

Sources: [Ground Cover](<https://devfeed.tech/sources/ground-cover.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [Prometheus](<https://devfeed.tech/topics/prometheus.md>)

Tags: [backlog](<https://devfeed.tech/tags/backlog.md>), [bottlenecks](<https://devfeed.tech/tags/bottlenecks.md>), [cloudwatch](<https://devfeed.tech/tags/cloudwatch.md>), [data-streaming](<https://devfeed.tech/tags/data-streaming.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [kafka-consumer](<https://devfeed.tech/tags/kafka-consumer.md>), [lag](<https://devfeed.tech/tags/lag.md>), [performance](<https://devfeed.tech/tags/performance.md>), [real-time-data-streaming](<https://devfeed.tech/tags/real-time-data-streaming.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

Kafka consumer lag is the delay between message production and processing, which can hurt throughput, delay dependent applications, and contribute to data loss. This guide explains common causes--including partition imbalance, inefficient consumer code, resource limits, network problems, and traffic spikes--and covers monitoring and mitigation strategies.

### Source excerpt

Prevent Kafka consumer lag for real-time data streaming. Learn causes, impacts, and solutions for slow Kafka consumers to ensure optimal system performance.

## Apache Flink 1.3.2 Released

DevFeed: [Apache Flink 1.3.2 Released](<https://devfeed.tech/articles/apache-flink-1-3-2-released-48307.md>)

Original publisher: [Read original article](<https://flink.apache.org/2017/08/05/apache-flink-1.3.2-released/>)

Published: 2017-08-05T09:00:00Z

Content type: release

Language: en

Sources: [Apache Flink](<https://devfeed.tech/sources/apache-flink-stateful-computations-over-data-streams-on-apache-flink.md>)

Topics: [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [bug](<https://devfeed.tech/topics/bug.md>), [upgrade](<https://devfeed.tech/topics/upgrade.md>), [version](<https://devfeed.tech/topics/version.md>)

Tags: [apache](<https://devfeed.tech/tags/apache.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [bug](<https://devfeed.tech/tags/bug.md>), [flink](<https://devfeed.tech/tags/flink.md>), [hdfs](<https://devfeed.tech/tags/hdfs.md>), [improvements](<https://devfeed.tech/tags/improvements.md>), [issue](<https://devfeed.tech/tags/issue.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [kafka-consumer](<https://devfeed.tech/tags/kafka-consumer.md>), [maven](<https://devfeed.tech/tags/maven.md>), [migration](<https://devfeed.tech/tags/migration.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [partition](<https://devfeed.tech/tags/partition.md>), [release](<https://devfeed.tech/tags/release.md>), [upgrade](<https://devfeed.tech/tags/upgrade.md>), [upload](<https://devfeed.tech/tags/upload.md>), [version](<https://devfeed.tech/tags/version.md>)

### AI overview

Apache Flink 1.3.2 is a bugfix release containing more than 60 fixes and minor improvements for Flink 1.3.1. It fixes a FlinkKafkaConsumer partition-assignment bug that could duplicate data in large Kafka deployments and includes Kafka, Kinesis, and dependency updates.

### Source excerpt

The Apache Flink community released the second bugfix version of the Apache Flink 1.3 series. This release includes more than 60 fixes and minor improvements for Flink 1.3.1. The list below includes a detailed list of all fixes. We highly recommend all users to upgrade to Flink 1.3.2. Important Notice: A user reported a bug in the FlinkKafkaConsumer (FLINK-7143) that is causing incorrect partition assignment in large Kafka deployments in the presence of inconsistent broker metadata.