# Parallelism Published articles for Parallelism. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## How We Built Automated Capacity Testing for Kafka Consumers DevFeed: [How We Built Automated Capacity Testing for Kafka Consumers]() Original publisher: [Read original article]() Author: Kaan Karakaya Published: 2026-09-14T09:46:34Z Content type: tutorial Language: en Sources: [Booking.com Development - Medium]() Topics: [Kafka](), [Testing](), [HTTP]() Tags: [automated](), [capacity](), [health-checks](), [kafka](), [load](), [load-balancer](), [parallelism](), [partition](), [recovery](), [scale](), [site-reliability-engineer](), [sre](), [testing]() ### AI overview This article describes Booking.com's extension of an existing capacity-testing platform for Kafka consumers. It explains how changing partition assignment can provide a controlled, measurable way to test consumer throughput and whether remaining consumers can absorb reassigned work after an instance or failure domain disappears. ### Source excerpt Photo by GuerrillaBuzz on Unsplash Kafka makes it easy to distribute work across consumer instances. It is much harder to prove, safely and repeatedly, how those instances behave when the distribution changes and one of them has to carry more than its usual share. For teams that run Kafka at scale, this is a practical reliability question: how much load can a consumer instance actually handle? We had automated capacity testing for HTTP services, but Kafka consumers were still tested with manual drills. Those drills could tell us something, but they were disruptive, difficult to reproduce, and risky precisely when the system was close to its limit. We wanted a controlled way to answer three questions: What is the maximum sustainable throughput of a consumer instance? If an instance or failure domain disappears, can the remaining consumers absorb the reassigned work? Are we overprovisioning resources because we do not know the real limit? The result was an extension to our capacity-testing platform that turns Kafka partition assignment into a safe, measurable load-control mechanism. Why HTTP capacity testing did not translate Our existing platform was designed for request-response services behind a load balancer. A scheduled test selects one instance, routes an increasing share of traffic to it, runs health checks after each step, and records the highest ratio the instance can sustain. After the test, traffic returns to its normal distribution and the result is reported to the service owner. Kafka has no equivalent traffic knob. Consumers pull records, and the unit of parallelism is the partition. Within a consumer group, each partition is owned by one consumer at a time. If a topic has 12 partitions and four equally loaded instances, each instance owns about three. When one instance disappears, a rebalance gives the survivors more partitions -- and the extra work arrives as a step change, not as a smooth increase from a load balancer. The key translation: for an HTTP ## Where Kafka Fits in Modern Systems DevFeed: [Where Kafka Fits in Modern Systems]() Original publisher: [Read original article]() Author: Nikki Siapno Published: 2026-08-25T12:02:09Z Content type: tutorial Language: en Sources: [Level Up Coding System Design Newsletter]() Topics: [Kafka](), [Event-Streaming](), [systems](), [distributed-systems](), [data]() Tags: [architecture](), [consumer](), [distributed-systems](), [durability](), [event-streaming](), [kafka](), [parallelism](), [partition](), [pipelines](), [real-time](), [systems]() ### AI overview This article explains when Apache Kafka is appropriate for distributed systems. It presents Kafka as a durable event log whose replayability, consumer fan-out, and partition-based parallelism support real-time data pipelines and shared event-driven architectures, while warning that Kafka can add unnecessary operational complexity for simpler asynchronous workflows. ### Source excerpt When should you actually use Kafka? And when is it just unnecessary complexity? ## Coroutine Essentials DevFeed: [Coroutine Essentials]() Original publisher: [Read original article]() Author: author@typealias.com (Dave Leeds) Published: 2026-08-21T00:00:00Z Content type: tutorial Language: en Sources: [Dave Leeds on Kotlin - typealias.com]() Topics: [Coroutines](), [Kotlin](), [coding](), [Software]() Tags: [async](), [cancellation](), [coding](), [concurrency](), [coroutine](), [coroutines](), [deferred](), [dispatcher](), [exception-handling](), [introduction](), [job](), [kotlin](), [launch](), [learn-to-program](), [parallelism](), [programming](), [runblocking](), [software](), [structured-concurrency](), [suspend-function](), [withcontext](), [yield]() ### AI overview This tutorial introduces essential Kotlin coroutine concepts for performing multiple tasks concurrently, such as making network calls while updating a screen. It aims to provide a foundation for day-to-day coding and more advanced coroutine concepts. ### Source excerpt When you're on hold during a phone call, you might also check your email. While brewing coffee, you might also cook breakfast. And while driving a car, you might listen to a podcast. In the same way, sometimes it's helpful for the software that we write to do more than one thing at a time. For example, it could make two or three network calls at one time--all while updating the screen to show the progress of each call. ## The Orchestrator's Tax DevFeed: [The Orchestrator's Tax]() Original publisher: [Read original article]() Author: Martin Fowler (martin@martinfowler.com) Published: 2026-07-28T13:10:00Z Content type: article Language: en Sources: [Martin Fowler]() Topics: [Multi Agent Systems](), [systems](), [Claude Code](), [incident]() Tags: [agents](), [delegation](), [incident](), [memory](), [multi-agent](), [parallelism](), [workflow]() ### AI overview Rahul Garg argues that the main cost of subagents in long-running multi-agent work is not necessarily execution time or parallelism, but the orchestrator's limited working memory. Drawing on an exploratory Claude Code session involving four subagents, the article examines delegation, context overhead, and the need for explicit delegation rules. ### Source excerpt Subagents get justified by time saved and parallel execution, but Rahul Garg explains that's not what matters most. Every token in the orchestrator's context is competing for its attention, and the real value of a subagent is what it keeps out of that context. Subagents should be treated as a tool for protecting the orchestrator's working memory, offloading reasoning it doesn't need to hold onto. Doing this well means giving the orchestrator explicit ground rules for when and how to delegate. more... ## How Lawmatics Cut CI Compute Cost by 39.3% and Shortened Pipeline Time by 15.8% DevFeed: [How Lawmatics Cut CI Compute Cost by 39.3% and Shortened Pipeline Time by 15.8%]() Original publisher: [Read original article]() Author: Christian Gómez Alonso Published: 2026-07-27T15:23:37Z Content type: article Language: en Sources: [Semaphore Engineering]() Topics: [ci](), [Optimization](), [cpu](), [browser]() Tags: [browser](), [ci](), [ci-cd](), [compute](), [cost](), [cpu](), [optimization](), [parallel](), [parallelism](), [pipeline](), [time]() ### AI overview This case study describes how Lawmatics optimized its CI pipeline by changing the machine class used for parallel browser end-to-end tests while initially preserving parallelism. The reported result was a 39.3% reduction in average application compute cost per pipeline and a 15.8% reduction in active pipeline duration. ### Source excerpt CI optimization is easiest to reason about when the problem is concrete: one pipeline, one critical path, and one cost model. Lawmatics reached out to us with that kind of problem. Their application pipeline was already parallelized and already using a sensible CI structure. The remaining question was whether the most expensive part of the [...] The post How Lawmatics Cut CI Compute Cost by 39.3% and Shortened Pipeline Time by 15.8% appeared first on Semaphore. ## How the Gradle Team Adopted Isolated Projects DevFeed: [How the Gradle Team Adopted Isolated Projects]() Original publisher: [Read original article]() Author: Alex Semin Published: 2026-07-22T04:00:00Z Content type: article Language: en Sources: [The Gradle Blog]() Topics: [Gradle](), [Scalability](), [Cache](), [configuration](), [Dependency management](), [ci](), [ide]() Tags: [cache](), [ci](), [configuration](), [dependency-management](), [gradle](), [ide](), [parallelism](), [performance]() ### AI overview The Gradle team describes adopting the experimental Isolated Projects feature in its own build. The article explains that Isolated Projects runs project configuration in parallel to improve scalability and performance, and outlines migration prerequisites, diagnostics, convention plugins, dependency management, and Shared Build Services. ### Source excerpt Isolated Projects is an experimental feature aimed at improving Gradle scalability and performance. It speeds up builds by running project configuration in parallel, which benefits practically every workflow, from CI builds to IDE sync. We want the feature to offer an excellent user experience, so we made it a goal to adopt Isolated Projects in our own build, even though it is still in active development. This lets us find and smooth out the rough spots early, though secretly, we were also looking forward to the productivity boost for our own team. With Isolated Projects, more build logic runs concurrently. To keep builds reliable under that parallelism, the feature introduces additional constraints, and adopting it means migrating your build to address the violations of those constraints. Migrating our build Isolated Projects builds directly on top of the Configuration Cache, so making the build Configuration Cache compatible is a prerequisite. Fortunately for us, we've been enjoying fast inner developer loops for many years now, so that box was already checked. The general approach to migrating against new constraints is to start with the simplest workflows, such as running the help task, and address the violations. Then proceed to more complex workflows, such as IDE sync or CI. Running in Diagnostics mode helps you see all violations at once in a single HTML report, and the guide describes the recommended migration path. For the Gradle build, we followed a similar path. What helped us the most and made the migration much smoother is following the best practices, especially the best practice of having convention plugins: each project applies the build logic it needs, and no project has to reach in and mutate another's state directly. We've leaned heavily on sharing artifacts between projects only via dependency management, which is inherently compatible with Isolated Projects by making each project define its "outputs", on which other projects can depend. For more ## Npb: Nixpkgs PR build diff CLI inspired by nixpkgs-review DevFeed: [Npb: Nixpkgs PR build diff CLI inspired by nixpkgs-review]() Original publisher: [Read original article]() Author: samestep Published: 2026-07-21T21:09:29Z Content type: article Language: en Sources: [Announcements - NixOS Discourse]() Topics: [Nix](), [Development](), [Command-line interface]() Tags: [announcements](), [caching](), [cli](), [memory](), [parallelism](), [text]() ### AI overview A developer introduces npb, a Nixpkgs pull-request build outcome diff CLI inspired by nixpkgs-review. It compares build results before and after a change, and adds features including passthru.test inclusion, cached failures, lower memory usage, parallel Nix evaluations, reproducible report commands, and progress indicators. ### Source excerpt Hello! While reviewing Nixpkgs PRs written by myself and others, I've found it difficult to keep track of which build failures were preexisting and which are regressions introduced by the PR itself. So I built a tool called npb which produces reports that are similar to nixpkgs-review, but show both before and after, instead of just after. github.com GitHub - samestep/npb: Nixpkgs build outcome diff CLI Nixpkgs build outcome diff CLI While building this, I also included some other features I had found myself wanting previously, such as: automatically including passthru.tests caching build failures in a local database reducing memory usage of Nix evaluations speeding up Nix evaluations via parallelism giving each report a command that reproduces the same report providing progress indicators during long-running operations I hope you find it useful! Disclosure, quoting the project's README: The development of this project was AI-assisted. There is no AI-generated text in this README, npb --help, or npb's generated reports. 7 posts - 4 participants Read full topic ## Releasing Execution Contexts DevFeed: [Releasing Execution Contexts]() Original publisher: [Read original article]() Author: Julien Portalier Published: 2026-07-12T00:00:00Z Content type: release Language: en Sources: [Crystal]() Topics: [Crystal](), [Concurrency](), [Concurrent Programming](), [context](), [cpu]() Tags: [concurrent](), [cpu](), [kotlin](), [main-thread](), [multithreading](), [parallelism](), [threads]() ### AI overview Crystal is introducing Execution Contexts, an interface for orchestrating fibers across one or more threads. The article describes concurrent and parallel contexts, configurable parallelism, and execution across CPU cores while preserving a single-threaded default. ### Source excerpt Two and a half years ago, with the invaluable support from 84codes, we re-examined the multithreading model inherited from Crystal 0.28 (preview MT). ## Using OxCaml to implement type-safe reference counting between OCaml and Python DevFeed: [Using OxCaml to implement type-safe reference counting between OCaml and Python]() Original publisher: [Read original article]() Author: Nicolas Trangez Published: 2026-06-15T00:00:00Z Content type: tutorial Language: en Sources: [Jane Street]() Topics: [OCaml](), [Python](), [Programming language](), [Programming](), [Data analysis](), [Machine learning]() Tags: [compiler](), [gc](), [memory](), [memory-management](), [ocaml](), [parallelism](), [programming-language](), [python]() ### AI overview Jane Street describes using OxCaml extensions to implement type-safe reference counting between OCaml and Python. The approach addresses delayed deallocation when objects cross the language boundary, including for large data frames and scarce resources such as GPU memory. ### Source excerpt Jane Street is known for being an OCaml shop, but for years now Python has been our second major programming language, acting as the primary tool for data analysis and (especially importantly these days) machine learning. Most of our traders and researchers think and write in Python, even as the majority of our infrastructure is written in OCaml. ## Achieving Fast Inner Dev Loops with Gradle: Configuration Cache and Beyond DevFeed: [Achieving Fast Inner Dev Loops with Gradle: Configuration Cache and Beyond]() Original publisher: [Read original article]() Author: Alex Semin Published: 2026-05-20T04:00:00Z Content type: article Language: en Sources: [The Gradle Blog]() Topics: [Gradle](), [code productivity](), [build performance](), [build times](), [Development]() Tags: [build](), [build-tool](), [compilation](), [context](), [developer-productivity](), [flow](), [gradle](), [incremental](), [kotlinconf](), [parallelism](), [productivity]() ### AI overview This article explains how Gradle is working to accelerate inner developer loops. It describes the shift from execution time to configuration time as the main build bottleneck in large codebases, while highlighting configuration cache, incremental builds, build cache, and parallelism as approaches to reduce waiting and preserve developer flow. ### Source excerpt How many engineers in your organization have to sit idle for 10 or more minutes just to rebuild the app locally after a small change? If you're an engineering leader, build engineer, or part of a developer productivity team, you're not alone in facing this challenge. Building complex software is inherently demanding, but the time your developers spend waiting has a profound impact beyond just the literal minutes lost. It fundamentally disrupts their psychological flow. "There are like these time scales of psychological flow, right? So if something takes longer than a minute, you already start planning for it... you suffer from context switch as well. If it's 1 second, that's where we want to be right in that seamless state of flow. And if it's less than one second, then you feel it's like magic... productive developers are happy developers." -- Rodrigo Oliveira When developers have to wait, their attention drifts, and the cost of context switching destroys their momentum. Our goal as enablers of developer productivity should be to eliminate this friction, bringing that feedback loop as close as possible to the "magical one second". In a recent presentation at KotlinConf 2025, Gradle engineers Rodrigo Oliveira and Alex Semin detailed how Gradle Build Tool is evolving to keep developers in that productive, happy "flow state". By tackling the most significant bottlenecks in modern software builds, Gradle is paving the way for incredibly fast inner developer loops. If you prefer to watch the presentation recording, you can find it here. The shifting bottleneck: From execution to configuration On every invocation, Gradle goes through three distinct phases: initialization, configuration, and execution. Historically, when we optimized Gradle builds, we focused almost exclusively on the execution phase. This is when the majority of work expected by a developer happens: production sources compilation, test execution, etc. For years, execution was the longest phase. We tackled thi ## Workflow-as-code: orchestration in pure code DevFeed: [Workflow-as-code: orchestration in pure code]() Original publisher: [Read original article]() Author: Ruben Fiszel Published: 2026-04-03T00:00:00Z Content type: release Language: en Sources: [Windmill Blog]() Topics: [Orchestration](), [Code](), [Python](), [TypeScript](), [Concurrency](), [Error Handling]() Tags: [architecture](), [code](), [concurrency](), [editor](), [error-handling](), [execution](), [launch-week](), [orchestration](), [parallelism](), [python](), [releases](), [typescript](), [version-control](), [workflow](), [workflows]() ### AI overview Windmill introduces workflow-as-code, allowing complex workflows to be defined in TypeScript or Python. The system uses checkpointing and replay, runs tasks as separate jobs, supports parallelism and fault tolerance, and releases workers while workflows wait. ### Source excerpt Introducing workflow-as-code: define complex workflows in TypeScript or Python with checkpointing, parallelism, and fault tolerance built in. ## How Apollo Built an AI-Powered Style Guide Reviewer for Documentation DevFeed: [How Apollo Built an AI-Powered Style Guide Reviewer for Documentation]() Original publisher: [Read original article]() Author: Daniel Abdelsamed Published: 2026-03-16T12:33:22Z Content type: article Language: en Sources: [Apollo Blog]() Topics: [Artificial Intelligence](), [Documentation](), [AI Development](), [GitHub](), [Caching]() Tags: [ai](), [caching](), [cost](), [docs](), [engineering](), [false-positives](), [github](), [parallelism](), [speed]() ### AI overview Apollo's documentation team built an AI-powered style guide reviewer that applies documentation rules contextually and reports findings to authors in GitHub. The article explains how restructuring the style guide, integrating the review into the existing workflow, and optimizing speed and cost made it practical to run on every commit. ### Source excerpt Learn how Apollo's docs team built an AI-powered style guide reviewer that runs on every commit -- and the three things that made it actually work: restructuring the style guide as a pattern library, integrating results directly into GitHub, and optimizing for speed and cost with caching, line-level review, and parallelism. ## How Agoda Load Balanced Kafka DevFeed: [How Agoda Load Balanced Kafka]() Original publisher: [Read original article]() Author: Saurabh Dashora Published: 2026-03-10T08:01:37Z Content type: article Language: en Sources: [System Design Codex]() Topics: [Kafka](), [data-processing](), [systems]() Tags: [data-centers](), [data-processing](), [kafka](), [parallelism](), [partitioning](), [round-robin]() ### AI overview A deep dive into how Agoda uses Kafka to process large volumes of real-time supplier price updates. It explains the distributor, processor, and Kafka components, then examines how partitions, partitioners, and consumer assignors support parallel processing and expose challenges when workloads and consumer capabilities are uneven. ### Source excerpt Deep Dive ## RCCLX: Innovating GPU Communications on AMD Platforms DevFeed: [RCCLX: Innovating GPU Communications on AMD Platforms]() Original publisher: [Read original article]() Author: Sudharssun Subramanian; Subodh Iyengar; Cen Zhao; Srinath Bayareddy; James Hongyi Zeng Published: 2026-02-24T21:30:54Z Content type: article Language: en Sources: [Meta AI Research]() Topics: [GPU](), [Meta](), [communications](), [AI Models](), [Latency](), [Inference](), [Large Language Model](), [Transformer]() Tags: [accelerate](), [ai](), [ai-models](), [ai-research](), [communications](), [data-center-engineering](), [gpu](), [inference](), [large-language-model](), [latency](), [layer](), [meta](), [ml-applications](), [networking-traffic](), [open-source](), [parallelism]() ### AI overview Meta describes the initial open-source release of RCCLX, an enhanced version of RCCL for AMD platforms integrated with Torchcomms. The article presents Direct Data Access algorithms and Low Precision Collectives, including approaches intended to reduce communication latency during large language model inference. ### Source excerpt We are open-sourcing the initial version of RCCLX - an enhanced version of RCCL that we developed and tested on Meta's internal workloads. RCCLX is fully integrated with Torchcomms and aims to empower researchers and developers to accelerate innovation, regardless of their chosen backend. Communication patterns for AI models are constantly evolving, as are hardware [...] Read More... The post RCCLX: Innovating GPU Communications on AMD Platforms appeared first on Engineering at Meta. ## Gastown, and where software is going DevFeed: [Gastown, and where software is going]() Original publisher: [Read original article]() Published: 2026-01-20T00:00:00Z Content type: opinion Language: en Sources: [Chainguard: Unchained]() Topics: [Development](), [AI Development](), [Terminal](), [Software Engineering]() Tags: [agent](), [agent-based-programming](), [agents](), [ai-coding](), [anthropic](), [claude](), [claude-gastown](), [gastown](), [parallelism](), [terminal](), [ux](), [workflows]() ### AI overview An opinion article examining Gastown as a prototype for agent-driven software work. It highlights terminal-based coordination, distributed tasks, worktree isolation, and the importance of CI, guardrails, and shared truth, while noting that the system remains unreliable in practice. ### Source excerpt Gastown hints at the future of software: agent-driven workflows where CI, guardrails, and shared truth matter more than writing code faster. ## The Hidden Cost of Convenience: Rethinking Old ORM Patterns for Scale DevFeed: [The Hidden Cost of Convenience: Rethinking Old ORM Patterns for Scale]() Original publisher: [Read original article]() Author: Andrew Houts Published: 2025-12-08T22:06:50Z Content type: article Language: en Sources: [Wealthfront]() Topics: [Object-relational mapping](), [Code](), [data](), [Tooling](), [dashboards](), [Database](), [Concurrency]() Tags: [backend](), [best-practices](), [code](), [data](), [database](), [flame-graph](), [orm](), [parallelism](), [performance](), [profiling](), [sql](), [tooling](), [wealthfront-engineering]() ### AI overview Wealthfront examines performance problems in an aging portfolio balance calculation system whose overnight batch processing scaled poorly as client growth and assets under management increased. The article describes profiling the job with command-line and sampling-profiler tools, using collected data and flame graphs to investigate where runtime was spent. ### Source excerpt Ever been here before? Stuck with a job that needs to be continually revisited because its performance gets worse with every passing day, and each attempt at improving said performance yields diminishing returns? This is the situation we found ourselves in with the portfolio balance calculation system--the code responsible for aggregating data from multiple sources... Read more ## How Zed Improved Project Search Performance in Release 214.4 DevFeed: [How Zed Improved Project Search Performance in Release 214.4]() Original publisher: [Read original article]() Author: Piotr Osiewicz Published: 2025-11-26T00:00:00Z Content type: article Language: en Sources: [Zed Industries - Blog]() Topics: [Code](), [vs-code]() Tags: [code](), [files](), [parallelism](), [performance](), [project](), [search](), [vs-code]() ### AI overview This article explains why Zed's project search was slow and describes the performance improvements released in Zed 214.4. It covers filesystem scanning, in-memory buffers, settings and .gitignore handling, and parallel execution. ### Source excerpt Why was it slow, and why is it still slow? ## Scaling LLM Inference: Innovations in Tensor Parallelism, Context Parallelism, and Expert Parallelism DevFeed: [Scaling LLM Inference: Innovations in Tensor Parallelism, Context Parallelism, and Expert Parallelism]() Original publisher: [Read original article]() Author: Cen Zhao; Xiaodong Wang; Jianyu Huang Published: 2025-10-17T16:00:50Z Content type: article Language: en Sources: [Meta AI Research]() Topics: [Large Language Model](), [AI Inference](), [Inference Performance](), [LLM Techniques](), [GPU](), [sharding](), [long-context]() Tags: [ai-research](), [gpu](), [inference](), [kv-cache](), [large-language-models-llms](), [latency](), [llms](), [parallelism](), [performance](), [sharding]() ### AI overview Meta describes three forms of parallelism--tensor, context, and expert parallelism--for scaling large language model inference across GPUs. The article explains how prefill and decoding differ computationally and how these techniques target resource efficiency, throughput, and latency. ### Source excerpt At Meta, we are constantly pushing the boundaries of LLM inference systems to power applications such as the Meta AI App. We're sharing how we developed and implemented advanced parallelism techniques to optimize key performance metrics related to resource efficiency, throughput, and latency. The rapid evolution of large language models (LLMs) has ushered in a [...] Read More... The post Scaling LLM Inference: Innovations in Tensor Parallelism, Context Parallelism, and Expert Parallelism appeared first on Engineering at Meta. ## ClickHouse Release 25.8 DevFeed: [ClickHouse Release 25.8]() Original publisher: [Read original article]() Author: ClickHouse Published: 2025-09-05T00:00:00Z Content type: release Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [parquet](), [Flight](), [Apache Iceberg]() Tags: [apache-arrow](), [bug](), [bug-fixes](), [clickhouse](), [data](), [data-lake](), [features](), [files](), [format](), [integration](), [new-features](), [parallelism](), [parquet](), [partitioning](), [performance](), [release](), [storage](), [streaming]() ### AI overview ClickHouse 25.8 introduces 45 new features, 47 performance optimizations, and 119 bug fixes. Highlights include a faster experimental native Parquet reader, Data Lake improvements, Hive-style partitioned writes, initial PromQL support, and Arrow Flight integration. ### Source excerpt ClickHouse 25.8 is available. In this post, you will learn about new features, including a new Parquet reader and Arrow Flight integration. ## Using Clojure channels to increase throughput DevFeed: [Using Clojure channels to increase throughput]() Original publisher: [Read original article]() Author: Abhinav Dubey Published: 2025-05-28T10:07:12Z Content type: tutorial Language: en Sources: [Helpshift]() Topics: [Clojure](), [Concurrency](), [Kafka](), [benchmarking](), [async](), [Grafana]() Tags: [benchmarking](), [channel](), [clojure](), [concurrency](), [core-async](), [grafana](), [kafka](), [multithreading](), [parallelism]() ### AI overview This tutorial explains how to increase throughput in a Clojure-based Kafka consumer by using core.async channels and multiple worker threads. It compares a single-threaded baseline with configurations using three and ten workers, reporting shorter processing times for 1,000 events in the described benchmark. ### Source excerpt When building systems that process large volumes of messages synchronously, performance bottlenecks can quickly become a challenge specially with single-threaded designs. In this post, we'll look at how leveraging worker threads in a Clojure-based Kafka consumer can significantly boost throughput & reduce total processing time. Using simple concurrency primitives, it's possible to achieve parallelism & scale gracefully, all while keeping the codebase clean & maintainable. We'll start with a baseline, introduce worker threads using Clojure's core.async & measure the impact. Setup & Context Kafka & Zookeeper For observability: Grafana Kafka producer: A simple script that sends messages to a Kafka topic at a configurable rate (messages per minute) for a fixed duration. After each event is pushed, a counter metric is emitted Kafka consumer: A simple script that listens to a topic & consumes messages & simulates processing time finding square-root of a number (henceforth, assume that it takes ~1 second to find the square root) . A counter metric is emitted after processing each message The Baseline: Single-Threaded Consumer If each message takes t seconds to process & there are n messages, total processing time becomes n x t seconds. This provides a clean baseline to evaluate the impact of using channel moving forward. Adding workers with core.asyncValues are conveyed on queue-like channels. By default channels require producer and consumer to rendezvous for the transfer of a value through the channel https://clojuredocs.org/clojure.core.async To improve throughput, we introduce parallelism using Clojure's core.async channels. Messages from Kafka are fed into a channel, & multiple worker threads read from this channel to process messages concurrently Here, we used >!! (blocking put) & ) Original publisher: [Read original article]() Published: 2025-05-12T00:00:00Z Content type: tutorial Language: en Sources: [William Kennedy]() Topics: [Concurrency](), [Go Language](), [benchmarking](), [IO](), [Sorting]() Tags: [ardan-labs](), [benchmarks](), [blog](), [concepts](), [concurrency](), [go](), [go-programming](), [golang](), [io](), [parallelism](), [performance](), [programming](), [sorting](), [technical]() ### AI overview This third post in a series about the Go scheduler explains concurrency as out-of-order execution and distinguishes it from parallelism. It discusses how CPU-bound and I/O-bound workloads affect the decision to use concurrency, using practical examples and benchmarks to show that parallelism can improve CPU-bound performance while concurrency alone can benefit I/O-bound work. ### Source excerpt Although originally written in 2018, the following concepts remain essential for developers working with concurrency. This blogpost focuses on concurrency, distinguishing it from parallelism by defining it as "out of order" execution. It emphasizes the importance of understanding workload types--CPU bound (e.g., summing, sorting) and IO bound (e.g., file reading)--to assess when concurrency is appropriate. Through practical examples and benchmarks, it shows that parallelism boosts performance for CPU bound tasks, while concurrency alone benefits IO bound workloads. The post underscores that identifying workload type is key to applying concurrency effectively without added complexity. ## Parallelism with Android SQLite DevFeed: [Parallelism with Android SQLite]() Original publisher: [Read original article]() Author: Pierre-Yves Ricau Published: 2025-02-05T05:35:36Z Content type: tutorial Language: en Sources: [Py's blog]() Topics: [SQLite](), [Android](), [Concurrency](), [Coroutines](), [IO]() Tags: [android](), [blocking](), [connection-pool](), [coroutines](), [io](), [parallelism](), [performance](), [sqlite]() ### AI overview This article explains how Android SQLite's default single-connection behavior causes parallel queries on the same database to block one another. It examines how excessive query parallelism can consume coroutine dispatcher threads and delay unrelated I/O tasks, then discusses using separate dispatchers for blocking database work. ### Source excerpt The SQLDelight documentation provides this example: val players: Flow> = playerQueries.selectAll() .asFlow() .mapToList(Dispatchers.IO) This looks reasonable, right? In the Square Point Of Sale application, we recently ... ## Fearless Concurrency Ep.3: Managing Threaded Programs and Data Races in Rust DevFeed: [Fearless Concurrency Ep.3: Managing Threaded Programs and Data Races in Rust]() Original publisher: [Read original article]() Published: 2024-10-10T00:00:00Z Content type: tutorial Language: en Sources: [William Kennedy]() Topics: [Rust](), [Concurrency](), [Library]() Tags: [advanced-rust-threading](), [concurrency](), [datasets](), [efficient-parallelism-in-rust](), [optimize-rust-code](), [parallel-processing-in-rust](), [parallelism](), [performance](), [performance-optimization-rust](), [pitfalls](), [rayon-library-rust](), [rust](), [rust-concurrency-best-practices](), [rust-concurrency-tutorial](), [rust-parallel-computing](), [rust-scientific-computing-with-threads](), [rust-scoped-threads-vs-std-thread](), [rust-threading-strategies](), [rust-work-stealing-with-rayon](), [safe-concurrency-in-rust](), [safety](), [scoped-threads-in-rust](), [series](), [techniques](), [threading](), [threads]() ### AI overview Episode 3 of the Fearless Concurrency series explains scoped threads and the Rayon library in Rust. It compares standard threads, scoped threads, and Rayon, covering safe shared-data access, parallel processing of datasets, CPU workload distribution, and the risk of over-parallelization. ### Source excerpt Introduction: Welcome to Episode 3 of the Fearless Concurrency series! In this episode, Herbert explores advanced threading techniques in Rust, focusing on scoped threads and the Rayon library to help developers build more efficient and parallelized applications. This episode is packed with practical insights on when and how to use different threading models to optimize performance and handle complex workloads. Scoped Threads: Ensuring safe access to shared data with guaranteed lifetimes for concurrent threads. ## Revisiting HPX DevFeed: [Revisiting HPX]() Original publisher: [Read original article]() Author: brakmic Published: 2024-09-23T09:53:03Z Content type: tutorial Language: en Sources: [Harris Brakmic - Coding]() Topics: [distributed-systems](), [Programming](), [Testing]() Tags: [article](), [asynchronous](), [c-plus-plus](), [coding](), [cpp](), [distributed](), [distributed-computing](), [examples](), [framework](), [hpx](), [installation](), [macos](), [parallel](), [parallelism](), [programming](), [testing]() ### AI overview This hands-on article revisits HPX, a runtime system for asynchronous, parallel, and distributed computing. It covers installation on macOS, configuration, compilation, and testing, and discusses HPX features including task-based parallelism and components. ### Source excerpt This article explores the powerful parallelization and distributed computing capabilities of the HPX framework. Starting with installation on macOS, it walks through configuring, compiling, and testing HPX on a modest machine, highlighting key features like asynchronous programming, task-based parallelism, and HPX components. Whether new to HPX or looking for practical examples, this guide offers a hands-on approach to understanding and utilizing HPX in real-world scenarios. [Next page]()