# Parallel Compression Revamp: Dramatically Reduced CPU Overhead

DevFeed: [Parallel Compression Revamp: Dramatically Reduced CPU Overhead](<https://devfeed.tech/articles/parallel-compression-revamp-dramatically-reduced-cpu-overhead-22395.md>)

Original publisher: [Read original article](<http://rocksdb.org/blog/2025/10/08/parallel-compression-revamp.html>)

Author: Peter Dillinger

Published: 2025-10-08T00:00:00Z

Content type: release

Language: en

Sources: [RocksDB](<https://devfeed.tech/sources/rocksdb.md>)

Topics: [Compression](<https://devfeed.tech/topics/compression.md>), [rocksdb](<https://devfeed.tech/topics/rocksdb.md>), [Concurrency](<https://devfeed.tech/topics/concurrency.md>), [Concurrent Programming](<https://devfeed.tech/topics/concurrent-programming.md>), [Pull Request](<https://devfeed.tech/topics/pull-request.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [blog](<https://devfeed.tech/tags/blog.md>), [compression](<https://devfeed.tech/tags/compression.md>), [lock-free](<https://devfeed.tech/tags/lock-free.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [pull-request](<https://devfeed.tech/tags/pull-request.md>), [rocksdb](<https://devfeed.tech/tags/rocksdb.md>), [storage](<https://devfeed.tech/tags/storage.md>), [synchronization](<https://devfeed.tech/tags/synchronization.md>)

## AI overview

RocksDB 10.7 is expected to include a reimplementation of parallel compression that reduces CPU overhead by up to 65% while maintaining or improving throughput for compression-heavy workloads. The redesign uses a ring buffer, work-stealing-style thread participation, automatic thread scaling, and primarily atomic, lock-free synchronization.

## Source excerpt

The upcoming RocksDB 10.7 release includes a major revamp of parallel compression that dramatically reduces the feature's CPU overhead by up to 65% while maintaining or improving throughput for compression-heavy workloads. We expect this to broaden the set of workloads that could benefit from parallel compression, especially for bulk SST generation and remote compaction use cases that are less sensitive to CPU responsiveness. Background Parallel compression in RocksDB (CompressionOptions::parallel_threads > 1) allows multiple threads to compress different blocks simultaneously during SST file generation, which can significantly improve compaction throughput for workloads where compression is a bottleneck. However, the original implementation had substantial CPU overhead that often outweighed the benefits, limiting its practical adoption. What's New: A Complete Reimplementation The parallel compression framework has been completely rewritten from the ground up in pull request #13910 to address the core inefficiencies: Ring Buffer Architecture Instead of separate compression and write queues with complex thread coordination, the new implementation uses a ring buffer of blocks-in-progress that enables efficient work distribution across threads. This bounds working memory while enabling high throughput with minimal cross-thread synchronization. Work-Stealing Design Previously, the calling thread could only generate uncompressed blocks, dedicated compression threads could only compress, and a writer thread could only write the SST file to storage. Now, all threads can participate in compression work in a quasi-work-stealing manner, dramatically reducing the need for threads to block waiting for work. While only one thread (the calling thread or "emit thread") can generate uncompressed SST blocks in the new implementation, feeding compression work to other threads and itself, all other threads are compatible with writing compressed blocks to storage. Auto-Scaling Thread M