# RocksDB RocksDB is an embeddable persistent key-value store for fast storage. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Native Async/Coroutine Reads in RocksDB DevFeed: [Native Async/Coroutine Reads in RocksDB]() Original publisher: [Read original article]() Author: Josh Kang Published: 2026-08-24T00:00:00Z Content type: release Language: en Sources: [RocksDB]() Topics: [rocksdb](), [Coroutines](), [C++](), [IO](), [API]() Tags: [async](), [asynchronous](), [blog](), [c-plus-plus](), [coroutine](), [io](), [native](), [rocksdb](), [thread]() ### AI overview RocksDB introduces experimental asynchronous Get and MultiGet APIs backed by native C++ coroutines. The APIs can suspend storage-bound reads, allowing a small executor to run other ready tasks and maintain more storage queue depth without one blocked application thread per read. The feature targets throughput for I/O-bound point lookups rather than reducing individual device-read latency. ### Source excerpt A point lookup that misses RocksDB's block cache can spend most of its time waiting for storage. The traditional way to keep more reads in flight is to add threads. That works, but each outstanding read parks a thread, carries a stack, and adds context-switching overhead. RocksDB now has experimental asynchronous Get and MultiGet APIs backed by native C++ coroutines. When a read reaches storage, RocksDB can suspend the request, let its read-executor worker run another ready task, and resume the request when the filesystem reports completion. A small executor can therefore maintain more storage queue depth without requiring one blocked application thread per read. These APIs are available in RocksDB 11.10.0. This is primarily a throughput feature for I/O-bound point lookups. It does not make an individual device read faster. Its benefit comes from keeping the device busy and using CPU threads for runnable work. The API surface RocksDB exposes the new read path through two public interfaces: DB::GetAsync and DB::MultiGetAsync return immediately on the native path and report completion through AsyncCallback::OnComplete. CoroDB::CoGet and CoroDB::CoMultiGet return lazy folly::coro::Task objects. CoGet produces a Status; CoMultiGet fills the same per-key values and statuses as synchronous MultiGet. The callback APIs suit applications that do not expose Folly tasks at their boundaries. The CoroDB APIs let coroutine-based callers await RocksDB directly, avoiding an application-side callback-to-Baton adapter and its extra completion handoff. Native execution requires RocksDB to be built with Folly and USE_COROUTINES=1. Neither interface requires ReadOptions::async_io. That flag continues to control the older internal async-I/O optimizations for synchronous MultiGet and iterators. The task APIs are lazy: no read begins until a task is awaited or started. Both interfaces take pointer and reference parameters, so keep the DB, column-family handles, ReadOptions, keys and their ## Range Tombstone Conversion: Faster Scans Over Long Runs of Deletes DevFeed: [Range Tombstone Conversion: Faster Scans Over Long Runs of Deletes]() Original publisher: [Read original article]() Author: Josh Kang Published: 2026-06-22T00:00:00Z Content type: article Language: en Sources: [RocksDB]() Topics: [Optimization](), [rocksdb]() Tags: [blog](), [conversion](), [optimization](), [performance](), [range](), [rocksdb]() ### AI overview This article explains a RocksDB optimization that converts contiguous point tombstones into a range tombstone during scans. The approach allows scans to skip a run of deleted entries in one step instead of processing each tombstone individually. ### Source excerpt RocksDB has historically been known for poor performance when tombstones accumulate. This has become a common problem within Meta, and the community has raised it as well. Here, we introduce an optimization that attempts to convert contiguous tombstones into a range tombstone during scans. As a result, instead of skipping through N tombstones, we only need to skip through a single range tombstone. Background: point tombstones and range tombstones RocksDB is an LSM-tree, so a delete does not erase data in place. It writes a tombstone: a marker that shadows older values. A point tombstone (from Delete or SingleDelete) shadows exactly one key, while a range tombstone (from DeleteRange) shadows an entire half-open key range [start, end) with a single entry. Because newer data (usually) sits above older data in the tree, a read merges from the top down and takes the first entry it finds for a key, so a tombstone at an upper level hides any value for that key, or for any key in a range tombstone's span, at the levels below. Point and range tombstones hide the values below them. The scan steps over each point tombstone but skips the range tombstone in one hop, and only the live keys (a, e, j) are returned to the user. In both cases the space is reclaimed only later, during compaction, and only once the tombstone reaches the bottommost level with no live snapshot still needing it. Until then the tombstones sit in the way of reads. A scan never returns a deleted key, but to work out which keys are live it still has to step through every entry in key order. A point tombstone is just an ordinary entry, so the scan walks each one individually, and a run of N point tombstones costs N steps. A range tombstone is different: it is a single entry that covers the whole span, so when a scan reaches it, it can skip straight to the end of the range in one step instead of walking every key inside. Existing solutions A bulk delete leaves a region of the key space full of tombstones, and u ## FIFO KV-Ratio Compaction for BlobDB-Backed TTL Workloads DevFeed: [FIFO KV-Ratio Compaction for BlobDB-Backed TTL Workloads]() Original publisher: [Read original article]() Author: Xingbo Wang Published: 2026-06-20T00:00:00Z Content type: article Language: en Sources: [RocksDB]() Topics: [rocksdb](), [benchmarking](), [Pull Request]() Tags: [blog](), [cost](), [files](), [pull-request](), [rocksdb](), [time]() ### AI overview This article explains FIFO KV-ratio compaction in RocksDB 11.0 for BlobDB-backed workloads with large values, point lookups, and TTL or bounded-size expiration. The new picker uses the ratio of SST bytes to blob bytes to select a target SST size and reduce the read overhead caused by many small L0 files. ### Source excerpt RocksDB 11.0 added CompactionOptionsFIFO::max_data_files_size and CompactionOptionsFIFO::use_kv_ratio_compaction for a specific but important shape of workload: FIFO compaction, integrated BlobDB, large values, point lookups, and data that naturally expires by TTL or by a bounded data-size budget. The implementation was added in pull request #14326. The goal is to keep FIFO's low write amplification while reducing the read overhead caused by many small L0 files. The new picker uses the observed ratio between SST bytes and blob bytes to choose a stable target SST size, then moves L0 files through size tiers until they reach that target. Background: FIFO and BlobDB FIFO compaction is designed for time-ordered or log-like data. All files remain in L0. When files become old enough for ttl, or when the configured size limit is exceeded, RocksDB drops the oldest files instead of rewriting them into lower levels. That is what keeps FIFO write amplification low. Integrated BlobDB changes the file-size picture. Large values are stored in blob files, while SST files mostly contain keys, metadata, filters, indexes, and blob references. For point lookup workloads with large values, this can be a good fit: the SST portion can stay small and cached, and the read can fetch the large value from the blob file. However, FIFO without intra-L0 compaction can accumulate many small L0 SST files. A point lookup may then need to probe many L0 files and many filters before finding the key. FIFO's optional intra-L0 compaction, enabled with CompactionOptionsFIFO::allow_compaction, addresses that by merging several small L0 SST files into fewer larger SST files. Intra-L0 compaction rewrites SST metadata only; it does not rewrite blob files. Why the old intra-L0 picker is not enough The existing FIFO intra-L0 picker is cost based. It tries to reduce L0 file count while limiting how many bytes are rewritten for each file removed: 1 compact_bytes_per_del_file = total_input_bytes / (num_input_file ## Blob Direct Write With Partitioned Blob Files DevFeed: [Blob Direct Write With Partitioned Blob Files]() Original publisher: [Read original article]() Author: Xingbo Wang Published: 2026-06-20T00:00:00Z Content type: article Language: en Sources: [RocksDB]() Topics: [rocksdb](), [Compression]() Tags: [blog](), [compression](), [rocksdb]() ### AI overview This RocksDB article explains Blob Direct Write, which externalizes qualifying large values to blob files earlier in the write path while storing compact BlobIndex references in the WAL and memtable. It also describes partitioning support that lets applications select blob-file destinations, including grouping values with similar TTLs. ### Source excerpt TL;DR Blob Direct Write moves large-value separation earlier in RocksDB's write path. When enable_blob_files and enable_blob_direct_write are enabled, values at or above min_blob_size can be written directly to blob files during a write, while the WAL and memtable store a compact BlobIndex reference instead of the full value. The companion partitioning support makes this more than a write-path optimization. A column family can have multiple direct-write blob partitions, and applications can provide a BlobFilePartitionStrategy to choose where each large value goes. That turns blob files into a policy-controlled grouping unit. For example, an application can route values with similar TTLs into the same set of blob files while using Universal Compaction for the key and metadata part of the LSM. The reduced-scope v1 implementation landed in pull request #14535, and custom partition selection was added in pull request #14565. Background Integrated BlobDB already separates large values from the LSM tree. The LSM stores keys plus blob references, and blob files store the large value bytes. This reduces compaction write amplification because compaction can rewrite keys and references without repeatedly copying large values. Before Blob Direct Write, however, large values still entered RocksDB through the normal write path first. They were serialized into a write batch, written to the WAL, inserted into the memtable, and later extracted into blob files during flush or compaction. That design is simple and broadly compatible, but it means large values still consume WAL bandwidth and memtable memory before they become out-of-line blobs. Blob Direct Write changes that placement point. The write path can externalize a large value immediately, then publish a BlobIndex through the normal WAL and memtable machinery. Write Path The core write-path logic lives in BlobWriteBatchTransformer and BlobFilePartitionManager. For a regular Put inside a WriteBatch, the transformer does the fo ## Resumable Remote Compaction DevFeed: [Resumable Remote Compaction]() Original publisher: [Read original article]() Author: Hui Xiao Published: 2026-05-19T00:00:00Z Content type: article Language: en Sources: [RocksDB]() Topics: [rocksdb](), [API](), [IO]() Tags: [api](), [architecture](), [blog](), [cpu](), [encoding](), [files](), [implementation](), [rocksdb](), [scale](), [state](), [stateless]() ### AI overview RocksDB's resumable remote compaction adds checkpointing so interrupted compaction jobs can continue from their latest completed output SST instead of restarting from scratch. The article explains checkpoint contents, delta encoding, safety constraints, resume behavior, and configuration requirements. ### Source excerpt Background RocksDB can offload compaction work to remote workers through the CompactionService API. In this model, the primary RocksDB instance selects the input files and sends a serialized CompactionServiceInput to a worker; the remote worker runs DB::OpenAndCompact(), writes output SSTs to output_directory, and returns a serialized CompactionServiceResult that the primary RocksDB instance installs into its LSM tree. See the Remote Compaction wiki for the full architecture. This lets operators scale compaction throughput with stateless workers while keeping the primary RocksDB instance's CPU and I/O available for serving reads and writes. However, remote compaction jobs can be long-running--sometimes processing hundreds of gigabytes of input. When a worker crashes, gets preempted, or times out, the entire compaction must restart from scratch, wasting all output produced before the interruption and increasing compaction debt on the primary RocksDB instance. How Resumable Remote Compaction Works Resumable remote compaction introduces a checkpoint-and-resume mechanism. During a compaction, the worker periodically saves its progress to the output_directory. If the compaction is interrupted, a subsequent call to OpenAndCompact() with the same output directory can pick up from the last checkpoint rather than starting over. Checkpointing After each output SST file is completed, the worker persists a progress checkpoint to a compaction progress file in the output directory output_directory. The checkpoint records which internal key to resume from and the metadata of all completed output files. Progress records use delta encoding--each record only contains files completed since the last checkpoint--to keep serialization cost linear. The worker skips checkpointing at boundaries where resuming could be unsafe or requires complicated handling: when range deletions span the file boundary or when adjacent output files share the same user key. These constraints ensure that resuming ## Interpolation search for SST index blocks DevFeed: [Interpolation search for SST index blocks]() Original publisher: [Read original article]() Author: Josh Kang Published: 2026-05-04T00:00:00Z Content type: article Language: en Sources: [RocksDB]() Topics: [rocksdb](), [Algorithm]() Tags: [algorithm](), [blog](), [rocksdb]() ### AI overview RocksDB adds interpolation search for SST index blocks as an alternative to binary search, targeting fewer probes for uniformly distributed keys. The article explains key conversion, fallback behavior, configuration, and automatic per-block selection based on a uniformity hint. ### Source excerpt For workloads with uniformly distributed keys, RocksDB now supports interpolation search for SST index blocks as an alternative to the default binary search. The idea Binary search always splits the remaining range in half: 1 mid = low + (high - low) / 2 That's Θ(log n) probes regardless of the data. Interpolation search instead estimates where the target should land based on its value relative to the current boundaries: 1 probe = low + (target - key[low]) * (high - low) / (key[high] - key[low]) On uniformly distributed keys, that's expected O(log log n) probes. The canonical example: for an index block with restart keys 0, 1, 2, ..., 1023 and a seek for 900, binary search needs about 10 hops; interpolation search lands on it in 1. The catch is that pure interpolation search degrades to O(n) on badly skewed data. Turning a key into a number The interpolation formula needs numeric values, but index keys are variable-length byte slices. RocksDB extracts a uint64_t per key by reading the first 8 bytes after the common prefix shared by the block's boundary keys, in big-endian, and zero-pads to the right if the remaining bytes are too short. 1 2 3 4 5 6 7 8 9 inline uint64_t ReadBe64FromKey(Slice s, bool is_user_key, size_t offset) { // ... strip internal seq/type bytes if needed ... if (s.size() - offset >= 8) { uint64_t val; memcpy(&val, s.data() + offset, sizeof(val)); return port::kLittleEndian ? EndianSwapValue(val) : val; } // pad short tails with zeros on the right (preserves bytewise order) } Big-endian + zero-pad preserves bytewise ordering, so the linear interpolation formula stays consistent with the comparator. This is also why the feature requires BytewiseComparator. Two distinct keys can still collapse to the same uint64_t once you go past the first 8 non-shared bytes. To avoid a divide-by-zero, we simply fall back to binary search in that case. How to enable it To force interpolation search on every index block: 1 2 3 rocksdb::BlockBasedTableOptions table_ ## RocksDB development finds a CPU bug DevFeed: [RocksDB development finds a CPU bug]() Original publisher: [Read original article]() Author: Peter Dillinger Published: 2026-02-17T00:00:00Z Content type: article Language: en Sources: [RocksDB]() Topics: [rocksdb](), [bug](), [cpu](), [Development](), [Caching](), [Filesystems]() Tags: [blog](), [bug](), [caching](), [cpu](), [cross-platform](), [development](), [filesystem](), [rocksdb]() ### AI overview A RocksDB unit test for file identifiers reportedly revealed a novel hardware bug in a newer CPU. The issue was serious enough to receive a high-severity CVE. The article also explains RocksDB's use of persisted random or quasi-random identifiers to support caching across filesystems. ### Source excerpt This is the story of how a RocksDB unit test I added four years ago, a mini-stress test you might call it, revealed a novel hardware bug in a newer CPU. It was scary enough to be assigned a "high severity" CVE. Background: Unique Identifiers About four years ago, we added unique identifiers to SST files to give them stable identifiers across different filesystems for caching purposes. Part of the motivation here was to eliminate our dependence on the uniqueness and non-recycling of unique identifiers on files provided by the OS filesystem. (Some filesystems were only guaranteeing uniqueness among existing files, not among all files even in recent history.) I would call this dependency problem the great tension between reusing existing solutions and code self-reliance. You don't want to duplicate others' work but you also don't want to be subject to their bugs or changing / misaligned requirements. Striking this balance can be tricky, but in this case it was clear to us that we didn't want to rely on all the possible filesystems providing quality unique identifiers. If you're comfortable with large random numbers (e.g. 128 bits), you probably agree that persisting random identifiers (or quasi-random, which I helped formalize in a paper, also on arXiv) with each file would be safer and more predictable than relying so crucially on a minor feature of OS filesystems. High Quality Randomness However, that assumes we have access to high quality random numbers (at least a good one or two to start from - see the paper). Because RocksDB intends to be cross-platform, we want to minimize platform-specific dependencies and prefer cross-platform dependencies. But that could easily land us back where we didn't want to be: susceptible to a bug or hiccup in one implementation of what we needed. Fortunately, the nature of random entropy allows combining sources so that your result is as good as your best input source, so even if one is bad, you only have a problem if they're all bad ## BitFields API: Type-Safe Bit Packing for Lock-Free Data Structures DevFeed: [BitFields API: Type-Safe Bit Packing for Lock-Free Data Structures]() Original publisher: [Read original article]() Author: Peter Dillinger Published: 2025-12-31T00:00:00Z Content type: tutorial Language: en Sources: [RocksDB]() Topics: [rocksdb](), [C++](), [Data structures](), [Cache](), [Code]() Tags: [atomic](), [blog](), [c-plus-plus](), [cache](), [concurrent](), [efficiency](), [implementation](), [lock-free]() ### AI overview This article introduces RocksDB's BitFields API, a type-safe, zero-overhead C++ abstraction for packing multiple logical fields into atomic variables. It explains how the API helps manage packed state and describes its use in the essentially lock-free HyperClockCache. ### Source excerpt Modern concurrent data structures increasingly rely on atomic operations to avoid the overhead of locking. A valuable but under-utilized technique for maximizing the effectiveness of atomic operations is bit packing--fitting multiple logical fields into a single atomic variable for algorithmic simplicity and efficiency. However, language support for bit packing does not guarantee dense packing, and manually managing bit manipulation quickly becomes error-prone, especially when dealing with complex state machines. To address this in RocksDB, we have developed a reusable BitFields API, a type-safe, zero-overhead abstraction for bit packing in C++. This works in conjunction with clean wrappers for std::atomic for powerful and relatively safe bit-packing of atomic data. For broader use, a variant of the code has been proposed for adding to folly. The Problem: Managing Packed Bit Fields Consider HyperClockCache, an essentially lock-free cache implementation in RocksDB, which was refactored to use this BitFields API. It is a hash table built on slots that can each hold a cache entry and relevant metadata. For atomic simplicity and efficiency, all the essential metadata for each slot is packed into a single 64-bit value: The reference count and eviction metadata are together encoded into acquire and release counters, 30 bits each. The possible states of {empty, under construction/destruction, occupied+visible, and occupied+invisible} are encoded into three state bits (instead of two, for easier decoding and manipulation). A hit bit is used for secondary cache integration. Traditionally, you might write code like this: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 // Old approach: manual bit manipulation constexpr uint64_t kAcquireCounterShift = 0; constexpr uint64_t kReleaseCounterShift = 30; constexpr uint64_t kCounterMask = 0x3FFFFFFF; constexpr uint64_t kHitBitShift = 60; constexpr uint64_t k ## Parallel Compression Revamp: Dramatically Reduced CPU Overhead DevFeed: [Parallel Compression Revamp: Dramatically Reduced CPU Overhead]() Original publisher: [Read original article]() Author: Peter Dillinger Published: 2025-10-08T00:00:00Z Content type: release Language: en Sources: [RocksDB]() Topics: [Compression](), [rocksdb](), [Concurrency](), [Concurrent Programming](), [Pull Request]() Tags: [architecture](), [blog](), [compression](), [lock-free](), [parallel](), [pull-request](), [rocksdb](), [storage](), [synchronization]() ### AI overview RocksDB 10.7 is expected to include a reimplementation of parallel compression that reduces CPU overhead by up to 65% while maintaining or improving throughput for compression-heavy workloads. The redesign uses a ring buffer, work-stealing-style thread participation, automatic thread scaling, and primarily atomic, lock-free synchronization. ### Source excerpt The upcoming RocksDB 10.7 release includes a major revamp of parallel compression that dramatically reduces the feature's CPU overhead by up to 65% while maintaining or improving throughput for compression-heavy workloads. We expect this to broaden the set of workloads that could benefit from parallel compression, especially for bulk SST generation and remote compaction use cases that are less sensitive to CPU responsiveness. Background Parallel compression in RocksDB (CompressionOptions::parallel_threads > 1) allows multiple threads to compress different blocks simultaneously during SST file generation, which can significantly improve compaction throughput for workloads where compression is a bottleneck. However, the original implementation had substantial CPU overhead that often outweighed the benefits, limiting its practical adoption. What's New: A Complete Reimplementation The parallel compression framework has been completely rewritten from the ground up in pull request #13910 to address the core inefficiencies: Ring Buffer Architecture Instead of separate compression and write queues with complex thread coordination, the new implementation uses a ring buffer of blocks-in-progress that enables efficient work distribution across threads. This bounds working memory while enabling high throughput with minimal cross-thread synchronization. Work-Stealing Design Previously, the calling thread could only generate uncompressed blocks, dedicated compression threads could only compress, and a writer thread could only write the SST file to storage. Now, all threads can participate in compression work in a quasi-work-stealing manner, dramatically reducing the need for threads to block waiting for work. While only one thread (the calling thread or "emit thread") can generate uncompressed SST blocks in the new implementation, feeding compression work to other threads and itself, all other threads are compatible with writing compressed blocks to storage. Auto-Scaling Thread M ## IO Activity Tagging DevFeed: [IO Activity Tagging]() Original publisher: [Read original article]() Author: Hui Xiao Published: 2025-09-25T00:00:00Z Content type: article Language: en Sources: [RocksDB]() Topics: [rocksdb](), [IO](), [systems]() Tags: [blog](), [caching](), [enum-class](), [io](), [management](), [operations](), [performance](), [rocksdb](), [structure](), [systems](), [verification]() ### AI overview This article explains RocksDB's IOActivity enum, which automatically tags operations such as reads, flushes, compactions, database opens, and verification. The tags are propagated through the storage stack so custom file systems can make activity-aware scheduling, caching, and resource-management decisions. RocksDB also provides per-activity IO time and count histograms. ### Source excerpt Context RocksDB performs a variety of IO operations--user reads, background compactions, flushes, database opens, and verification tasks. Treating all these operations the same makes it difficult for file system implementers to optimize performance, prioritize latency-sensitive IOs, and diagnose bottlenecks. To solve that, RocksDB internally tags every IO operation with its activity type using the IOActivity enum. This automatic tagging provides precise context for each IO, enabling file systems to make smarter, context-aware decisions for scheduling, caching, and resource management. How Internal IO Tagging Works RocksDB automatically assigns an IOActivity tag to each IO operation. This tag is propagated through the storage stack and included in the IO options passed to the file system. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 enum class IOActivity : uint8_t { kFlush = 0, // IO for flush operations (background write) kCompaction = 1, // IO for compaction (background read/write) kDBOpen = 2, // IO during database open (read/write) kGet = 3, // User Get() read kMultiGet = 4, // User MultiGet() read kDBIterator = 5, // User iterator read kVerifyDBChecksum = 6, // Verification: DB checksum kVerifyFileChecksums = 7, // Verification: file checksums kGetEntity = 8, // Entity Get (e.g., wide-column) kMultiGetEntity = 9, // Entity MultiGet kGetFileChecksumsFromCurrentManifest = 10, // Manifest checksum reads // 0x80-0xFE: Reserved for custom/internal use kUnknown = 0xFF // Unknown/unspecified activity }; Access IO Tag in File System Custom file systems can access the IOActivity tag via the IO options structure provided by RocksDB. This allows them to optimize behavior based on the specific IO activity. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Status CustomFileSystem::Append(uint64_t offset, const Slice& data, const IOOptions& io_opts, ...) { switch (io_opts.io_activity) { case Env::IOActivity::kGet: // Prioritize or cache user reads break; case Env::IOActivity::kCompaction: // T