# rocksdb Published articles for rocksdb. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Perplexity's AI agents helped build a database. They weren't allowed to run it. DevFeed: [Perplexity's AI agents helped build a database. They weren't allowed to run it.]() Original publisher: [Read original article]() Author: Amanda Caswell Published: 2026-09-16T21:51:15Z Content type: article Language: en Sources: [The New Stack]() Topics: [Database](), [DynamoDB](), [Rust](), [Artificial Intelligence](), [rocksdb](), [Amazon S3]() Tags: [agents](), [ai](), [ai-agents](), [ai-infrastructure](), [api](), [coding](), [coding-agents](), [database](), [databases](), [dynamodb](), [open-source](), [performance](), [perplexity](), [rocksdb](), [rust](), [s3]() ### AI overview Perplexity built CobbleDB, a Rust key-value store, after finding DynamoDB too costly and insufficiently controllable for its search workload. Coding agents helped develop it, but were not allowed to run it in production. Perplexity measured lower read latency and expects lower costs, with plans to open-source the database. ### Source excerpt Perplexity decided it was paying too much for DynamoDB and wasn't getting the control it wanted over read performance. So The post Perplexity's AI agents helped build a database. They weren't allowed to run it. appeared first on The New Stack. ## Native Async/Coroutine Reads in RocksDB DevFeed: [Native Async/Coroutine Reads in RocksDB]() Original publisher: [Read original article]() Author: Josh Kang Published: 2026-08-24T00:00:00Z Content type: release Language: en Sources: [RocksDB]() Topics: [rocksdb](), [Coroutines](), [C++](), [IO](), [API]() Tags: [async](), [asynchronous](), [blog](), [c-plus-plus](), [coroutine](), [io](), [native](), [rocksdb](), [thread]() ### AI overview RocksDB introduces experimental asynchronous Get and MultiGet APIs backed by native C++ coroutines. The APIs can suspend storage-bound reads, allowing a small executor to run other ready tasks and maintain more storage queue depth without one blocked application thread per read. The feature targets throughput for I/O-bound point lookups rather than reducing individual device-read latency. ### Source excerpt A point lookup that misses RocksDB's block cache can spend most of its time waiting for storage. The traditional way to keep more reads in flight is to add threads. That works, but each outstanding read parks a thread, carries a stack, and adds context-switching overhead. RocksDB now has experimental asynchronous Get and MultiGet APIs backed by native C++ coroutines. When a read reaches storage, RocksDB can suspend the request, let its read-executor worker run another ready task, and resume the request when the filesystem reports completion. A small executor can therefore maintain more storage queue depth without requiring one blocked application thread per read. These APIs are available in RocksDB 11.10.0. This is primarily a throughput feature for I/O-bound point lookups. It does not make an individual device read faster. Its benefit comes from keeping the device busy and using CPU threads for runnable work. The API surface RocksDB exposes the new read path through two public interfaces: DB::GetAsync and DB::MultiGetAsync return immediately on the native path and report completion through AsyncCallback::OnComplete. CoroDB::CoGet and CoroDB::CoMultiGet return lazy folly::coro::Task objects. CoGet produces a Status; CoMultiGet fills the same per-key values and statuses as synchronous MultiGet. The callback APIs suit applications that do not expose Folly tasks at their boundaries. The CoroDB APIs let coroutine-based callers await RocksDB directly, avoiding an application-side callback-to-Baton adapter and its extra completion handoff. Native execution requires RocksDB to be built with Folly and USE_COROUTINES=1. Neither interface requires ReadOptions::async_io. That flag continues to control the older internal async-I/O optimizations for synchronous MultiGet and iterators. The task APIs are lazy: no read begins until a task is awaited or started. Both interfaces take pointer and reference parameters, so keep the DB, column-family handles, ReadOptions, keys and their ## How LSM compaction causes write stalls and latency spikes DevFeed: [How LSM compaction causes write stalls and latency spikes]() Original publisher: [Read original article]() Author: hello@ankit-rana.com Published: 2026-08-24T00:00:00Z Content type: article Language: en Sources: [Ankit Rana | Mechanical Sympathy]() Topics: [Latency](), [rocksdb](), [Apache Cassandra](), [scylladb](), [data]() Tags: [capacity-planning](), [cassandra](), [compaction](), [durability](), [latency](), [lsm-tree](), [outage](), [rocksdb](), [scylladb](), [storage-engine](), [write](), [write-amplification]() ### AI overview LSM trees make writes fast by deferring work, but background compaction repeatedly rewrites stored data. When compaction falls behind, pending files increase, reads become slower, and the engine may throttle or stop writes to prevent unbounded read amplification. The article recommends watching pending compaction bytes and maintaining disk headroom for simultaneous compaction input and output. ### Source excerpt An LSM tree makes writes fast by never updating in place, which means every write is eventually rewritten several times by background compaction. When compaction falls behind, the engine deliberately throttles or stops incoming writes to avoid unbounded read amplification, so a background job becomes a foreground outage. The signal to watch is pending compaction bytes rather than disk utilisation, and the disk needs headroom for the largest compaction because merging requires space for input and output at the same time. ## RocksDB Performance and Zero-Copy DevFeed: [RocksDB Performance and Zero-Copy]() Original publisher: [Read original article]() Author: Davide Angelocola Published: 2026-08-08T00:00:00Z Content type: article Language: en Sources: [Davide Angelocola]() Topics: [rocksdb](), [C++](), [Java](), [Benchmark](), [Cache]() Tags: [benchmark](), [c-plus-plus](), [java](), [library](), [memory](), [performance](), [rocksdb](), [zero-copy]() ### AI overview This article examines zero-copy reads for an FFM-based RocksDB binding between C++ and Java. It describes callback-scoped memory management, pinned data, resource lifetimes, and benchmark results showing that zero-copy is not always faster for small values. ### Source excerpt rocksdbffm, the FFM-based RocksDB binding I wrote about last time. Here's what it took to get reads down to zero allocations -- and what the benchmarks had to say about it. ## 【FoundationDB 内核】Unbundled - OCC - TLog - Redwood - Simulation DevFeed: [【FoundationDB 内核】Unbundled - OCC - TLog - Redwood - Simulation]() Original publisher: [Read original article]() Author: Liao Tonglang Published: 2026-07-17T00:00:00Z Content type: article Language: zh Sources: [土法炼钢 - 系统与基础设施]() Topics: [transaction-log](), [rocksdb](), [SQLite](), [version](), [Disk image]() Tags: [database](), [deterministic-simulation](), [distributed](), [foundationdb](), [occ](), [redwood](), [resolver](), [rocksdb](), [sequencer](), [simulation](), [sqlite](), [storage](), [strict-serializability](), [transaction-log](), [unbundled](), [version]() ### AI overview A Chinese overview of an 18-part FoundationDB 7.x series. It explains the unbundled transaction path through Proxy, Sequencer, Resolver, TLog, and Storage Server, covering strict serializability, the five-second transaction limit, recovery, deterministic simulation, Redwood, Record Layer, operations, troubleshooting, and comparisons with TiKV and etcd. ### Source excerpt 补齐 FoundationDB 选型叙事与生产内核之间的一层：拆解 Proxy、Sequencer、Resolver、TLog、Storage Server 的 Unbundled 写路径，严格可串行化、Redwood 与确定性模拟，并以 Record Layer 和 TiKV/etcd 对照收束。 ## 【TiKV / HTAP 内核】Region - Multi-Raft - PD - Percolator - TiFlash DevFeed: [【TiKV / HTAP 内核】Region - Multi-Raft - PD - Percolator - TiFlash]() Original publisher: [Read original article]() Author: Liao Tonglang Published: 2026-07-16T00:00:00Z Content type: article Language: zh Sources: [土法炼钢 - 系统与基础设施]() Topics: [Raft](), [rocksdb](), [etcd](), [SQL](), [CockroachDB](), [Databases]() Tags: [database](), [distributed](), [distributed-kv](), [etcd](), [htap](), [learner](), [multi-raft](), [pd](), [percolator](), [raft](), [range](), [region](), [rocksdb](), [snapshot](), [split](), [sql](), [storage](), [tidb](), [tiflash](), [tikv](), [tso]() ### AI overview This article series explains TiKV 7.x/8.x internals through Region modeling, Multi-Raft replication, raftstore apply, RocksDB column families, PD and TSO scheduling, Percolator transactions, coprocessor boundaries, and TiFlash Learner freshness. It also compares selected aspects with etcd and CockroachDB. ### Source excerpt 补齐 RocksDB 单 Region、Raft 协议与 HTAP 范式之间的工程链路：以 TiKV 7.x/8.x 拆解 Region、Multi-Raft、raftstore、PD/TSO 与 Percolator 事务，并以 TiFlash Learner 收束新鲜度；CockroachDB 对照与选型。 ## Range Tombstone Conversion: Faster Scans Over Long Runs of Deletes DevFeed: [Range Tombstone Conversion: Faster Scans Over Long Runs of Deletes]() Original publisher: [Read original article]() Author: Josh Kang Published: 2026-06-22T00:00:00Z Content type: article Language: en Sources: [RocksDB]() Topics: [Optimization](), [rocksdb]() Tags: [blog](), [conversion](), [optimization](), [performance](), [range](), [rocksdb]() ### AI overview This article explains a RocksDB optimization that converts contiguous point tombstones into a range tombstone during scans. The approach allows scans to skip a run of deleted entries in one step instead of processing each tombstone individually. ### Source excerpt RocksDB has historically been known for poor performance when tombstones accumulate. This has become a common problem within Meta, and the community has raised it as well. Here, we introduce an optimization that attempts to convert contiguous tombstones into a range tombstone during scans. As a result, instead of skipping through N tombstones, we only need to skip through a single range tombstone. Background: point tombstones and range tombstones RocksDB is an LSM-tree, so a delete does not erase data in place. It writes a tombstone: a marker that shadows older values. A point tombstone (from Delete or SingleDelete) shadows exactly one key, while a range tombstone (from DeleteRange) shadows an entire half-open key range [start, end) with a single entry. Because newer data (usually) sits above older data in the tree, a read merges from the top down and takes the first entry it finds for a key, so a tombstone at an upper level hides any value for that key, or for any key in a range tombstone's span, at the levels below. Point and range tombstones hide the values below them. The scan steps over each point tombstone but skips the range tombstone in one hop, and only the live keys (a, e, j) are returned to the user. In both cases the space is reclaimed only later, during compaction, and only once the tombstone reaches the bottommost level with no live snapshot still needing it. Until then the tombstones sit in the way of reads. A scan never returns a deleted key, but to work out which keys are live it still has to step through every entry in key order. A point tombstone is just an ordinary entry, so the scan walks each one individually, and a run of N point tombstones costs N steps. A range tombstone is different: it is a single entry that covers the whole span, so when a scan reaches it, it can skip straight to the end of the range in one step instead of walking every key inside. Existing solutions A bulk delete leaves a region of the key space full of tombstones, and u ## FIFO KV-Ratio Compaction for BlobDB-Backed TTL Workloads DevFeed: [FIFO KV-Ratio Compaction for BlobDB-Backed TTL Workloads]() Original publisher: [Read original article]() Author: Xingbo Wang Published: 2026-06-20T00:00:00Z Content type: article Language: en Sources: [RocksDB]() Topics: [rocksdb](), [benchmarking](), [Pull Request]() Tags: [blog](), [cost](), [files](), [pull-request](), [rocksdb](), [time]() ### AI overview This article explains FIFO KV-ratio compaction in RocksDB 11.0 for BlobDB-backed workloads with large values, point lookups, and TTL or bounded-size expiration. The new picker uses the ratio of SST bytes to blob bytes to select a target SST size and reduce the read overhead caused by many small L0 files. ### Source excerpt RocksDB 11.0 added CompactionOptionsFIFO::max_data_files_size and CompactionOptionsFIFO::use_kv_ratio_compaction for a specific but important shape of workload: FIFO compaction, integrated BlobDB, large values, point lookups, and data that naturally expires by TTL or by a bounded data-size budget. The implementation was added in pull request #14326. The goal is to keep FIFO's low write amplification while reducing the read overhead caused by many small L0 files. The new picker uses the observed ratio between SST bytes and blob bytes to choose a stable target SST size, then moves L0 files through size tiers until they reach that target. Background: FIFO and BlobDB FIFO compaction is designed for time-ordered or log-like data. All files remain in L0. When files become old enough for ttl, or when the configured size limit is exceeded, RocksDB drops the oldest files instead of rewriting them into lower levels. That is what keeps FIFO write amplification low. Integrated BlobDB changes the file-size picture. Large values are stored in blob files, while SST files mostly contain keys, metadata, filters, indexes, and blob references. For point lookup workloads with large values, this can be a good fit: the SST portion can stay small and cached, and the read can fetch the large value from the blob file. However, FIFO without intra-L0 compaction can accumulate many small L0 SST files. A point lookup may then need to probe many L0 files and many filters before finding the key. FIFO's optional intra-L0 compaction, enabled with CompactionOptionsFIFO::allow_compaction, addresses that by merging several small L0 SST files into fewer larger SST files. Intra-L0 compaction rewrites SST metadata only; it does not rewrite blob files. Why the old intra-L0 picker is not enough The existing FIFO intra-L0 picker is cost based. It tries to reduce L0 file count while limiting how many bytes are rewritten for each file removed: 1 compact_bytes_per_del_file = total_input_bytes / (num_input_file ## Blob Direct Write With Partitioned Blob Files DevFeed: [Blob Direct Write With Partitioned Blob Files]() Original publisher: [Read original article]() Author: Xingbo Wang Published: 2026-06-20T00:00:00Z Content type: article Language: en Sources: [RocksDB]() Topics: [rocksdb](), [Compression]() Tags: [blog](), [compression](), [rocksdb]() ### AI overview This RocksDB article explains Blob Direct Write, which externalizes qualifying large values to blob files earlier in the write path while storing compact BlobIndex references in the WAL and memtable. It also describes partitioning support that lets applications select blob-file destinations, including grouping values with similar TTLs. ### Source excerpt TL;DR Blob Direct Write moves large-value separation earlier in RocksDB's write path. When enable_blob_files and enable_blob_direct_write are enabled, values at or above min_blob_size can be written directly to blob files during a write, while the WAL and memtable store a compact BlobIndex reference instead of the full value. The companion partitioning support makes this more than a write-path optimization. A column family can have multiple direct-write blob partitions, and applications can provide a BlobFilePartitionStrategy to choose where each large value goes. That turns blob files into a policy-controlled grouping unit. For example, an application can route values with similar TTLs into the same set of blob files while using Universal Compaction for the key and metadata part of the LSM. The reduced-scope v1 implementation landed in pull request #14535, and custom partition selection was added in pull request #14565. Background Integrated BlobDB already separates large values from the LSM tree. The LSM stores keys plus blob references, and blob files store the large value bytes. This reduces compaction write amplification because compaction can rewrite keys and references without repeatedly copying large values. Before Blob Direct Write, however, large values still entered RocksDB through the normal write path first. They were serialized into a write batch, written to the WAL, inserted into the memtable, and later extracted into blob files during flush or compaction. That design is simple and broadly compatible, but it means large values still consume WAL bandwidth and memtable memory before they become out-of-line blobs. Blob Direct Write changes that placement point. The write path can externalize a large value immediately, then publish a BlobIndex through the normal WAL and memtable machinery. Write Path The core write-path logic lives in BlobWriteBatchTransformer and BlobFilePartitionManager. For a regular Put inside a WriteBatch, the transformer does the fo ## Cloud Topics: the Metastore DevFeed: [Cloud Topics: the Metastore]() Original publisher: [Read original article]() Author: Andrew Wong Published: 2026-06-11T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Kafka](), [Cloud](), [rocksdb](), [data]() Tags: [architecture](), [cloud](), [consumer](), [engineering](), [kafka](), [learn](), [mapping](), [memory](), [object-storage](), [retention](), [rocksdb](), [scale](), [serialization](), [storage]() ### AI overview This article explains Redpanda's metastore for Cloud Topics. The metastore maps Apache Kafka offsets to byte ranges in L1 objects stored in object storage, while tracking metadata such as leader-term boundaries and compaction state. Redpanda built a general-purpose key-value store, inspired by LevelDB and RocksDB and implemented as an LSM tree, to scale metadata independently of memory, local disk, and metadata formats. ### Source excerpt Learn how Redpanda's metastore powers Cloud Topics, from offset lookups and whole cluster restore to cross-region read replicas, and why it's built to be a foundational primitive for the future. ## Resumable Remote Compaction DevFeed: [Resumable Remote Compaction]() Original publisher: [Read original article]() Author: Hui Xiao Published: 2026-05-19T00:00:00Z Content type: article Language: en Sources: [RocksDB]() Topics: [rocksdb](), [API](), [IO]() Tags: [api](), [architecture](), [blog](), [cpu](), [encoding](), [files](), [implementation](), [rocksdb](), [scale](), [state](), [stateless]() ### AI overview RocksDB's resumable remote compaction adds checkpointing so interrupted compaction jobs can continue from their latest completed output SST instead of restarting from scratch. The article explains checkpoint contents, delta encoding, safety constraints, resume behavior, and configuration requirements. ### Source excerpt Background RocksDB can offload compaction work to remote workers through the CompactionService API. In this model, the primary RocksDB instance selects the input files and sends a serialized CompactionServiceInput to a worker; the remote worker runs DB::OpenAndCompact(), writes output SSTs to output_directory, and returns a serialized CompactionServiceResult that the primary RocksDB instance installs into its LSM tree. See the Remote Compaction wiki for the full architecture. This lets operators scale compaction throughput with stateless workers while keeping the primary RocksDB instance's CPU and I/O available for serving reads and writes. However, remote compaction jobs can be long-running--sometimes processing hundreds of gigabytes of input. When a worker crashes, gets preempted, or times out, the entire compaction must restart from scratch, wasting all output produced before the interruption and increasing compaction debt on the primary RocksDB instance. How Resumable Remote Compaction Works Resumable remote compaction introduces a checkpoint-and-resume mechanism. During a compaction, the worker periodically saves its progress to the output_directory. If the compaction is interrupted, a subsequent call to OpenAndCompact() with the same output directory can pick up from the last checkpoint rather than starting over. Checkpointing After each output SST file is completed, the worker persists a progress checkpoint to a compaction progress file in the output directory output_directory. The checkpoint records which internal key to resume from and the metadata of all completed output files. Progress records use delta encoding--each record only contains files completed since the last checkpoint--to keep serialization cost linear. The worker skips checkpointing at boundaries where resuming could be unsafe or requires complicated handling: when range deletions span the file boundary or when adjacent output files share the same user key. These constraints ensure that resuming ## Interpolation search for SST index blocks DevFeed: [Interpolation search for SST index blocks]() Original publisher: [Read original article]() Author: Josh Kang Published: 2026-05-04T00:00:00Z Content type: article Language: en Sources: [RocksDB]() Topics: [rocksdb](), [Algorithm]() Tags: [algorithm](), [blog](), [rocksdb]() ### AI overview RocksDB adds interpolation search for SST index blocks as an alternative to binary search, targeting fewer probes for uniformly distributed keys. The article explains key conversion, fallback behavior, configuration, and automatic per-block selection based on a uniformity hint. ### Source excerpt For workloads with uniformly distributed keys, RocksDB now supports interpolation search for SST index blocks as an alternative to the default binary search. The idea Binary search always splits the remaining range in half: 1 mid = low + (high - low) / 2 That's Θ(log n) probes regardless of the data. Interpolation search instead estimates where the target should land based on its value relative to the current boundaries: 1 probe = low + (target - key[low]) * (high - low) / (key[high] - key[low]) On uniformly distributed keys, that's expected O(log log n) probes. The canonical example: for an index block with restart keys 0, 1, 2, ..., 1023 and a seek for 900, binary search needs about 10 hops; interpolation search lands on it in 1. The catch is that pure interpolation search degrades to O(n) on badly skewed data. Turning a key into a number The interpolation formula needs numeric values, but index keys are variable-length byte slices. RocksDB extracts a uint64_t per key by reading the first 8 bytes after the common prefix shared by the block's boundary keys, in big-endian, and zero-pads to the right if the remaining bytes are too short. 1 2 3 4 5 6 7 8 9 inline uint64_t ReadBe64FromKey(Slice s, bool is_user_key, size_t offset) { // ... strip internal seq/type bytes if needed ... if (s.size() - offset >= 8) { uint64_t val; memcpy(&val, s.data() + offset, sizeof(val)); return port::kLittleEndian ? EndianSwapValue(val) : val; } // pad short tails with zeros on the right (preserves bytewise order) } Big-endian + zero-pad preserves bytewise ordering, so the linear interpolation formula stays consistent with the comparator. This is also why the feature requires BytewiseComparator. Two distinct keys can still collapse to the same uint64_t once you go past the first 8 non-shared bytes. To avoid a divide-by-zero, we simply fall back to binary search in that case. How to enable it To force interpolation search on every index block: 1 2 3 rocksdb::BlockBasedTableOptions table_ ## Building an Experimental Java RocksDB Wrapper with the Foreign Function & Memory API DevFeed: [Building an Experimental Java RocksDB Wrapper with the Foreign Function & Memory API]() Original publisher: [Read original article]() Author: Davide Angelocola Published: 2026-04-11T00:00:00Z Content type: tutorial Language: en Sources: [Davide Angelocola]() Topics: [Java](), [rocksdb](), [C](), [Code](), [Development](), [Artificial Intelligence]() Tags: [ai](), [c](), [code](), [java](), [llm](), [rocksdb]() ### AI overview This article describes rocksdbffm, an experimental Java wrapper for RocksDB that uses Java's Foreign Function & Memory API instead of JNI. It explains FFM concepts, generated low-level bindings from RocksDB's C API, and the idiomatic Java API built on top. ### Source excerpt The best way I know to learn something is to build something real with it. I wanted to understand Java's Foreign Function & Memory (FFM) API properly -- not just read the Javadoc, but actually use it. RocksDB was the obvious vehicle: I'm a long-time user of RocksDB and it has a well-defined C API and a problem space I already understood. The result is rocksdbffm: an experimental RocksDB wrapper for Java using FFM instead of JNI. ## The RUM Conjecture: You Cannot Optimize Reads, Updates, and Memory at Once DevFeed: [The RUM Conjecture: You Cannot Optimize Reads, Updates, and Memory at Once]() Original publisher: [Read original article]() Author: hello@ankit-rana.com Published: 2026-03-17T00:00:00Z Content type: article Language: en Sources: [Ankit Rana | Mechanical Sympathy]() Topics: [systems](), [Databases](), [Apache Cassandra](), [rocksdb](), [MySQL](), [PostgreSQL]() Tags: [b-tree](), [capacity](), [cassandra](), [database](), [databases](), [dram](), [indexing](), [latency](), [memory](), [mysql](), [node](), [performance](), [rocksdb](), [storage-engine](), [system-design]() ### AI overview The article explains the RUM Conjecture, which describes a tradeoff among read overhead, update overhead, and memory overhead in database indexes and storage engines. It compares B-Trees, LSM-Trees, and hash indexes to show how each optimizes different tradeoffs. ### Source excerpt You can strictly optimise at most two of read overhead, update overhead, and memory overhead; the third will be expensive. B-Trees optimise reads and memory and pay on writes. LSM-Trees optimise writes and memory and pay on reads. Hash indexes optimise reads and writes and pay in RAM. The useful question is not whether a database is good but which corner it optimises and what you are willing to pay for the other two. ## RocksDB development finds a CPU bug DevFeed: [RocksDB development finds a CPU bug]() Original publisher: [Read original article]() Author: Peter Dillinger Published: 2026-02-17T00:00:00Z Content type: article Language: en Sources: [RocksDB]() Topics: [rocksdb](), [bug](), [cpu](), [Development](), [Caching](), [Filesystems]() Tags: [blog](), [bug](), [caching](), [cpu](), [cross-platform](), [development](), [filesystem](), [rocksdb]() ### AI overview A RocksDB unit test for file identifiers reportedly revealed a novel hardware bug in a newer CPU. The issue was serious enough to receive a high-severity CVE. The article also explains RocksDB's use of persisted random or quasi-random identifiers to support caching across filesystems. ### Source excerpt This is the story of how a RocksDB unit test I added four years ago, a mini-stress test you might call it, revealed a novel hardware bug in a newer CPU. It was scary enough to be assigned a "high severity" CVE. Background: Unique Identifiers About four years ago, we added unique identifiers to SST files to give them stable identifiers across different filesystems for caching purposes. Part of the motivation here was to eliminate our dependence on the uniqueness and non-recycling of unique identifiers on files provided by the OS filesystem. (Some filesystems were only guaranteeing uniqueness among existing files, not among all files even in recent history.) I would call this dependency problem the great tension between reusing existing solutions and code self-reliance. You don't want to duplicate others' work but you also don't want to be subject to their bugs or changing / misaligned requirements. Striking this balance can be tricky, but in this case it was clear to us that we didn't want to rely on all the possible filesystems providing quality unique identifiers. If you're comfortable with large random numbers (e.g. 128 bits), you probably agree that persisting random identifiers (or quasi-random, which I helped formalize in a paper, also on arXiv) with each file would be safer and more predictable than relying so crucially on a minor feature of OS filesystems. High Quality Randomness However, that assumes we have access to high quality random numbers (at least a good one or two to start from - see the paper). Because RocksDB intends to be cross-platform, we want to minimize platform-specific dependencies and prefer cross-platform dependencies. But that could easily land us back where we didn't want to be: susceptible to a bug or hiccup in one implementation of what we needed. Fortunately, the nature of random entropy allows combining sources so that your result is as good as your best input source, so even if one is bad, you only have a problem if they're all bad ## RocksDB Compression in Ceph: Space Savings with No Performance Cost DevFeed: [RocksDB Compression in Ceph: Space Savings with No Performance Cost]() Original publisher: [Read original article]() Author: Daniel Alexander Parkes, Anthony D'Atri Published: 2025-12-17T00:00:00Z Content type: article Language: en Sources: [Ceph Blog]() Topics: [Compression](), [Database](), [Hardware](), [ibm]() Tags: [blog-post](), [ceph](), [compression](), [configuration](), [cost](), [devices](), [en-article](), [en-blog-post](), [ibm](), [mon](), [nvme](), [osd](), [performance](), [rados](), [reef](), [rocksdb](), [space](), [storage]() ### AI overview This article reports Ceph performance tests showing that enabling RocksDB compression can substantially reduce metadata database space, especially for smaller objects, without harming throughput or resource consumption. The tests used IBM Storage Ceph 7.1 with BlueStore OSDs, HDD object storage, and NVMe devices for the RocksDB WAL and database. ### Source excerpt Introduction ¶ In the world of data storage, engineers and architects constantly face a fundamental dilemma: the trade-off between performance and efficiency. It's a balancing act. When you want to save space, you typically enable features like compression, but the common assumption is that this will cost you performance, a CPU cycle tax that slows throughput. But what if you could significantly reduce your metadata storage footprint without slowing things down? This search for an answer to this question started with research work from Mark Nelson, who published a blog post on ceph.io that covers RocksDB tuning in depth, exploring RocksDB compression with positive results. These promising results sparked a conversation on the upstream GitHub about enabling compression by default; a link to the PR is available here. To build on the previous investigation, the Ceph performance team ran tests on a robust hardware configuration running IBM Storage Ceph 7.1 (Reef). The cluster used the BlueStore OSDs for an erasure-coded (EC 4+2) pool, with a hybrid OSD storage setup: HDDs for object data and fast NVMe drives for the BlueStore WAL+DB. To understand the test, it's helpful to know what the WAL+DB is. In modern Ceph, the BlueStore storage engine manages all data on the OSDs (physical devices). To do this, it must maintain a vast catalog of internal metadata: think of it as a high-speed index that quickly locates every piece of data. RocksDB, a high-performance key-value database, manages this critical index. In our hybrid cluster, the RocksDB database runs on the fast NVMe deviceses, while the actual object data resides on the slower HDDs. Because this metadata can grow very large, RocksDB's efficiency, how much space it consumes on those expensive NVMe drives, is a critical factor in the cluster's overall cost and performance. Our test, therefore, focuses on a simple, high-stakes question: Can we compress this metadata to save space without paying a performance penalty? Ex ## Parallel Compression Revamp: Dramatically Reduced CPU Overhead DevFeed: [Parallel Compression Revamp: Dramatically Reduced CPU Overhead]() Original publisher: [Read original article]() Author: Peter Dillinger Published: 2025-10-08T00:00:00Z Content type: release Language: en Sources: [RocksDB]() Topics: [Compression](), [rocksdb](), [Concurrency](), [Concurrent Programming](), [Pull Request]() Tags: [architecture](), [blog](), [compression](), [lock-free](), [parallel](), [pull-request](), [rocksdb](), [storage](), [synchronization]() ### AI overview RocksDB 10.7 is expected to include a reimplementation of parallel compression that reduces CPU overhead by up to 65% while maintaining or improving throughput for compression-heavy workloads. The redesign uses a ring buffer, work-stealing-style thread participation, automatic thread scaling, and primarily atomic, lock-free synchronization. ### Source excerpt The upcoming RocksDB 10.7 release includes a major revamp of parallel compression that dramatically reduces the feature's CPU overhead by up to 65% while maintaining or improving throughput for compression-heavy workloads. We expect this to broaden the set of workloads that could benefit from parallel compression, especially for bulk SST generation and remote compaction use cases that are less sensitive to CPU responsiveness. Background Parallel compression in RocksDB (CompressionOptions::parallel_threads > 1) allows multiple threads to compress different blocks simultaneously during SST file generation, which can significantly improve compaction throughput for workloads where compression is a bottleneck. However, the original implementation had substantial CPU overhead that often outweighed the benefits, limiting its practical adoption. What's New: A Complete Reimplementation The parallel compression framework has been completely rewritten from the ground up in pull request #13910 to address the core inefficiencies: Ring Buffer Architecture Instead of separate compression and write queues with complex thread coordination, the new implementation uses a ring buffer of blocks-in-progress that enables efficient work distribution across threads. This bounds working memory while enabling high throughput with minimal cross-thread synchronization. Work-Stealing Design Previously, the calling thread could only generate uncompressed blocks, dedicated compression threads could only compress, and a writer thread could only write the SST file to storage. Now, all threads can participate in compression work in a quasi-work-stealing manner, dramatically reducing the need for threads to block waiting for work. While only one thread (the calling thread or "emit thread") can generate uncompressed SST blocks in the new implementation, feeding compression work to other threads and itself, all other threads are compatible with writing compressed blocks to storage. Auto-Scaling Thread M ## IO Activity Tagging DevFeed: [IO Activity Tagging]() Original publisher: [Read original article]() Author: Hui Xiao Published: 2025-09-25T00:00:00Z Content type: article Language: en Sources: [RocksDB]() Topics: [rocksdb](), [IO](), [systems]() Tags: [blog](), [caching](), [enum-class](), [io](), [management](), [operations](), [performance](), [rocksdb](), [structure](), [systems](), [verification]() ### AI overview This article explains RocksDB's IOActivity enum, which automatically tags operations such as reads, flushes, compactions, database opens, and verification. The tags are propagated through the storage stack so custom file systems can make activity-aware scheduling, caching, and resource-management decisions. RocksDB also provides per-activity IO time and count histograms. ### Source excerpt Context RocksDB performs a variety of IO operations--user reads, background compactions, flushes, database opens, and verification tasks. Treating all these operations the same makes it difficult for file system implementers to optimize performance, prioritize latency-sensitive IOs, and diagnose bottlenecks. To solve that, RocksDB internally tags every IO operation with its activity type using the IOActivity enum. This automatic tagging provides precise context for each IO, enabling file systems to make smarter, context-aware decisions for scheduling, caching, and resource management. How Internal IO Tagging Works RocksDB automatically assigns an IOActivity tag to each IO operation. This tag is propagated through the storage stack and included in the IO options passed to the file system. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 enum class IOActivity : uint8_t { kFlush = 0, // IO for flush operations (background write) kCompaction = 1, // IO for compaction (background read/write) kDBOpen = 2, // IO during database open (read/write) kGet = 3, // User Get() read kMultiGet = 4, // User MultiGet() read kDBIterator = 5, // User iterator read kVerifyDBChecksum = 6, // Verification: DB checksum kVerifyFileChecksums = 7, // Verification: file checksums kGetEntity = 8, // Entity Get (e.g., wide-column) kMultiGetEntity = 9, // Entity MultiGet kGetFileChecksumsFromCurrentManifest = 10, // Manifest checksum reads // 0x80-0xFE: Reserved for custom/internal use kUnknown = 0xFF // Unknown/unspecified activity }; Access IO Tag in File System Custom file systems can access the IOActivity tag via the IO options structure provided by RocksDB. This allows them to optimize behavior based on the specific IO activity. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Status CustomFileSystem::Append(uint64_t offset, const Slice& data, const IOOptions& io_opts, ...) { switch (io_opts.io_activity) { case Env::IOActivity::kGet: // Prioritize or cache user reads break; case Env::IOActivity::kCompaction: // T ## Get Running with Apache Flink on Kubernetes, part 2 of 2 DevFeed: [Get Running with Apache Flink on Kubernetes, part 2 of 2]() Original publisher: [Read original article]() Published: 2025-01-28T00:00:00Z Content type: tutorial Language: en Sources: [Gunnar Morling]() Topics: [apache-flink](), [Kubernetes](), [Availability](), [observability](), [Deployment](), [container](), [MinIO](), [Amazon S3](), [rocksdb](), [GitHub]() Tags: [apache-flink](), [availability](), [container-images](), [deployment](), [flink](), [github](), [kubernetes](), [minio](), [observability](), [rocksdb](), [s3](), [ui]() ### AI overview Part two of a tutorial series on running Apache Flink jobs on Kubernetes. It explains checkpointing, job manager high availability, savepoints, external object storage, incremental RocksDB checkpoints, observability, and related job-management practices. ### Source excerpt Table of Contents Fault Tolerance and High Availability Manually Triggering Savepoints Observability Bonus: Managing Flink Jobs With the Heimdall UI Summary and Discussion This post originally appeared on the Decodable blog. All rights reserved. Welcome back to this two-part blog post series about running Apache Flink on Kubernetes, using the Flink Kubernetes operator. In part one, we discussed installation and setup of the operator, different deployment types, how to deploy Flink jobs using custom Kubernetes resources, and how to create container images for your own Flink jobs. In this part, we'll focus on aspects such as fault tolerance and high availability of your Flink jobs running on Kubernetes, savepoint management, observability, and more. You can find the complete source code for all the examples shown in this series in the Decodable examples repository on GitHub: on GitHub. ## Decouple Allocations from Request Volume DevFeed: [Decouple Allocations from Request Volume]() Original publisher: [Read original article]() Author: Davide Angelocola Published: 2023-11-30T00:00:00Z Content type: tutorial Language: en Sources: [Davide Angelocola]() Topics: [rocksdb](), [Concurrency](), [Benchmark](), [Java]() Tags: [benchmark](), [concurrency](), [performance](), [rocksdb]() ### AI overview The article examines a RocksDB read path whose allocation rate grew with request volume rather than concurrency. It uses a JMH benchmark to identify heap-allocation and garbage-collection costs, then discusses pooled direct buffers and RocksDB's Java API as a way to reduce per-request copying and allocation. ### Source excerpt Millions of requests per day. RocksDB as the storage engine, returning variable-size values. Nothing about this load profile suggests a performance problem. The implementation has a structural flaw: allocation rate scales with request rate, when it should scale with concurrency. ## Reducing BigQuery Costs: How We Fixed A $1 Million Query DevFeed: [Reducing BigQuery Costs: How We Fixed A $1 Million Query]() Original publisher: [Read original article]() Author: Calvin Zhou Published: 2022-11-03T18:14:51Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [BigQuery](), [Streaming](), [apache-flink](), [Kafka](), [parquet](), [Google Cloud Platform (GCP)](), [Shopify](), [rocksdb]() Tags: [apache-flink](), [bigquery](), [cloud-storage](), [cost](), [kafka](), [parquet](), [pipelines](), [rocksdb](), [shopify](), [sql]() ### AI overview The article describes how a Shopify team discovered that a BigQuery query could cost nearly $1 million USD per month while building a marketing data pipeline. The team used Apache Flink with RocksDB and Apache Kafka, then evaluated BigQuery as an external data warehouse capable of querying large datasets and exporting results to Google Cloud Storage. The supplied text ends as the team encounters the cost problem, before detailing the eventual fix. ### Source excerpt We discovered a query that nearly cost us $1 million USD a month in BigQuery. Below, we'll share our tips for lowering costs in BigQuery. ## 7 Tips For Optimizing Apache Flink Applications DevFeed: [7 Tips For Optimizing Apache Flink Applications]() Original publisher: [Read original article]() Author: Yaroslav Tkachenko Published: 2022-03-08T21:20:39Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [apache-flink](), [Shopify](), [Kubernetes](), [rocksdb](), [Streaming](), [Tooling](), [cpu]() Tags: [apache-flink](), [cpu](), [data](), [debugging](), [jvm](), [kubernetes](), [performance](), [profiling](), [rocksdb](), [shopify](), [streaming]() ### AI overview This Shopify Engineering article presents lessons for optimizing large, stateful Apache Flink applications. It discusses profiling and debugging tools, performance tuning, resiliency, high availability, Kubernetes deployment, RocksDB state management, and checkpoint and savepoint storage in Google Cloud Storage. ### Source excerpt We've spent a lot of time on performance tuning our Apache Flink application. We'll walk you through key lessons for optimizing large stateful Apache Flink applications. ## Allas November 2020 incident details DevFeed: [Allas November 2020 incident details]() Original publisher: [Read original article]() Author: Kalle Happonen (noreply@blogger.com) Published: 2020-12-22T12:31:00Z Content type: article Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [ceph](), [incident](), [Amazon S3](), [rocksdb](), [NVMe](), [Processes](), [Vapor]() Tags: [ceph](), [deep-dive](), [incident](), [nvme](), [processes](), [rocksdb](), [s3](), [software](), [storage](), [swift](), [technical]() ### AI overview A technical deep dive into the November 2020 Allas downtime incident. It describes Allas as a Ceph cluster and explains its storage nodes, OSD processes, S3 and SWIFT access, replicated metadata pools, RocksDB metadata, placement groups, and erasure-coded object storage. ### Source excerpt Allas downtime November 2020 - technical deep-dive We use Ceph (https://ceph.io/) to provide software defined storage in our Allas service. I'll try to speak some Ceph here. There may be a lot of unfamiliar terms for most readers here as we dive deep into a technology. I'll try to clarify where I can, but architectural knowledge of Ceph helps when reading this. Allas is a Ceph cluster with 48 storage nodes (OSD nodes). Each has one nvme, 24 spinning disks, and 192 GiB of memory. Each OSD node runs one OSD process per disk. The Allas storage is provided over SWIFT and S3 protocols through radosgw daemons. The radowsgw layer is provided by separate internet facing servers. In addition we have 5 nodes running monitor processes (they control the state of the cluster), which also run Ceph manager (handles more advanced monitoring views into the cluster state). The object store metadata pools are 3-way replicated pools (data is stored in 3 different copies) on the nvmes across the OSD nodes. This metadata contains e.g. owners of buckets and objects, permissions, versions, etc. Everything that has to do with S3/SWIFT. The nvme also hosts the rocksdb (ceph internal metadata, as opposed to object store metadata) for the OSDs. This metadata is internal Ceph clustster status. Data on placement groups, etc. I'll use the term placement group (or pg) quite a lot in the post as they are central for Ceph. Basically when you store data objects into Ceph, they are mapped into placement groups, which define where in the cluster the data is are stored. The object store data is stored on an 8+3 erasure coded (data is split into 8 chunks + 3 parity chunks) pool on the spinning disks. The total amount of placement groups for the data pool is 16 384. In the graph the amount ~18 000 is shown, as the graph contains other pools too. The beginning Side note: I love that we collect trending data for our cluster (collectd+graphite in our case). As we had the data stored from a long time, this al