# Introducing HaloDB, a fast, embedded key-value storage engine written in Java

DevFeed: [Introducing HaloDB, a fast, embedded key-value storage engine written in Java](<https://devfeed.tech/articles/introducing-halodb-a-fast-embedded-key-value-storage-engine-written-in-java-20489.md>)

Original publisher: [Read original article](<https://yahooeng.tumblr.com/post/178262468576>)

Author: amberwilsonla-blog

Published: 2018-09-19T23:48:42Z

Content type: release

Language: en

Sources: [Yahoo](<https://devfeed.tech/sources/yahoo.md>)

Topics: [Databases](<https://devfeed.tech/topics/databases.md>), [Java](<https://devfeed.tech/topics/java.md>), [benchmarking](<https://devfeed.tech/topics/benchmarking.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Software](<https://devfeed.tech/topics/software.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [data](<https://devfeed.tech/tags/data.md>), [database-performance](<https://devfeed.tech/tags/database-performance.md>), [embedded](<https://devfeed.tech/tags/embedded.md>), [go](<https://devfeed.tech/tags/go.md>), [java](<https://devfeed.tech/tags/java.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [performance](<https://devfeed.tech/tags/performance.md>), [yahoo-engineering](<https://devfeed.tech/tags/yahoo-engineering.md>)

## AI overview

Yahoo developers introduces HaloDB, an embedded key-value storage engine written in Java. Built for an IO-bound workload with high read and write throughput, HaloDB uses append-only log files and an in-memory index. The article reports a 50% improvement in write capacity while maintaining submillisecond read latency at the 99th percentile, and announces the project's availability under the Apache license.

## Source excerpt

yahoodevelopers: By Arjun Mannaly, Senior Software Engineer At Oath, multiple ad platforms use a high throughput, low latency distributed key-value database that runs in data centers all over the world. The database stores billions of records and handles millions of read and write requests per second at millisecond latencies. The data we have in this database must be persistent, and the working set is larger than what we can fit in memory. Therefore, a key component of the database performance is a fast storage engine. Our current solution had served us well, but it was primarily designed for a read-heavy workload and its write throughput started to be a bottleneck as write traffic increased. There were other additional concerns as well; it took hours to repair a corrupted DB, or iterate over and delete records. The storage engine also didn't expose enough operational metrics. The primary concern though was the write performance, which based on our projections, would have been a major obstacle for scaling the database. With these concerns in mind, we began searching for an alternative solution. We searched for a key-value storage engine capable of dealing with IO-bound workloads, with submillisecond read latencies under high read and write throughput. After concluding our research and benchmarking alternatives, we didn't find a solution that worked for our workload, thus we were inspired to build HaloDB. Now, we're glad to announce that it's also open source and available to use under the terms of the Apache license. HaloDB has given our production boxes a 50% improvement in write capacity while consistently maintaining a submillisecond read latency at the 99th percentile. Architecture HaloDB primarily consists of append-only log files on disk and an index of keys in memory. All writes are sequential writes which go to an append-only log file and the file is rolled-over once it reaches a configurable size. Older versions of records are removed to make space by a bac