# Better Benchmarks Through Graphs

DevFeed: [Better Benchmarks Through Graphs](<https://devfeed.tech/articles/better-benchmarks-through-graphs-12552.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2024/02/12/parameters.html>)

Author: Marc Brooker

Published: 2024-02-12T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog.md>), [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog-2.md>)

Topics: [benchmarking](<https://devfeed.tech/topics/benchmarking.md>), [Databases](<https://devfeed.tech/topics/databases.md>), [Graphs](<https://devfeed.tech/topics/graphs.md>), [Concurrency](<https://devfeed.tech/topics/concurrency.md>)

Tags: [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [blog](<https://devfeed.tech/tags/blog.md>), [blog-post](<https://devfeed.tech/tags/blog-post.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [databases](<https://devfeed.tech/tags/databases.md>), [graphs](<https://devfeed.tech/tags/graphs.md>), [performance](<https://devfeed.tech/tags/performance.md>)

## AI overview

This article examines why database engineering needs better benchmarks, especially benchmarks available during system design. It discusses the limitations of real application workloads, traces, and standard benchmarks, and introduces graph-based benchmarking as the article's focus.

## Source excerpt

Better Benchmarks Through Graphs Isn't the ambiguity in the word *graphs* fun? This is a blog post version of a talk I gave at the Northwest Database Society meeting last week. The slides are here, but I don't believe the talk was recorded. I believe that one of the things that's holding back databases as an engineering discipline (and why so much remains stubbornly opinion-based) is a lack of good benchmarks, especially ones available at the design stage. The gold standard is designing for and benchmarking against real application workloads, but there are some significant challenges achieving this ideal. One challenge1 is that, as in any system with concurrency, traces capture the behavior of the application running on another system, and they might have issued different operations in a different order running on this one (for example, think about how in most traces it's hard to tell the difference between application thinking and application waiting for data, which could heavily influence results if we're trying to understand the effect of speeding up the waiting for data portion). Running real applications is better, but is costly and raises questions of access (not all customers, rightfully, are comfortable handing their applications over to their DB vendor). Industry-standard benchmarks like TPC-C, TPC-E, and YCSB exist. They're widely used, because they're easy to run, repeatable, and form a common vocabulary for comparing the performance of systems. On the other hand, these benchmarks are known to be poorly representative of real-world workloads. For the purposes of this post, mostly that's because they're too easy. We'll get to what that means later. First, here's why it matters. Designing, optimizing, or improving a database system requires a lot of choices and trade-offs. Some of these are big (optimistic vs pessimistic, distributed vs single machine, multi-writer vs single-writer, optimizing for reads or writes, etc), but there are also thousands of small