# Next-Level Benchmarking Overfitting

DevFeed: [Next-Level Benchmarking Overfitting](<https://devfeed.tech/articles/next-level-benchmarking-overfitting-122210.md>)

Original publisher: [Read original article](<https://databasearchitects.blogspot.com/2026/11/next-level-benchmarking-overfitting.html>)

Author: Thomas Neumann (noreply@blogger.com)

Published: 2026-11-07T10:14:11Z

Content type: opinion

Language: en

Sources: [Database Architects](<https://devfeed.tech/sources/database-architects.md>)

Topics: [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [DuckDB](<https://devfeed.tech/topics/duckdb.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [blog](<https://devfeed.tech/tags/blog.md>), [blog-post](<https://devfeed.tech/tags/blog-post.md>), [compression](<https://devfeed.tech/tags/compression.md>), [data](<https://devfeed.tech/tags/data.md>), [data-structures](<https://devfeed.tech/tags/data-structures.md>), [database](<https://devfeed.tech/tags/database.md>), [database-benchmarking](<https://devfeed.tech/tags/database-benchmarking.md>), [design](<https://devfeed.tech/tags/design.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [engine](<https://devfeed.tech/tags/engine.md>), [engines](<https://devfeed.tech/tags/engines.md>), [execution](<https://devfeed.tech/tags/execution.md>), [overfitting](<https://devfeed.tech/tags/overfitting.md>)

## AI overview

The author argues GenDB-V's reported 22x-61x TPC-H speedups over Umbra and DuckDB mainly reflect benchmark-specific precomputation and assumptions about the official data generator, not broadly faster query processing. At SF10, Umbra's roughly 540 ms total is already close to the estimated 280 ms minimum to move the data at 54 GB/s, making a 22-fold traditional processing gain implausible. GenDB-V builds per-query auxiliary structures at load time; in plain-array mode, it also exploits known row ordering to make joins such as Q18 cheap. Its verification checks these assumptions and falls back to a generic nested loop if they fail, but the official benchmark data does not trigger that path. The author warns that small data changes could force the slow fallback and argues that optimizing AI agents for individual benchmarks can make benchmark results poor indicators of production performance.

## Source excerpt

In my recent blog post I remarked that database benchmarking is in a crisis because AI agents start to overfit on individual benchmarks with questionable techniques, which drives execution times towards zero. This has happened with TPC-D in the past, and broke the benchmark. TPC-H tries to fight that with rules, but most people who publish (unofficial) TPC-H results simply ignore the rules. In combination with overly eager AI agents, this leads to brutal overfitting. We are seeing the next level of that with the GenDB-V system. The corresponding paper commands "Stop Handcrafting Your Engine! GenDB-V: Towards AI-Generated and Formally Verified Database Systems", and reports speedups of 22x to 61x over Umbra and DuckDB, and all that in a fully verified system. These are indeed impressive claims, and would suggest that traditional database engines are in trouble. Taking a step back, though, it should be clear that a performance improvement of a factor of 22 over Umbra is infeasible, at least if we are talking about traditional query processing. At SF10, the 22 TPC-H queries read roughly 15GB of (compressed) base data, and on a 9950X3D Umbra runs all queries in a total of roughly 540ms. On that machine the memory touched for data divided by the memory bandwidth (54GB/s) alone is about 280ms; anything below that is infeasible. Better compression might bring that down a bit, but not by an order of magnitude. Unsurprisingly, the most dramatic gains of GenDB-V come from precomputation. The agents have carefully handcrafted auxiliary data structures for each query, which are computed at load time, and which avoid most of the work. The physical design is heavily tuned to the queries, too, for example, there is an extra structure that lists the supplier rows in s_acctbal DESC order, which makes Q2 trivial. Similar per-query data structures are built for most queries. And we see that in the performance numbers: Several queries already run in near-constant time at SF100: Q2 in 0