# Is Systems Research Really Just About Making Numbers Bigger?

DevFeed: [Is Systems Research Really Just About Making Numbers Bigger?](<https://devfeed.tech/articles/is-systems-research-really-just-about-making-numbers-bigger-12579.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2025/10/12/barbarians.html>)

Author: Marc Brooker

Published: 2025-10-12T00:00:00Z

Content type: opinion

Language: en

Sources: [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog.md>), [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog-2.md>)

Topics: [systems](<https://devfeed.tech/topics/systems.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [machine learning overfitting](<https://devfeed.tech/topics/machine-learning-overfitting.md>), [Programming](<https://devfeed.tech/topics/programming.md>), [Algorithms, Complexity](<https://devfeed.tech/topics/algorithms-complexity.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [algorithms](<https://devfeed.tech/tags/algorithms.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [code](<https://devfeed.tech/tags/code.md>), [research](<https://devfeed.tech/tags/research.md>), [software-engineer](<https://devfeed.tech/tags/software-engineer.md>), [systems](<https://devfeed.tech/tags/systems.md>), [testing](<https://devfeed.tech/tags/testing.md>), [verification](<https://devfeed.tech/tags/verification.md>)

## AI overview

This opinion argues that systems research should broaden beyond making performance numbers bigger. It presents AI-driven solution discovery as especially effective in areas with reliable verifiers, such as software artifacts tested against predefined workloads, while emphasizing that robust testing is essential to prevent overfitting and reward hacking.

## Source excerpt

Is Systems Research Really Just About Making Numbers Bigger? The Barbarian F.C. of systems research would be pretty cool. Lots of folks online have been talking about Barbarians at the Gate: How AI is Upending Systems Research by Cheng, Liu, Pan, et al this week. Maybe unsurprisingly, given the fact that I work in AI for my day job, and both consume and produce systems research, I found it super interesting. Perhaps the most interesting discussion, however, isn't about AI at all. It's about the scope of systems research. What systems research is, or aught to be. The paper's core argument is well captured in the abstract: We argue that systems research, long focused on designing and evaluating new performance-oriented algorithms, is particularly well-suited for AI-driven solution discovery. This is because system performance problems naturally admit reliable verifiers: solutions are typically implemented in real systems or simulators, and verification reduces to running these software artifacts against predefined workloads and measuring performance. I, 100%, enthusiastically, agree with this point. I think it's a point that generalizes way beyond systems research to the entire software industry: AI is going to be most effective in problem spaces where there are what the authors call reliable verifiers. Where, in effect, we can do automated hill climbing towards a low-ambiguity solution or Pareto frontier of solutions. Much of the next decade is going to be defined by finding better techniques to build these reliable verifiers where none existed before. If you're a software engineer, what I'm saying here is testing is going to be the most important thing. A bit later, the paper highlights two of the challenges with building these verifiers for systems builders (and software developers generally): Prevent overfitting. Evaluating against narrow workloads lead to algorithm failures like overfitting, where the solutions either hard-code behaviors or overfit to specific tr