# Practice Makes (Im)Perfect: A Look Back at Benchmarking Practices for Microarchitectural Side-Channel Attacks

DevFeed: [Practice Makes (Im)Perfect: A Look Back at Benchmarking Practices for Microarchitectural Side-Channel Attacks](<https://devfeed.tech/articles/practice-makes-im-perfect-a-look-back-at-benchmarking-practices-for-microarchitectural-side-channel-attacks-118676.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.03893>)

Author: Iliana Fayolle, Antoine Geimer, Daniel De Almeida Braga, Cl\\'ementine Maurice

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [side channel](<https://devfeed.tech/topics/side-channel.md>), [Cryptography](<https://devfeed.tech/topics/cryptography.md>), [Cybersecurity](<https://devfeed.tech/topics/cybersecurity.md>), [Security](<https://devfeed.tech/topics/security.md>)

Tags: [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [completeness](<https://devfeed.tech/tags/completeness.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [reproducibility](<https://devfeed.tech/tags/reproducibility.md>), [security](<https://devfeed.tech/tags/security.md>), [side-channel](<https://devfeed.tech/tags/side-channel.md>), [side-channel-attacks](<https://devfeed.tech/tags/side-channel-attacks.md>), [survey](<https://devfeed.tech/tags/survey.md>)

## AI overview

A survey of 83 microarchitectural side-channel attack papers from top-ranked security and architecture conferences (2014-2024) finds an average of 5.72 recurring benchmarking flaws per paper. The 19 flaws affect evaluation completeness, relevance, soundness, and reproducibility, including unfair or missing comparisons, unavailable code or materials, and failure to measure key attack properties. The authors note that inherited benchmarks such as covert-channel bandwidth or key recovery on naive AES and RSA implementations may not assess newer primitives well, while small changes in experimental conditions can alter results. They propose relevant properties for evaluating new attacks and report trends over time and differences between security and architecture conferences.

## Source excerpt

arXiv:2609.03893v3 Announce Type: replace Abstract: Microarchitectural side-channel research has grown at an exceptional pace in recent years, increasing the need for rigorous and meaningful benchmarking. Early attack papers typically relied on indirect proxies, such as covert-channel bandwidth or key-recovery on naive AES and RSA implementations, setting de facto standards that many subsequent works continued to replicate, sometimes by directly comparing against raw numbers from prior work. While these practices offer convenient points of comparison, current benchmarks may not be the most relevant to assess specific properties of new primitives. Even more problematic, microarchitectural attacks are notoriously sensitive to experimental conditions: minimal changes in the target system can significantly alter outcomes and performance. As a result, inadequate evaluation practices undermine reproducibility and cast doubt on the relevance of comparisons, even in top-tier venues where such issues should be identified. This paper tackles the core problem of proper benchmarking for microarchitectural side-channel attacks and examines its broader impact on research quality in the field. We survey 83 attack papers published in top-ranked security and architecture conferences from 2014 to 2024. From this corpus, we identify and define 19 recurrent benchmarking flaws that affect evaluation completeness, relevance, soundness, and reproducibility. These flaws include unfair or absent comparisons, missing code or materials, and the failure to evaluate the key attack properties. On average, each paper exhibits 5.72 such flaws, highlighting how widespread the issue is, even in highly selective venues. Based on our findings, we identify and suggest key properties that are relevant to properly evaluate new attacks. We also highlight trends over time and different practices between security and architecture conferences.