# Data-Free Metrics Don't Work: The Missing Invariance Under Functionally Preserving Reparameterisations

DevFeed: [Data-Free Metrics Don't Work: The Missing Invariance Under Functionally Preserving Reparameterisations](<https://devfeed.tech/articles/data-free-metrics-don-t-work-the-missing-invariance-under-functionally-preserving-reparameterisations-132155.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15621>)

Author: Gabryel Mason-Williams, Israel Mason-Williams, Fredrik Dahlqvist

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [neural-networks](<https://devfeed.tech/tags/neural-networks.md>), [random-matrix-theory](<https://devfeed.tech/tags/random-matrix-theory.md>)

## AI overview

The paper argues that random matrix theory (RMT) metrics cannot reliably indicate a neural network's performance or training because functionally preserving changes can make those metrics arbitrary. Across multiple architectures and scales, the authors re-initialize or re-randomize robust layers--layers whose changes do not alter model behavior--while leaving critical layers intact. They also report that relationships among RMT metrics are spuriously associated with generalisation, and that models' apparently favorable metric ranges partly depend on initialization. The evidence summarized here concerns RMT-based metrics and does not establish conclusions about every data-free analysis method.

## Source excerpt

arXiv:2611.15621v1 Announce Type: cross Abstract: Data-free methods for analysing and understanding the layers of neural networks offer many metrics for quantifying notions of 'strong' versus 'weak' layers, with the promise of increased interpretability. In particular, random matrix theory (RMT) offers data-free metrics that claim predictive power over the quality of pre-trained models at a layerwise precision and the ability to identify model pathologies, indicating a unique relationship with generalisation. As a result, metrics from RMT are championed for pre-training optimisation strategies and post-training compression. We establish that RMT-based metrics are unrelated to performance or training by showing that functionally preserving reparametrisations of a pre-trained model can have arbitrary metrics. We show this across a range of architectures and scales. To create functionally preserving reparametrisations, we exploit the well-established phenomenon of criticality: some layers can be re-initialised or re-randomised without affecting the functional behaviour of the model -- they are called robust -- while others cannot -- they are called critical. Re-initialising or re-randomising robust layers provides functionally preserving reparametrisations for which RMT-based metrics are arbitrary. Moreover, we show that relationships between metrics offered by RMT are spuriously related to generalisation. We conclude by showing that if many pre-trained models have data-free metrics in a 'good' range, it is, in part, dependent on model initialisation. Project page available at: https://gmw99.github.io/data-free-metrics-dont-work