# Measured or Real? When Model-Derived Measurements Can Be Corrected or Compared

DevFeed: [Measured or Real? When Model-Derived Measurements Can Be Corrected or Compared](<https://devfeed.tech/articles/measured-or-real-when-model-derived-measurements-can-be-corrected-or-compared-108630.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.38015>)

Author: Yifeng Li, Ziang Ni

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>)

Tags: [arc](<https://devfeed.tech/tags/arc.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [compare](<https://devfeed.tech/tags/compare.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [confidence-bounds](<https://devfeed.tech/tags/confidence-bounds.md>), [correlation](<https://devfeed.tech/tags/correlation.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [error](<https://devfeed.tech/tags/error.md>), [family](<https://devfeed.tech/tags/family.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [infinite](<https://devfeed.tech/tags/infinite.md>), [lead](<https://devfeed.tech/tags/lead.md>), [llama](<https://devfeed.tech/tags/llama.md>), [measurements](<https://devfeed.tech/tags/measurements.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [noise](<https://devfeed.tech/tags/noise.md>), [number](<https://devfeed.tech/tags/number.md>)

## AI overview

Model-derived measurements can be corrected or compared only with attention to reliability and prompt variation. The standard correction for noise-attenuated correlations can produce values above one; the authors prove its mean squared error is infinite even when correlations and reliabilities use separate seeds. Their alternative corrects only when both reliability estimates pass a significance test, using their upper confidence bounds; it stays bounded, with coverage guarantees that assume Gaussian noise, and more than halved error in a simulation with 500 units once reliability was well estimated. Prompt choice can reverse model rankings: on ARC, Qwen2-1.5B trails Llama-3.2-3B by 3.8 points with no examples but leads by 16.6 with two. Across four benchmarks, averaging two settings selected one winner in 15 of 16 reversals. The paper also compares models within each prompt setting with asymptotic family-wise error control for a fixed prompt grid, and reports that aligning embedding coordinates across seeds raised held-out per-dimension reliability from 0.003 to 0.813.

## Source excerpt

arXiv:2610.38015v1 Announce Type: new Abstract: Numbers computed from trained models, such as an example's difficulty or a model's benchmark lead, change with seed and prompt. We ask when such measurements can be corrected or compared. Noise shrinks correlations, and the textbook correction divides by the square root of two estimated reliabilities. On two CIFAR-100 difficulty scores, it returned correlations above one in 70 of 100 classes. We prove that its mean squared error is infinite even when the correlation and the reliabilities use separate seeds. Our rule corrects only when both reliabilities pass a significance test, then divides by their upper confidence bounds. Its output is always bounded, and its coverage guarantees assume Gaussian noise. It helps once reliability is well estimated, more than halving the error with 500 simulated units. Prompt settings can reverse rankings. On ARC, Qwen2-1.5B trails Llama-3.2-3B by 3.8 points with no examples in the prompt and leads by 16.6 with two. Across four benchmarks, averaging the two settings names one winner for 15 of 16 such reversals. We compare models within each setting, controlling family-wise error asymptotically in the number of items for a fixed prompt grid. Aligning embedding coordinates across seeds raises held-out per-dimension reliability from 0.003 to 0.813.