# StrictPRMBench: A Replayable Evaluation Benchmark for Auditing Fixed-Score PRM Aggregation

DevFeed: [StrictPRMBench: A Replayable Evaluation Benchmark for Auditing Fixed-Score PRM Aggregation](<https://devfeed.tech/articles/strictprmbench-a-replayable-evaluation-benchmark-for-auditing-fixed-score-prm-aggregation-111570.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.41333>)

Author: Pengfei Zheng

Published: 2026-11-02T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [audit](<https://devfeed.tech/topics/audit.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [aggregates](<https://devfeed.tech/tags/aggregates.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [audit](<https://devfeed.tech/tags/audit.md>), [auditing](<https://devfeed.tech/tags/auditing.md>), [availability](<https://devfeed.tech/tags/availability.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [best](<https://devfeed.tech/tags/best.md>), [best-of](<https://devfeed.tech/tags/best-of.md>), [bootstrap](<https://devfeed.tech/tags/bootstrap.md>), [but](<https://devfeed.tech/tags/but.md>), [cells](<https://devfeed.tech/tags/cells.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [score](<https://devfeed.tech/tags/score.md>)

## AI overview

StrictPRMBench audits how product and geometric mean (GM) reducers select among fixed reasoning candidates using process reward model scores. In stored N=32 outcomes across seven scoring checkpoints and two benchmarks, deletion interventions produced greater selection instability for product in all 14 cells: the equal-cell difference was +0.0965 (95% shared-problem bootstrap interval 0.0846-0.1086). A sensitivity analysis excluding replacement slots that were either reducer's clean winner retained 4,459 of 4,622 records and found a +0.0986 difference (0.0875-0.1098). Effects vary: Qwen results had positive bootstrap intervals on retained support, but exact paired p-values exceeded 0.05; clean-pool accuracy did not establish equivalence. The benchmark offers versioned outcome records and CPU reanalysis, but missing primary edited-score vectors limit independent testing of new reducers under the same stress. The authors frame this as an audit of recorded behavior, not evidence of deployment-wide harm or a universally best reducer.

## Source excerpt

arXiv:2610.41333v1 Announce Type: new Abstract: Process reward models (PRMs) score reasoning steps, but best-of-N selection also requires a rule that aggregates those scores. We study that rule as a separate evaluation object. StrictPRMBench fixes mathematical reasoning pools, scorer evidence, and the source trace of a deletion intervention, so that different reducers face the same recorded selection problem. Across seven scoring checkpoints and two benchmarks, the stored N=32 outcomes give higher deletion-induced selection instability for product than geometric mean (GM) in all 14 cells. The equal-cell difference is +0.0965; a shared-problem bootstrap, conditional on these checkpoints and frozen pools, gives a 95% interval of [0.0846,0.1086]. A pre-injection-defined sensitivity excludes replacement slots that are either reducer's clean winner and retains 4,459 of 4,622 cell-problem records. The retained difference is +0.0986 [0.0875,0.1098]. Effects are heterogeneous: the two Qwen cells have positive percentile-bootstrap intervals on retained support but two-sided exact paired p-values of 0.0625 and 0.0703. Clean-pool Qwen accuracy does not detect a product advantage, but this is not an equivalence result. We provide versioned outcome records, CPU reanalysis, and an explicit evidence-availability contract. The contribution is a diagnostic evaluation protocol and a reproducible audit of recorded selector behavior, not a new PRM, a universally optimal reducer, or evidence of deployment-wide harm. Raw-score replay and outcome-record reanalysis are distinguished: missing primary edited-score vectors limit independent evaluation of new reducers under the same stress.