# The Price of Choice in Selective Prediction

DevFeed: [The Price of Choice in Selective Prediction](<https://devfeed.tech/articles/the-price-of-choice-in-selective-prediction-94870.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.29067>)

Author: Daniel Cie\\'slak, Andrzej Czy\\.zewski

Published: 2026-10-22T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [test](<https://devfeed.tech/topics/test.md>), [audit](<https://devfeed.tech/topics/audit.md>)

Tags: [audit](<https://devfeed.tech/tags/audit.md>), [choice](<https://devfeed.tech/tags/choice.md>), [classifiers](<https://devfeed.tech/tags/classifiers.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [contracts](<https://devfeed.tech/tags/contracts.md>), [cost](<https://devfeed.tech/tags/cost.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [error](<https://devfeed.tech/tags/error.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [gamma](<https://devfeed.tech/tags/gamma.md>), [gap](<https://devfeed.tech/tags/gap.md>), [gate](<https://devfeed.tech/tags/gate.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [image](<https://devfeed.tech/tags/image.md>), [it](<https://devfeed.tech/tags/it.md>), [linear](<https://devfeed.tech/tags/linear.md>), [lower-bound](<https://devfeed.tech/tags/lower-bound.md>), [minimax](<https://devfeed.tech/tags/minimax.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [power](<https://devfeed.tech/tags/power.md>)

## AI overview

Refuting an existential risk-coverage claim requires ruling out every rule in the predeclared menu. The intersection-union test (IUT) does so without a multiplicity correction, but its sample cost grows with menu richness: for K-interval menus, the minimax sample size scales as K(1−κ)/(κγ²), up to logarithmic factors, in the stated regime, and exact IUT attains this rate. The lower bound holds even when the score distribution is known; the upper bound extends to menus of VC dimension d. Audit procedures that refuse to reject whenever any rule appears feasible can incur linear K cost, though a square-root rate may be possible; unknown heterogeneous risks can make linear cost unavoidable. Experiments report positive exact-hiding windows across tested VLM confidence-gate families, with weaker transfer on a cleaner multi-annotator target. Frozen image and multimodal classifiers show boundary behavior consistent with the guarantee and strong power against practical certificates.

## Source excerpt

arXiv:2610.29067v1 Announce Type: new Abstract: A selective classifier may be claimed to satisfy a risk-coverage contract if some rule from a predeclared menu (a threshold or a union of $K$ score intervals) attains coverage at least $\kappa$ and error at most $\alpha$. Refuting this existential claim requires ruling out every permitted rule. The intersection-union test (IUT) achieves validity without a multiplicity correction, but its sample cost can increase with menu richness. We characterize this cost and compare it with the optimal audit. For a fixed interior risk target $\alpha$ and gap $\gamma$, the minimax sample size for K-interval menus at coverage floor $\kappa$ is $K(1-\kappa)/(\kappa\gamma^2)$, up to logarithmic factors, in the stated small-gap, large-menu regime; exact IUT attains this rate. The lower bound holds even with a known score distribution, while the upper bound extends to menus of VC dimension $d$. Procedures that refuse to reject whenever any rule appears feasible on the audit sample can pay a linear $K$ cost even when a square-root rate is possible. Reference-free testing reaches a smaller scale under constant-risk alternatives, whereas unknown heterogeneous risks can make the linear cost unavoidable. In simulations with $K=4$, the raw menu cost varies 7.7-fold between $\kappa=.5$ and $\kappa=.95$; after theorem-motivated normalization the endpoint values differ by approximately 5%. Held-out experiments with three VLM judges show positive exact-hiding windows in all 60 split-specific confidence-gate families, with 28/30 contracts chosen on audit-A remaining hidden on untouched audit-B. On a cleaner multi-annotator target, hiding is weaker and transfers in 3/10 resplits. Frozen image and multimodal classifiers show boundary behavior consistent with the IUT guarantee and strong power relative to practical certificates.