# Bet on Features: Anytime-Valid and Feature-Aware Auditing of Conditional Quantile Forecasters

DevFeed: [Bet on Features: Anytime-Valid and Feature-Aware Auditing of Conditional Quantile Forecasters](<https://devfeed.tech/articles/bet-on-features-anytime-valid-and-feature-aware-auditing-of-conditional-quantile-forecasters-108519.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.11653>)

Author: Ivane Antonov, Sohom Mukherjee, Richard Pibernik, Yo Joong Choe

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>)

Tags: [auditing](<https://devfeed.tech/tags/auditing.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [detection](<https://devfeed.tech/tags/detection.md>), [features](<https://devfeed.tech/tags/features.md>), [framework](<https://devfeed.tech/tags/framework.md>), [quantile](<https://devfeed.tech/tags/quantile.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [testing](<https://devfeed.tech/tags/testing.md>)

## AI overview

The paper proposes a distribution-free, game-theoretic framework for continuously auditing black-box conditional quantile forecasts when observations may be dependent and nonstationary. It defines calibration relative to the contextual features available to an auditor, showing that coarse information can hide miscalibration. For contextual bets linear in those features, it gives finite-time detection guarantees without assuming i.i.d. data; the resulting evidence processes are intended to make miscalibration interpretable and feature-aware. Experiments on simulated and real data report that Chronos-2 is highly miscalibrated with respect to multiple relevant features.

## Source excerpt

arXiv:2607.11653v2 Announce Type: replace Abstract: Black-box conditional quantile forecasts are widely used for sequential decisions under asymmetric costs. Once deployed, such forecasters must be monitored continuously as data streams drift and regimes change; this invalidates standard, fixed-horizon backtests for calibration. Further, existing backtests do not take into account that the notion of calibration is, in fact, information-dependent: forecasts can look calibrated to an auditor with coarse information while being miscalibrated to an auditor with richer information. We develop a distribution-free and game-theoretic testing framework for continuously auditing such forecasters with dependent, potentially nonstationary observations. We first formalize notions of conditional quantile calibration relative to the contextual features available to the auditor, showing how coarser information can limit detection. We then identify detectable alternative classes and, for contextual bets linear in the features, derive finite-time detection guarantees without an i.i.d. assumption. The resulting evidence processes provide interpretable, "feature-aware" evidence for miscalibration. We empirically validate these methods on simulated and real data, finding that a popular time series forecaster (Chronos-2) is highly miscalibrated w.r.t. multiple relevant features.