# Fisher-Glass Learning: Classification Bounds and Subpopulation Audits

DevFeed: [Fisher-Glass Learning: Classification Bounds and Subpopulation Audits](<https://devfeed.tech/articles/fisher-glass-learning-classification-bounds-and-subpopulation-audits-125584.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.12702>)

Author: El Mustapha Mansouri

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Predictive AI](<https://devfeed.tech/topics/predictive-ai.md>), [data augmentation](<https://devfeed.tech/topics/data-augmentation.md>)

Tags: [auditing](<https://devfeed.tech/tags/auditing.md>), [audits](<https://devfeed.tech/tags/audits.md>), [background](<https://devfeed.tech/tags/background.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diagnostic](<https://devfeed.tech/tags/diagnostic.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [extend](<https://devfeed.tech/tags/extend.md>), [fail](<https://devfeed.tech/tags/fail.md>), [feature](<https://devfeed.tech/tags/feature.md>), [framework](<https://devfeed.tech/tags/framework.md>), [glass](<https://devfeed.tech/tags/glass.md>), [information](<https://devfeed.tech/tags/information.md>), [learning](<https://devfeed.tech/tags/learning.md>), [lower-bounds](<https://devfeed.tech/tags/lower-bounds.md>), [minimax](<https://devfeed.tech/tags/minimax.md>), [performance](<https://devfeed.tech/tags/performance.md>), [risk](<https://devfeed.tech/tags/risk.md>), [search](<https://devfeed.tech/tags/search.md>), [selection](<https://devfeed.tech/tags/selection.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [vulnerability](<https://devfeed.tech/tags/vulnerability.md>)

## AI overview

The paper links nuisance-adjusted predictive information to local classification difficulty, deriving lower bounds on local minimax excess risk under stated distinguishability and decision-separation conditions, and proposes audits that account for estimation error and regularization. In Waterbirds experiments, a feature-based background-alignment diagnostic selects a correctly classified group before held-out evaluation; across five seeds, preserving background type yields 93.2% accuracy, versus 40.6% after changing it while keeping the extracted bird foreground and labels fixed. A separate search identifies a population with 98.8% membership-weighted accuracy and low estimated projected predictive Fisher information. Training and acquisition experiments also show that information gains can diverge from predictive gains. The authors argue for assessing predictive performance, information under declared nuisance models, and controlled sensitivity together.

## Source excerpt

arXiv:2611.12702v1 Announce Type: new Abstract: Accurate classifiers can fail when background cues change without altering the correct labels. Predictive performance alone does not quantify how much task information remains after accounting for such nuisance variation. Here we extend the Fisher-glass framework to connect nuisance-adjusted information with local classification difficulty and subpopulation auditing. We derive lower bounds on local minimax excess classification risk under explicit distinguishability and decision-separation conditions, alongside statistical audits accounting for estimation error and regularization. On Waterbirds, a feature-based diagnostic of task-background alignment selects an accurately classified group, with selection frozen before held-out evaluation. Across five training seeds, background replacements preserving the extracted bird foreground and labels yield 93.2% accuracy when background type is preserved and 40.6% when it changes, a controlled difference of 52.6 percentage points. Separately, search using task labels and a fixed nuisance direction identifies an independently evaluated population with 98.8% membership-weighted accuracy and low estimated projected predictive Fisher information. Training and acquisition experiments show that information gains and predictive gains can diverge. These findings support joint assessment of predictive performance, information under declared nuisance models and controlled sensitivity when diagnosing vulnerability and evaluating model improvements.