# When Can You Trust Neural Critics? Auditing Mutual Information Estimates of Higher-Order Epistasis in Protein Language Models

DevFeed: [When Can You Trust Neural Critics? Auditing Mutual Information Estimates of Higher-Order Epistasis in Protein Language Models](<https://devfeed.tech/articles/when-can-you-trust-neural-critics-auditing-mutual-information-estimates-of-higher-order-epistasis-in-protein-language-models-123347.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09046>)

Author: Maryam Rahimimovassagh, Waldemar Karwowski, Niloofar Yousefi

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>)

Tags: [auditing](<https://devfeed.tech/tags/auditing.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [esm](<https://devfeed.tech/tags/esm.md>), [estimator](<https://devfeed.tech/tags/estimator.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [information](<https://devfeed.tech/tags/information.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [neural](<https://devfeed.tech/tags/neural.md>), [pairing](<https://devfeed.tech/tags/pairing.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [signal](<https://devfeed.tech/tags/signal.md>), [stable](<https://devfeed.tech/tags/stable.md>)

## AI overview

The audit finds that stabilized neural mutual-information estimates are not sufficient to support order-resolved dependence claims under blind search at the tested dimensionality and sample size. For ESM-2 on the four-site GB1 landscape, run-to-run differences show no systematic bias and are comparable to cross-seed retraining variability, while evaluation pairing shifts scores substantially. Two of three alternative estimator families stabilize estimates across seeds at the original dimensionality; one does not. In synthetic Gaussian-channel calibrations at two other protein language models' representation dimensionalities, all three recover too little signal despite stable seeds and correct signal ordering. A control given the true signal direction recovers 93.3% and 98.7% of analytic MI, showing recovery with privileged information but not explaining blind-estimation failure. The work packages these checks as a reusable diagnostic protocol.

## Source excerpt

arXiv:2611.09046v1 Announce Type: new Abstract: Mutual information (MI) estimation is used to probe what neural representations encode, including in protein language models, but variational MI estimators have variance that scales at least exponentially in the true MI. We audit a prior neural-MI study of order-resolved epistasis in ESM-2 on the four-site GB1 landscape. Run-to-run differences show no systematic bias and a magnitude comparable to ordinary cross-seed retraining variability, and the evaluation-pairing scheme shifts scores substantially. At the original dimensionality, two of three alternative estimator families stabilize the estimate across seeds and one does not. In a synthetic calibration at the representation dimensionalities of two other protein language models (a Gaussian channel with known MI; no model representations), all three families recover materially less signal than required, despite correct signal ordering and stable seeds. A control critic given the true signal direction, which also reduces its input to one dimension, recovers 93.3% and 98.7% of the analytic MI at gain 1.5 (at 5,120 and 4,096 dimensions), so the signal is recoverable with privileged information, but the control does not identify why blind estimation fails. We package the checks into a reusable diagnostic protocol and conclude that even a stabilized estimator is not currently sufficient for order-resolved dependence claims made under blind search at this dimensionality and sample size.