# Black-Box Uncertainty Quantification for Large Language Models via Ensemble-of-Ensembles

DevFeed: [Black-Box Uncertainty Quantification for Large Language Models via Ensemble-of-Ensembles](<https://devfeed.tech/articles/black-box-uncertainty-quantification-for-large-language-models-via-ensemble-of-ensembles-135200.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18453>)

Author: Wang Ma, Debarun Bhattacharjya, Junkyu Lee, Nhan H Pham, Harsha Kokel, Qiang Ji

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Uncertainty quantification LLMs](<https://devfeed.tech/topics/uncertainty-quantification-llms.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [bayesian](<https://devfeed.tech/tags/bayesian.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bias](<https://devfeed.tech/tags/bias.md>), [class](<https://devfeed.tech/tags/class.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [contribute](<https://devfeed.tech/tags/contribute.md>), [covariance](<https://devfeed.tech/tags/covariance.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [error](<https://devfeed.tech/tags/error.md>), [estimator](<https://devfeed.tech/tags/estimator.md>), [eu](<https://devfeed.tech/tags/eu.md>), [failure](<https://devfeed.tech/tags/failure.md>), [gap](<https://devfeed.tech/tags/gap.md>), [generation](<https://devfeed.tech/tags/generation.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>), [uncertainty-quantification](<https://devfeed.tech/tags/uncertainty-quantification.md>)

## AI overview

The method estimates prompt ambiguity (aleatoric uncertainty) separately from model knowledge gaps (epistemic uncertainty) using only black-box sampling. It samples stochastic answers to a fixed prompt, then applies meaning-preserving prompt perturbations; embedding-space variability and the law of total covariance decompose within- and between-perturbation uncertainty. The work also gives a finite-sample bias correction, bounds linking its estimates to Bayesian targets, and identifies failure modes consistency-only black-box methods miss. Across five short QA benchmarks and four instruction-tuned models, it was competitive with white-box baselines and matched leading black-box methods on error ranking at equal generation budgets. A long-form pilot reports claim-level factuality assessment without retraining.

## Source excerpt

arXiv:2611.18453v1 Announce Type: cross Abstract: Reliable use of large language models (LLMs) requires uncertainty estimates that separate ambiguity intrinsic to a prompt from genuine knowledge gaps. Bayesian and deep-ensemble methods deliver this aleatoric/epistemic decomposition in principle, but are computationally prohibitive at LLM scale and require white-box access. Existing black-box methods, in contrast, summarize to a single scalar that cannot make the distinction. We introduce a two-level ensemble that bridges this gap. Stochastic decoding of a fixed prompt forms the inner ensemble; meaning-preserving semantic perturbations of the prompt form the outer ensemble. Variability is measured in a continuous embedding space, and the law of total covariance gives an exact within-/between-perturbation decomposition to estimate aleatoric and epistemic uncertainty (AU/EU); it requires only black-box sampling access. On top of this estimator, we contribute three theoretical results that ground the framework: (i) a finite-sample bias identity for nested Monte Carlo with a closed-form bias-corrected epistemic estimator; (ii) a Bayesian bridge bounding the gap between perturbation-population AU/EU and Bayesian targets; and (iii) a separation result identifying a class of failure modes that any consistency-only black-box method provably misses while perturbation-based EU detects. Across five short QA benchmarks and four instruction-tuned LLMs spanning two families and two scales, our estimator is competitive with strong white-box baselines, and at a matched generation budget it matches leading black-box methods on error ranking while decomposing AU and EU; controlled paired interventions show that AU and EU respond selectively to distinct, actionable sources of failure: prompt ambiguity and model knowledge gaps; and a long-form pilot on the NQ long-answer split shows the framework extends to claim-level factuality without retraining.