# Bootstrap Confidence Intervals for LLM Evaluation

DevFeed: [Bootstrap Confidence Intervals for LLM Evaluation](<https://devfeed.tech/articles/bootstrap-confidence-intervals-for-llm-evaluation-29996.md>)

Original publisher: [Read original article](<https://engineering.indeedblog.com/blog/2026/07/bootstrap-confidence-intervals-for-llm-evaluation/>)

Author: James Fiedler

Published: 2026-07-08T14:30:02Z

Content type: article

Language: en

Sources: [Indeed](<https://devfeed.tech/sources/indeed.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [batching](<https://devfeed.tech/tags/batching.md>), [confidence-interval](<https://devfeed.tech/tags/confidence-interval.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [developers](<https://devfeed.tech/tags/developers.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-evaluation](<https://devfeed.tech/tags/llm-evaluation.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [metric](<https://devfeed.tech/tags/metric.md>), [performance](<https://devfeed.tech/tags/performance.md>), [research](<https://devfeed.tech/tags/research.md>), [variance](<https://devfeed.tech/tags/variance.md>), [vs](<https://devfeed.tech/tags/vs.md>)

## AI overview

This article explains how to construct confidence intervals for non-deterministic LLM evaluations using bootstrap resampling. It recommends resampling inputs while carrying along all model runs for each selected input, then using the 2.5th and 97.5th percentiles for a 95% interval. It also discusses the tradeoff between the number of inputs and repeated runs, recommending k = 3 or 5 when feasible.

## Source excerpt

Introduction As Large Language Models (LLMs) move from research prototypes to production systems, the developers of these systems need rigorous performance evaluation. In particular, we need confidence intervals around estimates of system accuracy. However, LLMs introduce a challenge that is unusual for ML systems: they are (operationally) non-deterministic. Even with the temperature set to zero, [...]