# BenchMIRT: What are LLM benchmarks actually measuring?

DevFeed: [BenchMIRT: What are LLM benchmarks actually measuring?](<https://devfeed.tech/articles/benchmirt-what-are-llm-benchmarks-actually-measuring-7081.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/allenai/benchmirt>)

Author: Kyle Wiggers

Published: 2026-09-01T21:39:07Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [benchmarking](<https://devfeed.tech/topics/benchmarking.md>), [Jailbreak](<https://devfeed.tech/topics/jailbreak.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [jailbreak](<https://devfeed.tech/tags/jailbreak.md>), [llm](<https://devfeed.tech/tags/llm.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [safety](<https://devfeed.tech/tags/safety.md>)

## AI overview

BenchMIRT is a multidimensional item-response-theory method for auditing what individual prompts in LLM benchmarks measure. It separates capabilities associated with benchmark performance so aggregate scores do not conceal differences among task groups.

## Source excerpt

Today we're introducing BenchMIRT, a new method for auditing LLM benchmarks at the level of individual prompts--the questions and tasks a model is scored on. A benchmark is usually designed to measure a particular ability, such as safety, general reasoning, or instruction following. But the individual tasks inside it may depend on more than that stated goal. Take BBQ, a benchmark designed to test whether models rely on social stereotypes.