# Quantiles, Not Points: Streaming Quantile Matching for Ordinal-Aware LLM-as-a-Judge Calibration

DevFeed: [Quantiles, Not Points: Streaming Quantile Matching for Ordinal-Aware LLM-as-a-Judge Calibration](<https://devfeed.tech/articles/quantiles-not-points-streaming-quantile-matching-for-ordinal-aware-llm-as-a-judge-calibration-115032.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.00447>)

Author: Ilija Subasic, Andrew Rabinovich, Zhao Chen

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [distributions](<https://devfeed.tech/tags/distributions.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [llm-as-a-judge](<https://devfeed.tech/tags/llm-as-a-judge.md>), [matching](<https://devfeed.tech/tags/matching.md>), [optimal-transport](<https://devfeed.tech/tags/optimal-transport.md>), [quantile](<https://devfeed.tech/tags/quantile.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

## AI overview

LLM-as-a-judge scores can be distributionally shifted from human ratings, and post-hoc calibration uses a small labeled anchor set to map scores. The paper presents streaming quantile matching (SQM), which treats ratings as ordinal draws from distributions and computes an optimal-transport map between LLM and human score distributions. In the authors' evaluation, SQM respects the human anchors' ordinality and achieves the strongest distributional fidelity and top overall rank across models and data, unlike point-wise calibrations. They also report that calibration generally narrows the raw error gap between smaller and larger judge models, suggesting smaller judges may be more viable. These claims are limited to the abstract's reported results.

## Source excerpt

arXiv:2611.00447v1 Announce Type: new Abstract: LLM-as-a-judge scores are often distributionally shifted from human ratings on any given evaluation task, and post-hoc calibration (mapping LLM to human scores in a small labeled anchor set) has become a popular method to correct for this shift. In this paper we derive the calibration from two positions: that LLMaJ scores are ordinal ratings drawn from a distribution rather than isolated points. We present streaming quantile matching (SQM), a version of a classical method not previously bench marked for LLMaJ calibration, which is built on the two positions by computing the optimal-transport map between the LLM and human score distributions. We develop an evaluation framework and within it we show that unlike point-wise calibrations, SQM respects the ordinality of the human anchor set and achieves the best distributional fidelity and the top overall rank across models and data. We also show that in general, post-hoc calibration narrows the raw error gap between smaller and larger judge models, which opens the possibility of more efficient LLM judges.