# Recalibrating a Typed-Decision Model in an Afternoon: A Hands-On Recipe for Laya's Confidence Scores

DevFeed: [Recalibrating a Typed-Decision Model in an Afternoon: A Hands-On Recipe for Laya's Confidence Scores](<https://devfeed.tech/articles/recalibrating-a-typed-decision-model-in-an-afternoon-a-hands-on-recipe-for-laya-s-confidence-scores-131733.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16499>)

Author: Mohammed Nayeem Khan, Aliyas Ahmad Misri

Published: 2026-11-13T05:00:00Z

Content type: tutorial

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Uncertainty quantification LLMs](<https://devfeed.tech/topics/uncertainty-quantification-llms.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Learning](<https://devfeed.tech/topics/learning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [decision](<https://devfeed.tech/tags/decision.md>), [decision-model](<https://devfeed.tech/tags/decision-model.md>), [laya](<https://devfeed.tech/tags/laya.md>), [model](<https://devfeed.tech/tags/model.md>), [recipe](<https://devfeed.tech/tags/recipe.md>)

## AI overview

Laya's pooled calibration score hides errors that vary by question type and option count: on 1,065 synthetic, LLM-labelled items, pooled ECE was 0.073 while bucket-macro ECE was 0.143. A single global temperature worsened pooled ECE to 0.173, whereas cross-fitted temperatures fitted per bucket reduced macro-ECE to 0.073; the result was 0.089 with 10 labelled items per bucket and 0.099 when one of seven evaluation sets was held out. A gate targeting 10% error answered 46.3% of items at 10.5% error after refitting, versus 6.9% answered using published confidences, though bucket-specific cut-offs alone reached 45.4%, accounting for most of the coverage gain. Confidence-only methods needed at least 20 labels per bucket to approach temperature scaling; with 320, Platt had the lowest macro-ECE. Deployment limitations include missing per-bucket tables in retrained checkpoints and a stored 12-option value clamped at runtime. Because fitting and testing shared a generator and LLM labels, these results apply to the described checkpoints and data.

## Source excerpt

arXiv:2611.16499v1 Announce Type: cross Abstract: A typed-decision model such as Laya reads a text and a list of options that the caller defines, for example ticket categories, and returns a probability for each option. Code around it compares the top probability with a threshold to answer, abstain or escalate to a person. Earlier work reports, for a Turkish model, that temperature scaling can raise held-out calibration error (ECE). This tutorial gives a five-step recipe (measure, diagnose, fit, validate, deploy) in 130 lines of numpy with standard methods, none new. On 1,065 synthetic, LLM-labelled items the released Laya checkpoint has a pooled ECE of 0.073 but a bucket-macro ECE of 0.143 (0.127-0.184), the mean over buckets (questions grouped by type and option count): the 12-option bucket is over-confident by +0.22, the three-to-five-option bucket under-confident by 0.05, and they cancel. One global temperature makes pooled ECE worse (0.173); a cross-fitted temperature per bucket brings macro-ECE to 0.073 (0.078 over 10 fold assignments) and reaches 0.089 with 10 labelled items per bucket, but holding out one of seven evaluation sets gives 0.099. After the refit, a gate aimed at 10% error answers 46.3% of items (10.5% wrong) against 6.9% on the published confidences, but one cut-off per bucket on the published confidences answers 45.4%, so most of the gain comes from separating buckets. Confidence-only maps (Platt, histogram, isotonic) need 20 or more labelled items per bucket to approach the temperature; with 320 per bucket the lowest macro-ECE belongs to Platt. In these files the retrained checkpoints store no per-bucket table and the stored 12-option value 0.1006 is clamped to 0.5 at runtime; the refit helps 31 of 33 checkpoints on random folds and 21 with a set held out. Fit and test share one generator and LLM labels, so these are findings about these checkpoints and this data.