# Evaluating Token Probabilities as Confidence Estimators in Human Behavior Simulations: A Case Study in Luxembourgish Next Like Prediction

DevFeed: [Evaluating Token Probabilities as Confidence Estimators in Human Behavior Simulations: A Case Study in Luxembourgish Next Like Prediction](<https://devfeed.tech/articles/evaluating-token-probabilities-as-confidence-estimators-in-human-behavior-simulations-a-case-study-in-luxembourgish-next-like-prediction-120243.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07735>)

Author: Nils Schwager, Simon M\\"unker, Alistair Plum, Achim Rettinger

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [agents](<https://devfeed.tech/tags/agents.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [case-study](<https://devfeed.tech/tags/case-study.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [data](<https://devfeed.tech/tags/data.md>), [digital](<https://devfeed.tech/tags/digital.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [learning](<https://devfeed.tech/tags/learning.md>), [like](<https://devfeed.tech/tags/like.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llms](<https://devfeed.tech/tags/llms.md>), [next](<https://devfeed.tech/tags/next.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [simulations](<https://devfeed.tech/tags/simulations.md>), [study](<https://devfeed.tech/tags/study.md>), [token](<https://devfeed.tech/tags/token.md>)

## AI overview

In Luxembourgish RTL.lu Next Like Prediction, Ministral 3 models using in-context learning were overconfident and scored below a simple statistical baseline on both accuracy and calibration. Accuracy and confidence calibration diverged: moving from 3B to 8B improved accuracy but worsened calibration, while longer behavioral histories improved accuracy as confidence declined. Task-specific supervised fine-tuning reduced Expected Calibration Error from 0.2 to 0.006 and raised accuracy from 0.536 to 0.758. The case study therefore supports evaluating confidence calibration alongside accuracy when assessing LLM simulations of behavior on noisy digital traces.

## Source excerpt

arXiv:2611.07735v1 Announce Type: cross Abstract: As large language models (LLMs) are increasingly used to simulate human behavior, their operational validity is typically assessed through accuracy alone, implicitly treating human behavior as deterministic. We argue that on noisy digital trace data, a model's performance depends not only on predicting the correct outcome but on its calibration. We present a case study testing whether the token probabilities of open-weight LLMs can serve as confidence estimates in a Next Like Prediction task: predicting how a user of the Luxembourgish news platform RTL.lu rates a comment, conditioned on their prior interactions. Evaluating Ministral 3 (3B, 8B), we find that, with in-context learning only, the tested models are overconfident and are outperformed on accuracy and calibration by a simple statistical baseline. Accuracy and calibration do not move together: scaling from 3B to 8B improves accuracy while degrading calibration, and longer behavioral histories raise accuracy while confidence declines. Task-specific supervised fine-tuning acts as an effective calibration intervention, reducing Expected Calibration Error from 0.2 to 0.006 while improving accuracy from 0.536 to 0.758. We conclude that confidence estimation should be evaluated alongside accuracy to improve validity when using LLM-based agents.