# Calibrated uncertainty

A machine-learning concept in which a classifier's predicted probabilities are calibrated to reflect observed outcome frequencies.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Learning Which Evidence-Seeking Decisions Matter: Calibrated Counterfactual Replay for Language Agents

DevFeed: [Learning Which Evidence-Seeking Decisions Matter: Calibrated Counterfactual Replay for Language Agents](<https://devfeed.tech/articles/learning-which-evidence-seeking-decisions-matter-calibrated-counterfactual-replay-for-language-agents-144902.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.05263>)

Author: Renwei Meng, Chengyi Sun, Hongyi Zhang, Xiaoyi Hou

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [learning](<https://devfeed.tech/tags/learning.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>)

### AI overview

PCCC-RL estimates which evidence-seeking actions matter by restoring a saved tool state, comparing the chosen action with a feasible alternative, and letting the same frozen policy complete both branches; the difference in final score estimates the action's value. To reduce replay costs, it predicts credit, checks a random subset, and corrects prediction errors using recorded selection probabilities. Alternatives depend only on the prefix, preserving a usable gradient baseline. On HotpotQA and MuSiQue, joint exact match--requiring both a correct answer and complete supporting evidence--was 56.71% and 36.82%, respectively, 4.19 and 3.86 points above a baseline with the same replay access. Gains concentrate in complete evidence chains and accumulate over many updates; single-update effects remain uncertain, while citation coverage and distinctions such as "none" versus "not all" remain open.

### Source excerpt

arXiv:2606.05263v2 Announce Type: replace Abstract: Language agents answer questions by searching, reading, and checking claims. A reward on the final answer says little about which of those decisions earned it, since an agent can reach the right string after redundant searches or with half the evidence missing. We treat the question as an experiment. At a saved decision point, PCCC-RL, for prefix-conditioned counterfactual credit, restores the tool state, runs the action the agent chose alongside a feasible alternative, and lets the same frozen policy finish both branches. The difference in final score then samples what the decision was worth. Replaying everything is too expensive, so the method predicts each action's credit up front, checks a random subset, and uses the recorded selection probabilities to correct the errors of that prediction, spending the budget where a further check can still move the training signal. Alternatives are drawn from the prefix alone, which keeps the comparison a usable gradient baseline. On HotpotQA and MuSiQue, joint exact match, requiring a correct answer together with complete supporting evidence, reaches 56.71% and 36.82%, gains of 4.19 and 3.86 points over a baseline with the same replay access. The gain concentrates in complete evidence chains and accumulates over many updates. Single-update effects stay uncertain, and citation coverage and scope distinctions such as "none" versus "not all" remain open.

## LLMs Are Already Confidence-Aware Zero-Shot Classifiers

DevFeed: [LLMs Are Already Confidence-Aware Zero-Shot Classifiers](<https://devfeed.tech/articles/llms-are-already-confidence-aware-zero-shot-classifiers-145035.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25289>)

Author: Owen Cook, Xingyi Song

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Uncertainty quantification LLMs](<https://devfeed.tech/topics/uncertainty-quantification-llms.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>)

Tags: [black-box](<https://devfeed.tech/tags/black-box.md>), [brier-score](<https://devfeed.tech/tags/brier-score.md>), [classification](<https://devfeed.tech/tags/classification.md>), [classifier](<https://devfeed.tech/tags/classifier.md>), [classifiers](<https://devfeed.tech/tags/classifiers.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [decision](<https://devfeed.tech/tags/decision.md>), [decode](<https://devfeed.tech/tags/decode.md>), [discrimination](<https://devfeed.tech/tags/discrimination.md>), [f1](<https://devfeed.tech/tags/f1.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [gemma-4](<https://devfeed.tech/tags/gemma-4.md>), [gemma-4-31b](<https://devfeed.tech/tags/gemma-4-31b.md>), [generation](<https://devfeed.tech/tags/generation.md>), [jev](<https://devfeed.tech/tags/jev.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llms](<https://devfeed.tech/tags/llms.md>), [zero-shot](<https://devfeed.tech/tags/zero-shot.md>)

### AI overview

Across eleven open-weight LLMs and five classification datasets, confidence method choice rarely changed accuracy significantly, but token-probability logit scoring discriminated correct from incorrect predictions better than verbalised confidence or sampling (AUROC 0.74-0.77 vs. 0.57-0.66). Replacing labels with single-token identifiers preserved discrimination but reduced accuracy by 3-6 points on average. Decision-only scoring, which scores only positions where labels diverge in a prefix trie, cut decode steps by 88-96% on the three largest label sets while performing comparably to full-label scoring. The best tested open-weight configurations, Gemma 4 31B and Qwen 3.5 9B, were reported as competitive with Jev 1.13.0. These findings are from the supplied abstract.

### Source excerpt

arXiv:2611.25289v1 Announce Type: cross Abstract: Zero-shot large language models (LLMs) are increasingly used as text classifiers, but their labels are only useful when we know to what extent we can trust them. We compare black-box methods (verbalised confidence and sampling) with white-box methods that read token probabilities (P(True), and multiple logit-scoring methods using constrained generation). We evaluate eleven open-weight LLMs (1B-32B parameters) on five classification datasets with 3 to 151 labels, measuring classification performance (accuracy, F1-macro), discrimination (AUROC), and calibration (ECE and Brier score). The choice of confidence method rarely impacts accuracy significantly, but logit-based scoring methods lead to much better discrimination (AUROC 0.74-0.77) than black-box methods (AUROC 0.57-0.66). Replacing labels with single-token identifiers preserves discrimination but costs 3-6 accuracy points on average. We also introduce decision-only scoring, which only scores positions where labels diverge in the prefix tree (trie). It reduces decode steps by 88-96% on the three largest label sets and performs comparably to full-label scoring. The best open-weight model-method configurations (Gemma 4 31B and Qwen 3.5 9B) are competitive with the commercial Jev 1.13.0 classifier.

## Calibrated Decision Rewards: Training Language Models with a Judge That Knows How Sure It Is

DevFeed: [Calibrated Decision Rewards: Training Language Models with a Judge That Knows How Sure It Is](<https://devfeed.tech/articles/calibrated-decision-rewards-training-language-models-with-a-judge-that-knows-how-sure-it-is-145472.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25135>)

Author: Rudransh Agnihotri, Manasvi Kapoor

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [grpo](<https://devfeed.tech/topics/grpo.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>)

Tags: [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [decision-model](<https://devfeed.tech/tags/decision-model.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [probability](<https://devfeed.tech/tags/probability.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward-model](<https://devfeed.tech/tags/reward-model.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

RLCDR uses an LLM decision model to assign calibrated probability distributions to typed questions about a response, such as whether its final answer is correct. The probabilities are read in one forward pass, adjusted for option order, and calibrated using about a hundred labels. In GRPO training of Qwen3-1.7B on MATH, this reference-free reward raised MATH500 accuracy from 72.2% to 76.8%, close to the answer-key reward's 76.9%; a random reward gained 0.65 points. On the hardest level, accuracy rose from 44.8% to 51.7%. The authors report lower calibration error than a softmax judge using the same 27B backbone, and better first-error detection than a dedicated 7B process reward model. A limitation appears on problems the base model mostly fails: RLCDR still improved MATH500 by 1.9 points but lagged the answer-key reward as its probabilities became overconfident on that distribution.

### Source excerpt

arXiv:2611.25135v1 Announce Type: new Abstract: Reinforcement learning (RL) can only teach a language model what its reward can recognize. Answer checkers are exact but need a reference answer; learned reward models and LLM judges work everywhere, but their scores are not probabilities. We study RLCDR (Reinforcement Learning from Calibrated Decision Rewards). An LLM converted into a decision model answers a typed question about a response ("is the final answer correct?", "which step is the first error?") with a probability distribution, read in one forward pass, debiased over option orders and calibrated with about a hundred labels. Training Qwen3-1.7B with GRPO on MATH, this reference-free reward raises MATH500 accuracy from 72.2% to 76.8% (+4.65 points, p < 1e-4), matching training on the answer key (76.9%), while a random reward gains only +0.65. The gain grows with difficulty, from 44.8% to 51.7% on the hardest level. On the policy's own changing samples the reward stays calibrated: its expected calibration error is half that of a softmax judge on the same 27B backbone (0.074 vs. 0.141), and it shows no sign of reward hacking. Without step training, the same model finds the first wrong step better than a dedicated 7B process reward model (ProcessBench F1 0.772 vs. 0.722, 20/20 splits). We also map where the approach stops being enough: trained only on problems the base model mostly fails, the reference-free reward still improves the policy (+1.9 MATH500 points, p = 0.03) but trails the answer key, because its probabilities become overconfident on that distribution. We state six hypotheses in advance and test each with a control designed to falsify it.

## Loss-Guided Tsetlin Machines: Making the Learning Objective Explicit

DevFeed: [Loss-Guided Tsetlin Machines: Making the Learning Objective Explicit](<https://devfeed.tech/articles/loss-guided-tsetlin-machines-making-the-learning-objective-explicit-145145.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26114>)

Author: Mayur Shende, Runar Helin, Ole-Christoffer Granmo, Vladimir I. Zadorozhny

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Model Development](<https://devfeed.tech/topics/model-development.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>)

Tags: [classification](<https://devfeed.tech/tags/classification.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [interpretability](<https://devfeed.tech/tags/interpretability.md>), [loss-functions](<https://devfeed.tech/tags/loss-functions.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>)

### AI overview

Loss-Guided Tsetlin Machines replace the fixed, implicit margin-based update rule with a swappable explicit loss that drives clause feedback and weight updates, while leaving the Tsetlin Automaton and clause logic unchanged. The paper evaluates cross entropy (CE), symmetric cross entropy (SCE), and asymmetric loss (ASL) on six multiclass, multilabel, and binary classification datasets. With a suitable loss, accuracy and F1 match or exceed the baseline on five datasets; ASL improves macro F1 by up to 9 percentage points on imbalanced multilabel data. CE reduces calibration error by a factor of 3 to 11 across all six datasets. A qualitative per-class inspection on image and text data reports that clause-based global interpretability is preserved.

### Source excerpt

arXiv:2611.26114v1 Announce Type: new Abstract: The Tsetlin Machine (TM) has no explicit notion of a loss function. Its learning signal is fixed to an implicit, margin-based update rule (MBUR), with no way to choose an objective function suited to a specific task. This paper introduces Loss-Guided TM, which replaces MBUR with an explicit and swappable loss function. In our proposed method, clause feedback and weight updates are driven by the loss, while keeping the Tsetlin Automaton and clause logic unchanged. The approach incorporates cross entropy (CE), symmetric cross entropy (SCE), and asymmetric loss (ASL). We evaluate Loss-Guided TM across six datasets that span multiclass, multilabel, and binary classification tasks. With a suitable loss, Loss-Guided TM matches or improves accuracy and F1 over MBUR on five of the six datasets. The gains are greatest for the imbalanced multilabel datasets, where ASL improves macro F1 by up to 9 percentage points. With CE, Loss-Guided TM is also significantly better calibrated than MBUR on all six datasets, reducing calibration error by a factor of 3 to 11. A qualitative inspection shows that the clause-based global interpretability, examined at the per-class level on image and text data, is preserved when using the Loss-Guided mechanism. These results show that our explicit TM learning objective allows direct targeting of uncertainty calibration, data imbalance, and task structure, opening the TM to the broader literature on loss functions.

## Bradley-Terry Training Can Distort Outcome Probabilities of Initially Accurate Language World Models

DevFeed: [Bradley-Terry Training Can Distort Outcome Probabilities of Initially Accurate Language World Models](<https://devfeed.tech/articles/bradley-terry-training-can-distort-outcome-probabilities-of-initially-accurate-language-world-models-145074.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25594>)

Author: Xinze Shi, Litian Zhang, Zhenjie Su, Zelong Yu, Yuting Zhang, Qihua Feng, Zifeng Kang, Chaozhuo Li, Zhengxuan Hu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [World models](<https://devfeed.tech/topics/world-models.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>)

Tags: [bradley-terry](<https://devfeed.tech/tags/bradley-terry.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distributions](<https://devfeed.tech/tags/distributions.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [probability](<https://devfeed.tech/tags/probability.md>), [robot](<https://devfeed.tech/tags/robot.md>), [robust](<https://devfeed.tech/tags/robust.md>), [shift](<https://devfeed.tech/tags/shift.md>), [training](<https://devfeed.tech/tags/training.md>), [utility](<https://devfeed.tech/tags/utility.md>), [world-models](<https://devfeed.tech/tags/world-models.md>)

### AI overview

In a simulated robot push, fitting a language world model with a Bradley-Terry objective distorted its initially accurate outcome probabilities: 33-43 percentage points shifted to outcomes absent from the enumerated branches, despite 2.4 pp hazard error on unseen layouts before feedback. The paper attributes the shift to the objective: its precomputed optimum predicts the trained result, while a likelihood that accounts for the judge's rule avoids the shift using the same comparisons. The effect recurs on unseen conditions and under varied candidate laws and denser enumeration; about 29 pp remain after excluding physically impossible events. Decision costs depend on those events: with them, inherited thresholds cause false rejects and utility losses of about 3 pp per decision on trained conditions and 1-2 pp on unseen ones. Without them, average utility loss is below 1 pp over six seeds, though small action-choice regret remains.

### Source excerpt

arXiv:2611.25594v1 Announce Type: new Abstract: Comparative feedback, in which a judge says which of two candidate outcomes is closer to what happened, is a practical way to train language world models that assign probabilities to an action's outcomes. We show that, fitted with a Bradley-Terry objective, it need not teach the true outcome probabilities and can distort accurate ones. On a simulated robot push with enumerated outcome distributions, this objective takes a model that is accurate on unseen layouts (hazard error 2.4 percentage points, pp) and moves 33 to 43 pp of its probability onto outcomes that none of a condition's enumerated branches produces. The cause is the objective, not the comparisons: its optimum, computed before training, predicts where the model ends up, and a likelihood that knows the judge's rule learns from the same comparisons without the shift. The shift is robust: it recurs on unseen conditions, under five candidate laws and a three-fold denser enumeration, and about 29 pp remain without the physically impossible events. Its cost is conditional: with those events it inflates hazard probabilities, and a threshold inherited from the pre-feedback model costs about 3 pp of utility per decision on trained and 1 to 2 pp on unseen conditions, through false rejects; without them the utility loss averages below 1 pp over six seeds, though a small regret in action choice remains. Audits of readout, response and decision trace the effect from probabilities to actions.

## Risk-Calibrated Intent Routing for Conversational AI via Signed Lexical Evidence

DevFeed: [Risk-Calibrated Intent Routing for Conversational AI via Signed Lexical Evidence](<https://devfeed.tech/articles/risk-calibrated-intent-routing-for-conversational-ai-via-signed-lexical-evidence-142570.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.00262>)

Author: Yezhou Cheng, Zehua Yang, Bojun Lin

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Conversational AI](<https://devfeed.tech/topics/conversational-ai.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>)

Tags: [confidence](<https://devfeed.tech/tags/confidence.md>), [conversational-ai](<https://devfeed.tech/tags/conversational-ai.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [lexical](<https://devfeed.tech/tags/lexical.md>), [risk](<https://devfeed.tech/tags/risk.md>), [routing](<https://devfeed.tech/tags/routing.md>)

### AI overview

The method combines the gap between a sentence classifier's top two intent scores with signed lexical evidence from words and short character patterns, then uses separate calibration data and a statistical upper bound on error to set an acceptance threshold. Across ten runs per dataset, it reduced risk-coverage area versus a confidence model using classifier scores alone by 15.8% on BANKING77, 15.1% on CLINC150, and 11.8% on HWU64. At a 5% error target, accepted-request coverage increased by 1.83 percentage points on BANKING77 and 5.14 on HWU64; CLINC150 already had near-full coverage. At a 2% target, calibration found an accepting rule in all 30 runs.

### Source excerpt

arXiv:2610.00262v2 Announce Type: replace-cross Abstract: Conversational AI has become widely used as advances in language models let people find information, get writing help, and ask for practical guidance in their own words. When an assistant handles specific tasks, it must infer what the user wants and decide whether to act on that prediction. A sentence classifier's confidence can miss clues in the wording. We combine the gap between the classifier's two highest scores with signed lexical evidence, a score based on words and short character patterns. The score is positive when those patterns favor the predicted intent and negative when they favor another. We use separate calibration data and a statistical upper bound on the error rate to set the minimum confidence required to accept a request. Across ten runs per dataset, our method reduces the area under the risk-coverage curve, which summarizes error as more requests are accepted, by 15.8%, 15.1%, and 11.8% on BANKING77, CLINC150, and HWU64, respectively, compared with a confidence model that uses only sentence classifier scores. At a 5% error target, the share of accepted requests rises by 1.83 percentage points on BANKING77 and 5.14 points on HWU64. CLINC150 already has near-full coverage at that target. At a 2% target, calibration finds a rule that accepts some requests in all 30 runs. Our contribution is a two-feature confidence model that improves routing by measuring how strongly the wording supports or challenges the original intent prediction.

## Learning Calibrated and Samplable Predictive Distributions of Surface-Ocean $\\Delta p$CO$\_2$ Using Tabular Diffusion Models

DevFeed: [Learning Calibrated and Samplable Predictive Distributions of Surface-Ocean $\\Delta p$CO$\_2$ Using Tabular Diffusion Models](<https://devfeed.tech/articles/learning-calibrated-and-samplable-predictive-distributions-of-surface-ocean-delta-p-co-2-using-tabular-diffusion-models-141674.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21769>)

Author: Gabriele Accarino, Mohammad M. Amirian, Thea H. Heimdal, Amanda R. Fay, Galen A. McKinley, Viviana Acquaviva

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Climate / Weather / Ocean Modeling](<https://devfeed.tech/topics/climate-weather-ocean-modeling.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [Predictive AI](<https://devfeed.tech/topics/predictive-ai.md>)

Tags: [conditional](<https://devfeed.tech/tags/conditional.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [diffusion-models](<https://devfeed.tech/tags/diffusion-models.md>), [distributions](<https://devfeed.tech/tags/distributions.md>), [generative](<https://devfeed.tech/tags/generative.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [learning](<https://devfeed.tech/tags/learning.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [model](<https://devfeed.tech/tags/model.md>), [ocean](<https://devfeed.tech/tags/ocean.md>), [physics-ao-ph](<https://devfeed.tech/tags/physics-ao-ph.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [probabilistic](<https://devfeed.tech/tags/probabilistic.md>), [spatial](<https://devfeed.tech/tags/spatial.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>), [uncertainty-quantification](<https://devfeed.tech/tags/uncertainty-quantification.md>), [xgboost](<https://devfeed.tech/tags/xgboost.md>)

### AI overview

Treeffuser estimates a conditional predictive distribution for sea-air ΔpCO₂ from 13 environmental, spatial and temporal predictors, providing observation-level uncertainty for sparse ocean measurements. On a held-out SOCAT test set, it had the best continuous ranked probability score among the evaluated uncertainty methods (8.229 μatm) and coverage closest to nominal 68% and 95% levels (0.686 and 0.930). Its point-prediction skill was comparable to deterministic XGBoost, with RMSE 18.4 versus 17.7 μatm and R² 0.77 versus 0.79. The abstract says generated samples may support probabilistic analyses; spatially coherent field reconstruction remains future work requiring cross-grid-cell dependence modeling and evaluation on spatially held-out data.

### Source excerpt

arXiv:2611.21769v1 Announce Type: new Abstract: The ocean absorbs almost 30% of anthropogenic CO$_2$ emissions, but quantifying this sink requires estimating surface-ocean CO$_2$ from observations that are sparse in space and time. Most existing machine-learning estimates of this field are deterministic, giving a single value per grid point, with uncertainty approximated post hoc via residuals, ensembles, or prescribed distributions. We instead learn a conditional predictive distribution for each observation: using Treeffuser, a conditional diffusion model with gradient-boosted-tree score functions, we estimate the full predictive distribution of $\Delta p$CO$_2$ (the sea-air $p$CO$_2$ difference) from 13 environmental, spatial, and temporal predictors. On a held-out SOCAT test set, Treeffuser attains the lowest continuous ranked probability score among the uncertainty-quantification methods evaluated (8.229 $\mu$atm) and the coverage closest to the nominal 68% / 95% levels (0.686 / 0.930), while retaining point-prediction skill comparable to a deterministic XGBoost baseline (RMSE 18.4 vs. 17.7 $\mu$atm; $R^2$ 0.77 vs. 0.79). These results show that conditional diffusion can provide well-calibrated, observation-level uncertainties while largely preserving point-prediction skill. Because Treeffuser is generative, its samples can support probabilistic analyses of $\Delta p$CO$_2$; extending this to spatially coherent field reconstruction will require modeling dependence across grid cells and evaluation on spatially held-out data.

## WoRD: Diffusion-Based Posterior Inference for Online Goal Recognition

DevFeed: [WoRD: Diffusion-Based Posterior Inference for Online Goal Recognition](<https://devfeed.tech/articles/word-diffusion-based-posterior-inference-for-online-goal-recognition-140129.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23192>)

Author: Almog Anschel, Sarah Keren

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Robot Navigation](<https://devfeed.tech/topics/robot-navigation.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [diffusion-transformers](<https://devfeed.tech/topics/diffusion-transformers.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [bayes](<https://devfeed.tech/tags/bayes.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [denoising](<https://devfeed.tech/tags/denoising.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [inference](<https://devfeed.tech/tags/inference.md>), [motion](<https://devfeed.tech/tags/motion.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [online](<https://devfeed.tech/tags/online.md>), [recognition](<https://devfeed.tech/tags/recognition.md>), [robot](<https://devfeed.tech/tags/robot.md>), [vehicle](<https://devfeed.tech/tags/vehicle.md>)

### AI overview

WoRD infers an agent's goal from partial continuous trajectories by using a learned trajectory prior rather than requiring an explicit dynamics model. For each candidate goal, it runs a warm-started denoising chain, steers samples toward observations with a closed-form log-likelihood gradient, then applies Bayes' rule to form a calibrated goal posterior. In evaluations on simulated robots, pedestrians, and vehicles, the method reportedly preserved multiple plausible goals through ambiguous prefixes, while filter- and classifier-based observers sometimes collapsed to one mode or became overconfident. The supplied text is an abstract and does not provide metric names or detailed results.

### Source excerpt

arXiv:2611.23192v1 Announce Type: new Abstract: Goal recognition is the task of inferring an agent's intended goal from its observed behavior. Most existing work focuses on high-level symbolic recognition, abstracting away the continuous motion through which agents reach their goal. Methods that do reason over continuous trajectories rely on an explicit dynamics model that is hard to acquire and rarely transfers across settings. Neither approach explicitly learns how agents move toward each candidate goal, limiting accuracy on ambiguous trajectory prefixes where such knowledge is most needed. We present WoRD (Waypoint Recognition with Diffusion), which uses a learned trajectory prior to infer the agent's goal from partial observations. At each step, WoRD runs one warm-started denoising chain per candidate goal, steers each chain toward the observed trajectory through a closed-form log-likelihood gradient, and reweights the resulting samples by Bayes' rule into a calibrated posterior over goals. We evaluate WoRD on simulated robot settings, pedestrian trajectories, and vehicle trajectories using four metrics targeting distinct failure modes. Across all three settings, the same inference procedure preserves calibrated multimodal beliefs through ambiguous prefixes where filter- and classifier-based observers collapse to a single mode or grow overconfident.

## Better Calibrated, Better Planned: Uncertainty-Aware Multi-ASV Informative Path Planning for Oil Spill Monitoring

DevFeed: [Better Calibrated, Better Planned: Uncertainty-Aware Multi-ASV Informative Path Planning for Oil Spill Monitoring](<https://devfeed.tech/articles/better-calibrated-better-planned-uncertainty-aware-multi-asv-informative-path-planning-for-oil-spill-monitoring-141675.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21783>)

Author: Alejandro Casado-P\\'erez, Marija Popovi\\'c, Daniel Guti\\'errez-Reina, Sergio L. Toral., Samuel Yanes

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Robot Navigation](<https://devfeed.tech/topics/robot-navigation.md>)

Tags: [autonomous](<https://devfeed.tech/tags/autonomous.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cells](<https://devfeed.tech/tags/cells.md>), [coupling](<https://devfeed.tech/tags/coupling.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ma](<https://devfeed.tech/tags/cs-ma.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [data](<https://devfeed.tech/tags/data.md>), [deep](<https://devfeed.tech/tags/deep.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [energy](<https://devfeed.tech/tags/energy.md>), [error](<https://devfeed.tech/tags/error.md>), [field](<https://devfeed.tech/tags/field.md>), [fleet](<https://devfeed.tech/tags/fleet.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [information](<https://devfeed.tech/tags/information.md>), [ipp](<https://devfeed.tech/tags/ipp.md>), [learning](<https://devfeed.tech/tags/learning.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [oil](<https://devfeed.tech/tags/oil.md>), [path-planning](<https://devfeed.tech/tags/path-planning.md>), [planning](<https://devfeed.tech/tags/planning.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>)

### AI overview

In physics-based oil spill simulations across 60 maps, the uncertainty model mattered far more than planner choice among deep learning approaches, explaining 80% versus 1% of the variation in final reconstruction error. Evidential Deep Learning (EDL) had the lowest error and highest IoU under all five planners, was best calibrated, and ran in 3.9 ms per step. Using 50 cells per vehicle, EDL outperformed Monte Carlo Dropout and Deep Ensembles using 150. Ensembles grew more confident as their errors increased, while EDL was unaffected. The results suggest that uncertainty-aware multi-ASV planning requires an information model robust to the data generated by its own planner.

### Source excerpt

arXiv:2611.21783v1 Announce Type: cross Abstract: Monitoring oil spills with fleets of Autonomous Surface Vehicles (ASVs) requires mapping a heterogeneous pollutant field from sparse, noisy measurements under tight time and energy budgets. Informative Path Planning (IPP) addresses this problem by coupling a reconstruction model, which infers the field and its uncertainty, with a planner that decides where to sample next. Prior work has focused on planning strategies and has largely relied on Gaussian Processes (GPs), leaving the calibration of the model's uncertainty underexplored. We study how uncertainty calibration conditions multi-vehicle IPP with a fleet of ASVs that coordinates online through a shared uncertainty map. We introduce Evidential Deep Learning (EDL) as an uncertainty-calibrated information model. We benchmark EDL, Monte Carlo Dropout and Deep Ensembles against a GP in closed loop with five planners on physics-based oil spill simulations. Across 60 maps, the choice of model dominates the outcome, explaining 80\% of the variance of the final reconstruction error, against 1\% for the planner among the deep models. EDL achieves the lowest error and the highest IoU under every planner, is the best calibrated model and runs at 3.9 ms per step. With 50 cells per vehicle it reconstructs the spill better than MC Dropout and Deep Ensembles with 150. Deep Ensembles become more confident as their error grows, whereas EDL is unaffected. Robustness of the information model to the data its own planner collects is therefore a requirement of uncertainty-aware IPP.

## Beveridge: Open Weight System One Decision Models with Calibrated Probabilities

DevFeed: [Beveridge: Open Weight System One Decision Models with Calibrated Probabilities](<https://devfeed.tech/articles/beveridge-open-weight-system-one-decision-models-with-calibrated-probabilities-138772.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18655>)

Author: Sathiskumar Jothi

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [apache-2-0](<https://devfeed.tech/tags/apache-2-0.md>), [attention](<https://devfeed.tech/tags/attention.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [browser](<https://devfeed.tech/tags/browser.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decision-models](<https://devfeed.tech/tags/decision-models.md>), [engine](<https://devfeed.tech/tags/engine.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [lora](<https://devfeed.tech/tags/lora.md>), [nvidia-h200](<https://devfeed.tech/tags/nvidia-h200.md>), [probability](<https://devfeed.tech/tags/probability.md>)

### AI overview

Beveridge is an open-weight family for document decisions: it answers up to 15 typed questions in one pass and returns calibrated probabilities without generating text. Beveridge-Large-C adds a compact decision network to ModernBERT-large, using typed number and date features, rule-engine facts, evidence attention, and a recursive core; the Pro variants adapt Gemma 4 26B-A4B, with a Web variant for browser next steps. On the authors' operations-document benchmark, the 398M-parameter model scored 93.0%, versus 91.1% for the best of five tested general LLMs and 87.9% for a commercial System One API. Its reported ECE was 0.029, enabling automatic handling of 76% of decisions at 95% accuracy; none of the tested general LLMs could hand over any at that threshold. Reported latency was 4.9 ms on an NVIDIA H200 and 60 ms on an Apple M4 Pro CPU. Pro-C-Web scored 98.2% on next-step decisions for unseen website types, up from 49.4%, while maintaining document accuracy. Weights and inference code are released under Apache 2.0.

### Source excerpt

arXiv:2611.18655v1 Announce Type: new Abstract: Many production systems need a model to make a narrow decision about a document (route a ticket, check a limit, refuse a request, pick the next step of a browser agent) rather than to write text about it. We present Beveridge, a family of open weight System One decision models that take a document and up to 15 typed questions (yes/no, choice among described options, or a level on an ordered scale) and return, in one forward pass without generating text, an answer with a calibrated probability for every option. Beveridge-Large-C (398M parameters) adds a 3.5M parameter decision network to ModernBERT-large: typed number and date features in the input, exact facts from a rule engine, question-to-document evidence attention, a three-step recursive core with learned halting and an exact yes/no path. Beveridge-Pro-C adapts Gemma 4 26B-A4B with LoRA to score option letters, and Beveridge-Pro-C-Web extends it to the next step of a browser agent. On held-out tests the 0.4B model reaches 93.0% on our operations-document benchmark against 91.1% for the best of five general LLMs (31B-675B) and 87.9% for a commercial System One API, and, because its confidences are calibrated (ECE 0.029 on that benchmark), it can hand over 76% of decisions automatically at 95% accuracy, where no general LLM we tested can hand over any. It answers in 4.9 ms per decision on one NVIDIA H200 GPU and 60 ms on the 14 core CPU of an Apple M4 Pro laptop. Beveridge-Pro-C-Web raises next-step accuracy on web pages from unseen site types from 49.4% to 98.2% and completes live browsing tasks without losing document accuracy. We describe the models, the training recipes and an evaluation on 19 held-out test sets against a commercial System One API and five general LLMs, including where the models are weak. Weights and inference code are released under Apache 2.0.

## CI Failure Triage Needs Evidence Before It Needs a Classifier

DevFeed: [CI Failure Triage Needs Evidence Before It Needs a Classifier](<https://devfeed.tech/articles/ci-failure-triage-needs-evidence-before-it-needs-a-classifier-133413.md>)

Original publisher: [Read original article](<https://hackernoon.com/ci-failure-triage-needs-evidence-before-it-needs-a-classifier?source=rss>)

Author: Saqib Jan

Published: 2026-11-13T22:05:11Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Software Testing](<https://devfeed.tech/topics/software-testing.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [intermittent failures](<https://devfeed.tech/topics/intermittent-failures.md>), [human review](<https://devfeed.tech/topics/human-review.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-errors](<https://devfeed.tech/tags/agent-errors.md>), [capture](<https://devfeed.tech/tags/capture.md>), [ci](<https://devfeed.tech/tags/ci.md>), [ci-failure](<https://devfeed.tech/tags/ci-failure.md>), [ci-pipeline](<https://devfeed.tech/tags/ci-pipeline.md>), [cicd](<https://devfeed.tech/tags/cicd.md>), [defects](<https://devfeed.tech/tags/defects.md>), [devops](<https://devfeed.tech/tags/devops.md>), [environment](<https://devfeed.tech/tags/environment.md>), [errors](<https://devfeed.tech/tags/errors.md>), [failure](<https://devfeed.tech/tags/failure.md>), [failures](<https://devfeed.tech/tags/failures.md>), [flaky-tests](<https://devfeed.tech/tags/flaky-tests.md>), [jev](<https://devfeed.tech/tags/jev.md>), [like](<https://devfeed.tech/tags/like.md>), [model](<https://devfeed.tech/tags/model.md>), [needs](<https://devfeed.tech/tags/needs.md>), [noise](<https://devfeed.tech/tags/noise.md>), [product](<https://devfeed.tech/tags/product.md>), [software-testing](<https://devfeed.tech/tags/software-testing.md>), [sort](<https://devfeed.tech/tags/sort.md>), [test](<https://devfeed.tech/tags/test.md>), [triage](<https://devfeed.tech/tags/triage.md>)

### AI overview

CI failure triage should collect evidence before using a classifier. The source distinguishes product defects (reproduce on the same commit), environment noise (dependency timeouts or 5xx errors), agent errors (changed or inconsistent agent steps in a healthy environment), and test defects (often shared-state or order problems that disappear with one worker). It recommends recording an evidence bundle with commit and model/prompt versions, dependency health, retry results, worker count, and agent-step differences, then routing each label to its responsible team. Run classification in shadow mode against human labels and measure calibration on the team's own failures; automate routing only when confidence is supported by observed accuracy, sending lower-confidence cases to human triage. The article notes that Jev's calibration claims are vendor-reported and that public calibration results are unavailable.

### Source excerpt

Sort CI failures into product defects, environment noise, agent errors and test defects, and capture the evidence before a model like Jev triages them.

## Calibrated Weak Supervision for Post-Harvest Burned-Cropland Mapping Under Label Scarcity

DevFeed: [Calibrated Weak Supervision for Post-Harvest Burned-Cropland Mapping Under Label Scarcity](<https://devfeed.tech/articles/calibrated-weak-supervision-for-post-harvest-burned-cropland-mapping-under-label-scarcity-132837.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.05040>)

Author: Raunak Bhagate, Maitri Polisetty, R I Minu

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [mlp](<https://devfeed.tech/topics/mlp.md>)

Tags: [assessment](<https://devfeed.tech/tags/assessment.md>), [classifiers](<https://devfeed.tech/tags/classifiers.md>), [cohen](<https://devfeed.tech/tags/cohen.md>), [concatenation](<https://devfeed.tech/tags/concatenation.md>), [correlation](<https://devfeed.tech/tags/correlation.md>), [cross-validation](<https://devfeed.tech/tags/cross-validation.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [external](<https://devfeed.tech/tags/external.md>), [features](<https://devfeed.tech/tags/features.md>), [fire](<https://devfeed.tech/tags/fire.md>), [hotspot](<https://devfeed.tech/tags/hotspot.md>), [india](<https://devfeed.tech/tags/india.md>), [lightgbm](<https://devfeed.tech/tags/lightgbm.md>), [linear](<https://devfeed.tech/tags/linear.md>), [mapping](<https://devfeed.tech/tags/mapping.md>), [neural](<https://devfeed.tech/tags/neural.md>), [optical](<https://devfeed.tech/tags/optical.md>), [perceptron](<https://devfeed.tech/tags/perceptron.md>), [pixel](<https://devfeed.tech/tags/pixel.md>), [supervision](<https://devfeed.tech/tags/supervision.md>)

### AI overview

For Punjab burned-cropland mapping under scarce labels, a weak-supervision framework combining Sentinel-2 spectral change, VIIRS fire context, and MODIS MCD64A1 calibration performed better for district-scale burden assessment than for identifying individual burn scars. The best cross-validation configuration was a very-strict pseudo-label recipe with a multilayer perceptron and full optical features; Random Forest, XGBoost, and LightGBM were practically tied. Deployment retained Random Forest with a threshold of 0.60, giving district-level MODIS agreement R² 0.636 and a mapped-to-MODIS burned-area ratio of 1.005, while pixel-level agreement remained modest (F1 0.205, κ 0.093). Higher agreement with pseudo-labels did not prove greater label correctness. Transfer to Haryana was promising, but Punjab cross-year stability was weak; adding Sentinel-1 did not improve the optical baseline. Optical coverage ended before late-season burns, limiting exact scar mapping and stable annual estimates.

### Source excerpt

arXiv:2610.05040v2 Announce Type: replace Abstract: Mapping post-harvest burned cropland is difficult when fires are small and fragmented and reliable labels are scarce. We developed a calibrated weak-supervision framework for Punjab, India, using Sentinel-2 spectral change, VIIRS active-fire context, and MODIS MCD64A1 as a coarse external calibration and agreement reference. Three pseudo-label recipes, five feature representations, and linear, tree-based, boosted, and neural classifiers were evaluated using nested district-held-out cross-validation over three seeds and five folds. NBR and dNBR were excluded from classifier inputs. The best configuration used the very-strict recipe, a multilayer perceptron, and the full optical feature set (mean Cohen's kappa 0.395, AUROC 0.753, F1 0.747, balanced accuracy 0.703); Random Forest, XGBoost, and LightGBM were practically tied. Higher agreement with held-out pseudo-labels did not establish improved label correctness or independent burned-area accuracy. For deployment, a Random Forest with the very-strict recipe and full optical features was retained. An externally calibrated threshold of 0.60 yielded district-level MODIS agreement of R-squared 0.636, a mapped-to-MODIS burned-area ratio of 1.005, and Spearman correlation of 0.779 with district fire counts. Pixel-level MODIS agreement remained modest (F1 0.205, kappa 0.093). Zero-shot transfer to Haryana was promising (mean kappa 0.641), but Punjab cross-year stability was weak, and Sentinel-1/Sentinel-2 feature concatenation did not improve the optical baseline. Optical observations ended before the seasonal fire context, limiting coverage of late burns. The framework supports district-scale burden assessment and hotspot screening, with limited support for exact scar boundaries or temporally stable annual mapping.

## Learning a calibrated decision head from logged bandit feedback

DevFeed: [Learning a calibrated decision head from logged bandit feedback](<https://devfeed.tech/articles/learning-a-calibrated-decision-head-from-logged-bandit-feedback-133011.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15572>)

Author: Ye Yuan

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [lora](<https://devfeed.tech/topics/lora.md>), [machine learning overfitting](<https://devfeed.tech/topics/machine-learning-overfitting.md>)

Tags: [1-5](<https://devfeed.tech/tags/1-5.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [adapters](<https://devfeed.tech/tags/adapters.md>), [all](<https://devfeed.tech/tags/all.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [bandit](<https://devfeed.tech/tags/bandit.md>), [base](<https://devfeed.tech/tags/base.md>), [beats](<https://devfeed.tech/tags/beats.md>), [channel](<https://devfeed.tech/tags/channel.md>), [code](<https://devfeed.tech/tags/code.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [decision](<https://devfeed.tech/tags/decision.md>), [emotion](<https://devfeed.tech/tags/emotion.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [error](<https://devfeed.tech/tags/error.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [fine](<https://devfeed.tech/tags/fine.md>), [learning](<https://devfeed.tech/tags/learning.md>)

### AI overview

The abstract reports that Chosen-Action Proper Scoring (CAPS), which applies a proper score to the probability of the action actually chosen, outperforms fine-tuning on confirmed answers alone using logged bandit feedback. On Banking77 it gains 2.6 accuracy points, lowers ECE by .053 and reduces selective risk, though the comparison uses three CAPS seeds versus one baseline seed; on emotion data it gains 21.7 points in one seed. CAPS remains 1.5 points below full-label training at 0.6B and .6 points below at 1.7B for N=5000, each compared with one full-label seed. Propensity weights add nothing on the reported log. The authors also find that using the sampled action's Brier or log score as a REINFORCE reward can collapse learning; pathwise loss gradients and rewards r or r−p avoid that failure. With full labels, none of four bandit arms beats cross-entropy plus temperature scaling on ECE across four unseen schemas. For feedback flipped at known rates, backward correction recovers most clean calibration, and held-out logged outcomes support early stopping to limit memorization of noisy draws.

### Source excerpt

arXiv:2611.15572v1 Announce Type: new Abstract: A decision head reads a state and a list of options and returns a probability per option in one forward pass. Once deployed, it usually observes only whether the option it chose was right: logged bandit feedback. We adapt such a head, LoRA on Qwen3-0.6B-Base and 1.7B-Base, from that feedback and report three results. First, the Brier or log score of the sampled action's own probability is not a usable REINFORCE reward: every wrong action earns at least gold's reward, with a mean baseline the lowest-probability wrong action gets the largest advantage, and at two options the signal is zero; in one run it cut accuracy on ordered-scale questions from .636 to .111. Scoring the probability as a loss by the pathwise gradient avoids the collapse, as do the REINFORCE rewards r and r minus p. Second, with full labels, none of four tested bandit arms beat cross-entropy plus one temperature on mean expected calibration error (ECE) over four unseen label schemas. Third, from logged feedback alone, Chosen-Action Proper Scoring (CAPS), the direct method with a proper score on the chosen action's probability, beats fine-tuning on confirmed answers only, which discards negatives: on Banking77 by 2.6 accuracy points, .053 lower ECE and lower selective risk (three seeds against one baseline seed), and by 21.7 points on one seed on emotion data with an often-wrong logger. At N 5000, CAPS stays 1.5 points below full-label training at 0.6B and .6 points at 1.7B against one full-label seed, and propensity weights add nothing on this log. Under feedback flipped at a known rate, one seed per rate, CAPS's confidence tracks the noisy channel on average, backward correction recovers most of the clean calibration, and early stopping on held-out logged outcomes prevented memorising noisy draws in all three runs. Code, adapters, logs and per-record results are public.

## Calibeating Prediction-Powered Inference

DevFeed: [Calibeating Prediction-Powered Inference](<https://devfeed.tech/articles/calibeating-prediction-powered-inference-131903.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2604.21260>)

Author: Lars van der Laan, Mark Van Der Laan

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [linear-regression](<https://devfeed.tech/topics/linear-regression.md>)

Tags: [aipw](<https://devfeed.tech/tags/aipw.md>), [black-box](<https://devfeed.tech/tags/black-box.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [econ-em](<https://devfeed.tech/tags/econ-em.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [estimator](<https://devfeed.tech/tags/estimator.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [inference](<https://devfeed.tech/tags/inference.md>), [linear](<https://devfeed.tech/tags/linear.md>), [model](<https://devfeed.tech/tags/model.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [original](<https://devfeed.tech/tags/original.md>), [package](<https://devfeed.tech/tags/package.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [q-bio-qm](<https://devfeed.tech/tags/q-bio-qm.md>), [regression](<https://devfeed.tech/tags/regression.md>), [sample](<https://devfeed.tech/tags/sample.md>), [scale](<https://devfeed.tech/tags/scale.md>), [score](<https://devfeed.tech/tags/score.md>), [simulations](<https://devfeed.tech/tags/simulations.md>), [standard](<https://devfeed.tech/tags/standard.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [study](<https://devfeed.tech/tags/study.md>), [variance](<https://devfeed.tech/tags/variance.md>)

### AI overview

Calibrated Prediction-Powered Inference post-hoc calibrates a black-box prediction score on the labeled sample before using it for semisupervised mean estimation, requiring no model retraining. The paper studies linear and isotonic calibration: isotonic calibration has first-order optimality guarantees, while linear calibration is first-order equivalent to PPI++. It describes the original PPI estimator as a special case of AIPW that can be inefficient when predictions are accurate, and PPI++ as AIPW with empirical efficiency maximization. In simulations and real-data experiments, calibrated estimators often outperform PPI and are competitive with or outperform AIPW and PPI++. These are abstract-level claims; the excerpt gives no experiment-specific results.

### Source excerpt

arXiv:2604.21260v2 Announce Type: replace-cross Abstract: We study semisupervised mean estimation with a small labeled sample, a large unlabeled sample, and a black-box prediction model whose output may be miscalibrated. A standard approach in this setting is augmented inverse-probability weighting (AIPW) [Robins et al., 1994], which protects against prediction-model misspecification but may be inefficient when the prediction score is poorly aligned with the outcome scale. We introduce Calibrated Prediction-Powered Inference, which post-hoc calibrates the prediction score on the labeled sample before using it for semisupervised estimation. This simple step requires no retraining and can calibeat the original score, improving it both as a predictor of the outcome and as a regression adjustment for semisupervised inference. We study both linear and isotonic calibration. For isotonic calibration, we establish first-order optimality guarantees: isotonic post-processing can improve predictive accuracy and estimator efficiency relative to the original score and simpler post-processing rules, while replacing the fitted isotonic score by its conditional-mean oracle yields no additional first-order variance reduction. For linear calibration, we show first-order equivalence to PPI++. We also clarify the relationship between existing estimators, showing that the original PPI estimator is a special case of AIPW and can be inefficient when the prediction model is accurate, while PPI++ is AIPW with empirical efficiency maximization [Rubin and van der Laan, 2008]. In simulations and real-data experiments, our calibrated estimators often outperform PPI and are competitive with, or outperform, AIPW and PPI++. We provide an accompanying Python package, ppi_aipw, at https://larsvanderlaan.github.io/ppi-aipw/.

## Conformal Risk Certificates for Portfolio Optimization Under Uncertainty: A Gated Reinforcement-Learning Approach

DevFeed: [Conformal Risk Certificates for Portfolio Optimization Under Uncertainty: A Gated Reinforcement-Learning Approach](<https://devfeed.tech/articles/conformal-risk-certificates-for-portfolio-optimization-under-uncertainty-a-gated-reinforcement-learning-approach-132905.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.14699>)

Author: Mohammad Zarei, Amirhossein MohammadpourAzari, Moslem Habibi

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [audit](<https://devfeed.tech/topics/audit.md>)

Tags: [audit](<https://devfeed.tech/tags/audit.md>), [audit-trail](<https://devfeed.tech/tags/audit-trail.md>), [certificate](<https://devfeed.tech/tags/certificate.md>), [certificates](<https://devfeed.tech/tags/certificates.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [disclosures](<https://devfeed.tech/tags/disclosures.md>), [earnings](<https://devfeed.tech/tags/earnings.md>), [execution](<https://devfeed.tech/tags/execution.md>), [factor](<https://devfeed.tech/tags/factor.md>), [gate](<https://devfeed.tech/tags/gate.md>), [human-oversight](<https://devfeed.tech/tags/human-oversight.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [math-oc](<https://devfeed.tech/tags/math-oc.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [portfolio](<https://devfeed.tech/tags/portfolio.md>), [ppo](<https://devfeed.tech/tags/ppo.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [premium](<https://devfeed.tech/tags/premium.md>), [q-fin-pm](<https://devfeed.tech/tags/q-fin-pm.md>), [q-fin-rm](<https://devfeed.tech/tags/q-fin-rm.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [risk](<https://devfeed.tech/tags/risk.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>)

### AI overview

Gated Conformal Semantic Risk Control (Gated-CSRC) checks PPO portfolio proposals against a calibrated worst-case semantic loss budget before execution. It converts timestamped disclosures into validated risk states across macro, liquidity, earnings, and sector channels, calibrates uncertainty sets using rolling finite-sample predictions, then uses an asymmetric second-order cone gate that intervenes only when a proposal exceeds the budget. In walk-forward tests on nine U.S. Select Sector SPDR ETFs, raw PPO returned 12.7% annually with a 4.05% semantic violation rate; Gated-CSRC had no observed out-of-sample breaches and returned 11.7%. The authors describe the 100-basis-point return reduction as smaller than the yield loss from unconditional hard projection. Exact McNemar testing found a compliance gain (p < 10^-15), while Loughran-McDonald lexicon and time-shuffled text controls did not reproduce the certificate. These results are reported for the stated test setting.

### Source excerpt

arXiv:2611.14699v1 Announce Type: cross Abstract: Reinforcement-learning portfolio policies can optimize allocations from prices, covariances, and prior positions, yet they lack a calibrated, replayable mechanism for certifying that a proposed allocation respects a bounded risk envelope before execution. Regulators increasingly expect automated portfolio systems to provide exactly this kind of pre-trade audit trail, interpretable risk limit, and human-oversight mechanism. Gated Conformal Semantic Risk Control addresses this gap by separating return-seeking allocation from text-conditioned downside-risk verification. An instruction-tuned language model maps timestamped disclosures into schema-validated semantic risk states over macro, liquidity, earnings, and sector channels. These states induce correlated ellipsoidal factor-shock uncertainty sets whose conformal radii are calibrated by rolling finite-sample prediction over localized semantic nonconformity buckets. An asymmetric second-order cone gate then checks each raw PPO proposal against a certified worst-case semantic loss budget. The gate stays inactive for compliant proposals and intervenes only when that budget is exceeded. In walk-forward out-of-sample tests on nine liquid U.S. Select Sector SPDR ETFs, Raw PPO earns a 12.7% annual return while incurring a 4.05% semantic violation rate. Gated-CSRC removes all observed out-of-sample semantic limit breaches while retaining an 11.7% annual return. This 100-basis-point auditable safety premium is smaller than the yield loss imposed by unconditional hard projection, and exact McNemar testing confirms the compliance gain at $p < 10^{-15}$. Loughran-McDonald lexicon controls and time-shuffled text controls fail to reproduce this certificate.

## Calibrated selective prediction using deep ensembles for thyroid nodule ultrasound classification under dataset shift

DevFeed: [Calibrated selective prediction using deep ensembles for thyroid nodule ultrasound classification under dataset shift](<https://devfeed.tech/articles/calibrated-selective-prediction-using-deep-ensembles-for-thyroid-nodule-ultrasound-classification-under-dataset-shift-131950.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.12075>)

Author: Md. Sadibul Hasan Sadib, Md. Mohayminul Mukit, Rahmatul Kabir Rasel Sarker, Tahmid Alam Tamim, Md. Nazmul Hasan, Md. Monir Hossain Shimul

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [Image processing](<https://devfeed.tech/topics/image-processing.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>)

Tags: [classification](<https://devfeed.tech/tags/classification.md>), [cross-validation](<https://devfeed.tech/tags/cross-validation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [deep](<https://devfeed.tech/tags/deep.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [eess-iv](<https://devfeed.tech/tags/eess-iv.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fna](<https://devfeed.tech/tags/fna.md>), [framework](<https://devfeed.tech/tags/framework.md>), [image](<https://devfeed.tech/tags/image.md>), [msp](<https://devfeed.tech/tags/msp.md>), [performance](<https://devfeed.tech/tags/performance.md>), [policy](<https://devfeed.tech/tags/policy.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [risk](<https://devfeed.tech/tags/risk.md>), [roc](<https://devfeed.tech/tags/roc.md>), [selection](<https://devfeed.tech/tags/selection.md>), [shift](<https://devfeed.tech/tags/shift.md>), [site](<https://devfeed.tech/tags/site.md>), [test](<https://devfeed.tech/tags/test.md>), [ultrasound](<https://devfeed.tech/tags/ultrasound.md>)

### AI overview

A five-member ConvNeXt-Tiny ensemble with uncertainty-aware selective triage performed well on TN5000 but showed substantial dataset-shift degradation on TN3K, limiting its transportability. On TN5000, out-of-fold AUC-ROC was 0.9377 and expected calibration error 0.0138. The primary mutual-information policy assigned 8.86% of images a No-FNA suggestion, 60.00% an FNA recommendation, and 31.14% for radiologist review; 0.39% of malignant images received No-FNA. Predictive variance and maximum-softmax probability ranked errors better than mutual information, while no uncertainty measure optimized every triage outcome. Applied to TN3K without recalibration, AUC-ROC fell to 0.7870, calibration error rose to 0.1899, and 83.7% of images were sent for review. The authors call for multicentre validation, site-specific recalibration, threshold validation, and prospective evaluation before clinical use.

### Source excerpt

arXiv:2607.12075v3 Announce Type: replace-cross Abstract: Background: Deep learning can classify thyroid nodules on ultrasound, but discrimination alone does not establish clinical reliability. We developed a calibrated deep-ensemble framework with uncertainty-aware selective prediction and conservative image-based triage. Methods: Nested five-fold cross-validation with three-fold inner architecture selection chose ConvNeXt-Tiny with a squeeze-and-excitation block, used to build a deterministic five-member ensemble. Training, early-stopping, calibration, and threshold subsets were separated within each fold; outer-test data remained isolated. Mutual information (MI), predictive variance, and maximum-softmax-probability (MSP) uncertainty were compared by risk-coverage and triage analyses, and predictive values across assumed malignancy prevalences of 5%-70%. The frozen TN5000 ensemble was applied to TN3K without recalibration. Results: On TN5000, pooled out-of-fold AUC-ROC was 0.9377, average precision 0.9708, expected calibration error 0.0138, and Brier score 0.0827. Under the primary MI-guided policy, 8.86% of images received a No-FNA suggestion, 60.00% an FNA recommendation, and 31.14% radiologist review; of 3574 malignant images, 14 (0.39%) received No-FNA and 708 (19.81%) review. Predictive variance and MSP ranked errors better than MI (AURC, AUGRC); no measure optimized all triage outcomes. Across prevalences of 5%-70%, NPV ranged 99.95%-97.93% and PPV 29.28%-94.75%. On TN3K, AUC-ROC was 0.7870, average precision 0.7254, expected calibration error 0.1899, and radiologist-review rate 83.7%. Conclusion: The ensemble showed strong internal discrimination and calibration, but performance deteriorated substantially on TN3K, indicating limited transportability under dataset shift. Multicentre validation, site-specific recalibration, threshold validation, and prospective evaluation are needed before clinical use.

## A Matching Lower Bound for Online Smooth Calibration

DevFeed: [A Matching Lower Bound for Online Smooth Calibration](<https://devfeed.tech/articles/a-matching-lower-bound-for-online-smooth-calibration-133081.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16097>)

Author: Kaixuan Ji, Quanquan Gu

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [adversary](<https://devfeed.tech/tags/adversary.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [astra](<https://devfeed.tech/tags/astra.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [design](<https://devfeed.tech/tags/design.md>), [error](<https://devfeed.tech/tags/error.md>), [gap](<https://devfeed.tech/tags/gap.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [gpt-6](<https://devfeed.tech/tags/gpt-6.md>), [gpt-6-astra](<https://devfeed.tech/tags/gpt-6-astra.md>), [guidance](<https://devfeed.tech/tags/guidance.md>), [ideas](<https://devfeed.tech/tags/ideas.md>), [lower-bound](<https://devfeed.tech/tags/lower-bound.md>), [lower-bounds](<https://devfeed.tech/tags/lower-bounds.md>), [matching](<https://devfeed.tech/tags/matching.md>), [minimax](<https://devfeed.tech/tags/minimax.md>), [number](<https://devfeed.tech/tags/number.md>), [online](<https://devfeed.tech/tags/online.md>), [paper](<https://devfeed.tech/tags/paper.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

For sequential binary prediction against adaptive adversaries, the paper establishes a matching minimax rate of Θ(√T) for smooth calibration error, closing the prior gap between Ω(T^{1/3}) and O(√T). Its constructed adversary makes the outcome mean track the forecaster's cumulative residual, making it costly to compensate for past residuals and forcing every forecaster to incur expected error Ω(√T). The result also yields Ω(√T) lower bounds for calibration distance and lower calibration distance. The evidence provided is an abstract, so it does not describe the proof in full.

### Source excerpt

arXiv:2611.16097v1 Announce Type: new Abstract: We study smooth calibration error in sequential binary prediction, which measures how far a sequence of probabilistic forecasts is from being calibrated and, unlike the classical expected calibration error, is robust to small perturbations of the forecasts. Against adaptive adversaries, the minimax smooth calibration error is known to lie between $\Omega(T^{1/3})$ and $O(\sqrt{T})$ (Qiao and Zheng, 2024), where $T$ is the number of rounds. In this paper, we close this gap by constructing a new adversary whose outcome mean moves with the forecaster's cumulative residual. This design makes compensating for past residuals costly, and forces every forecaster to incur an expected smooth calibration error of $\Omega(\sqrt{T})$. This lower bound matches the known upper bound up to universal constants, and further implies $\Omega(\sqrt{T})$ lower bounds for calibration distance and lower calibration distance. Overall, our results settle the minimax rates of these calibration measures in sequential prediction. The proof ideas are partially developed by GPT-6 Astra under the authors' guidance.

## Training Calibrated Conditional Generative Models with a Sample-Only Coverage Loss

DevFeed: [Training Calibrated Conditional Generative Models with a Sample-Only Coverage Loss](<https://devfeed.tech/articles/training-calibrated-conditional-generative-models-with-a-sample-only-coverage-loss-133043.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15829>)

Author: Benjamin Remy, Raphael Rossellini, Chihway Chang, Rebecca Willett

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [generative](<https://devfeed.tech/tags/generative.md>), [generative-models](<https://devfeed.tech/tags/generative-models.md>), [imaging](<https://devfeed.tech/tags/imaging.md>), [sample](<https://devfeed.tech/tags/sample.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [training](<https://devfeed.tech/tags/training.md>), [uncertainty-quantification](<https://devfeed.tech/tags/uncertainty-quantification.md>)

### AI overview

The paper introduces a differentiable coverage loss for calibrating conditional generative models using samples alone, without requiring tractable densities or architectural changes. The authors prove the loss is a strict divergence: it vanishes only when the model matches the true posterior. On simulation-based inference benchmarks, it matches a density-based coverage loss without evaluating densities; on high-dimensional imaging inverse problems, it improves posterior calibration over scoring-rule regularizers while changing point-estimate reconstruction quality little.

### Source excerpt

arXiv:2611.15829v1 Announce Type: new Abstract: Conditional generative models have become the tool of choice for sampling posteriors in scientific inverse problems, where we seek to sample images or signals conditioned on a set of indirect and noisy measurements. Yet they are often overconfident, which is a critical failure in scientific applications where accurate uncertainty quantification is essential. Existing regularizers fall short in complementary ways: differentiable coverage objectives require the model's density and do not identify the posterior, while strictly proper scoring rules such as the energy score need only samples but do not directly enforce coverage. We close this gap by introducing a differentiable loss function based on the expected coverage probability, estimated from samples alone. It applies to any conditional generative model, with no need for tractable densities or architectural changes. We prove that it is a strict divergence, vanishing if and only if the model equals the true posterior. On simulation-based inference benchmarks, it matches a density-based coverage loss without evaluating the density. On high-dimensional imaging inverse problems, it yields better-calibrated posteriors than scoring-rule regularizers, with little change in the reconstruction quality of point estimates.

## Learning calibrated probabilistic forecasts using Koopman mode decomposition and the continuous ranked probability score

DevFeed: [Learning calibrated probabilistic forecasts using Koopman mode decomposition and the continuous ranked probability score](<https://devfeed.tech/articles/learning-calibrated-probabilistic-forecasts-using-koopman-mode-decomposition-and-the-continuous-ranked-probability-score-125347.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11257>)

Author: Benjamin M. Kent, Richard E. Turner

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [boundary](<https://devfeed.tech/tags/boundary.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distributions](<https://devfeed.tech/tags/distributions.md>), [eigenvalues](<https://devfeed.tech/tags/eigenvalues.md>), [fluid-dynamics](<https://devfeed.tech/tags/fluid-dynamics.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [large](<https://devfeed.tech/tags/large.md>), [learning](<https://devfeed.tech/tags/learning.md>), [machine](<https://devfeed.tech/tags/machine.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [math-ds](<https://devfeed.tech/tags/math-ds.md>), [noise](<https://devfeed.tech/tags/noise.md>), [probabilistic](<https://devfeed.tech/tags/probabilistic.md>), [robustness](<https://devfeed.tech/tags/robustness.md>), [small](<https://devfeed.tech/tags/small.md>), [systems](<https://devfeed.tech/tags/systems.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [training](<https://devfeed.tech/tags/training.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>), [weather](<https://devfeed.tech/tags/weather.md>), [weather-prediction](<https://devfeed.tech/tags/weather-prediction.md>)

### AI overview

The method combines probabilistic machine learning with Koopman mode decomposition, learns observation, dynamics, and reconstruction operators, and generates predictive distributions by propagating injected noise samples. Training with the continuous ranked probability score targets calibrated, accurate forecasts. Tests on two nonlinear ODEs and two PDEs, including vortex shedding and cavity flow, report identification of known Koopman eigenfunctions and spectra, and multimodal forecasts for a trajectory on a boundary between basins of attraction. Ensemble spread was close to calibrated over the training horizon in several experiments, but extended rollouts and noisy data could yield under- or over-dispersed forecasts. In low-noise experiments, accurate rollouts generally reached at least twice the training horizon and up to twenty times in some cases. Over the training horizon, approximation quality was comparable to a deep ensemble at lower computational cost, with greater robustness in identifying Koopman eigenvalues.

### Source excerpt

arXiv:2611.11257v1 Announce Type: cross Abstract: Well-calibrated forecasting for non-linear dynamical systems is essential in science and engineering applications such as fluid dynamics and weather prediction. We present a forecasting method combining probabilistic machine learning with the structured linear dynamics of Koopman mode decomposition. Within this framework, the observation, linear dynamics, and state reconstruction operators are constructed using standard machine learning architectures, producing predictive distributions via the pushforward of injected noise samples. The model is trained using a proper scoring rule, the continuous ranked probability score, to directly target calibrated, accurate forecasts. The approach is validated on two non-linear ordinary differential equation examples and two partial differential equation examples, vortex shedding and cavity flow, using both large and small, noisy datasets. Results demonstrate that known Koopman eigenfunctions and spectra can be identified and used for accurate forecasting. The model can also produce multi-modal predictive distributions, demonstrated for a trajectory initialised on the boundary between two basins of attraction. Uncertainty is captured by the sample ensemble spread, with spread--skill ratios close to one over the training horizon in several experiments, although for extended rollouts and noisy datasets, the ensemble may become under- or over-dispersed. In low-noise experiments, temporal rollout is generally accurate for at least twice the training horizon, and up to twenty times in some cases. Approximation quality is comparable to a deep ensemble approximation over the training horizon, but at a reduced computational cost and with better robustness in identifying Koopman eigenvalues.

## Calibrated Uncertainty Quantification on TDC ADMET: What Can Be Resolved Across 22 Datasets?

DevFeed: [Calibrated Uncertainty Quantification on TDC ADMET: What Can Be Resolved Across 22 Datasets?](<https://devfeed.tech/articles/calibrated-uncertainty-quantification-on-tdc-admet-what-can-be-resolved-across-22-datasets-123401.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09373>)

Author: Laurits Fredsgaard, Michael Riis Andersen, Mikkel N. Schmidt

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [Predictive AI](<https://devfeed.tech/topics/predictive-ai.md>), [generalization in machine learning](<https://devfeed.tech/topics/generalization-in-machine-learning.md>)

Tags: [22](<https://devfeed.tech/tags/22.md>), [absorption](<https://devfeed.tech/tags/absorption.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [also](<https://devfeed.tech/tags/also.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [best](<https://devfeed.tech/tags/best.md>), [but](<https://devfeed.tech/tags/but.md>), [calibrated-uncertainty](<https://devfeed.tech/tags/calibrated-uncertainty.md>), [can](<https://devfeed.tech/tags/can.md>), [compare](<https://devfeed.tech/tags/compare.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cross-validation](<https://devfeed.tech/tags/cross-validation.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [data-commons](<https://devfeed.tech/tags/data-commons.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [resolved](<https://devfeed.tech/tags/resolved.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>), [uncertainty-quantification](<https://devfeed.tech/tags/uncertainty-quantification.md>)

### AI overview

Across 22 TDC ADMET datasets, ensemble MC dropout has the best average rank for predictive performance, Brier score, and 90% interval score, but the benchmark does not establish it as a clear winner: other ensemble pipelines are not separated from it on performance, and LightGBM is also not ruled out for the two uncertainty scores. LightGBM wins the most individual datasets for performance and Brier score, while performing poorly on several endpoints. The study compares eleven calibrated pipelines using nested five-fold cross-validation, ten chemical partitions, and five model seeds; smaller evaluations can identify a different leader and miss the broader comparison. The findings support using LightGBM and an ensemble neural model as complementary baselines, and checking comparative claims across multiple chemical partitions with statistical analyses suited to the question.

### Source excerpt

arXiv:2611.09373v1 Announce Type: new Abstract: Absorption, distribution, metabolism, excretion, and toxicity (ADMET) models often rely on small, heterogeneous datasets, yet their predictions may determine which compounds proceed to experimental testing. Useful evaluations must therefore assess both predictive performance and the quality of reported probabilities or intervals. We ask which common uncertainty-quantification methods perform best on the 22 Therapeutics Data Commons (TDC) ADMET datasets and what this benchmark can resolve. We compare eleven calibrated pipelines using nested five-fold cross-validation across ten chemical partitions and five model seeds. Ensemble Monte Carlo (MC) dropout has the best average rank for TDC performance, Brier score, and 90% interval score. For TDC performance, however, it is not separated from the other ensemble pipelines; for Brier and interval score, LightGBM is also not ruled out as best. LightGBM wins more individual datasets than any other method for TDC performance and Brier score, but performs poorly on several endpoints. Smaller evaluations can identify a different leader and often fail to recover the broader comparison. These results make LightGBM and an ensemble neural model complementary baselines. Comparative claims should be checked across multiple chemical partitions using statistical analyses matched to the question being asked.

## ELPAC: Endpoint-Anchored Latent Progression with Stage-Varying Multimodal Coordination

DevFeed: [ELPAC: Endpoint-Anchored Latent Progression with Stage-Varying Multimodal Coordination](<https://devfeed.tech/articles/elpac-endpoint-anchored-latent-progression-with-stage-varying-multimodal-coordination-123274.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.08541>)

Author: Shengxian Ding, Yifei Zhang, Xinyuan Tian, Yuanshuang Guo, Yize Zhao

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [VAE](<https://devfeed.tech/topics/vae.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>)

Tags: [aging](<https://devfeed.tech/tags/aging.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [inference](<https://devfeed.tech/tags/inference.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [stage](<https://devfeed.tech/tags/stage.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [subtype](<https://devfeed.tech/tags/subtype.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [velocity](<https://devfeed.tech/tags/velocity.md>)

### AI overview

ELPAC jointly infers latent stage, progression subtype, and changing multimodal coordination from partially observed cohort data. Its Bayesian framework builds subtype trajectories backward from clinically reliable endpoint groups, using modality-specific monotone directions and optional sequential ordering priors. It then decomposes residual variation around the stage axis into sparse coordination patterns, with stage-resolved variation supporting conditional identifiability. Amortized variational inference enables cohort-scale fitting and single-pass inference for new subjects. In synthetic data and a harmonized ADNI+NACC/SCAN Alzheimer's cohort, the authors report accurate latent-structure recovery, calibrated prediction intervals, and coordination programs that differ by subtype. These are abstract-level claims; the supplied text does not provide detailed metrics.

### Source excerpt

arXiv:2611.08541v1 Announce Type: cross Abstract: Aging cohorts increasingly combine imaging, plasma, and cognitive measurements, but most subjects provide only a partially observed cross-sectional snapshot, leaving latent stage, progression subtype, and stage-varying multimodal coordination to be inferred jointly. Existing methods estimate the cohort-level aging process, model covariance under observed covariates, or fuse modalities for prediction. However, they rarely target the joint problem of endpoint-anchored staging and progression-resolved residual coordination across modalities, even though subject staging, subtype assignment, and multimodal interplay are mutually dependent. We introduce ELPAC, a Bayesian framework with two interlocking constructions. The first integrates subtype trajectories backward from clinically reliable endpoint groups along biologically structured velocity components encoding modality-specific monotone direction and optional sequential ordering priors. The second factorizes residual multimodal variation around the inferred stage axis into sparse coordination axes whose loadings and activations expose which multimodal patterns are operative in each subtype and when they emerge, with stage-resolved variation supplying the auxiliary structure for conditional identifiability. We use a scalable amortized variational inference scheme that enables joint posterior inference at cohort scale with single-pass deployment on new subjects. On synthetic data and a harmonized ADNI+NACC/SCAN Alzheimer's cohort, ELPAC demonstrates accurate latent-structure recovery, well-calibrated prediction intervals, and biologically interpretable multimodal coordination programs that differ by subtype.

## Evaluating Token Probabilities as Confidence Estimators in Human Behavior Simulations: A Case Study in Luxembourgish Next Like Prediction

DevFeed: [Evaluating Token Probabilities as Confidence Estimators in Human Behavior Simulations: A Case Study in Luxembourgish Next Like Prediction](<https://devfeed.tech/articles/evaluating-token-probabilities-as-confidence-estimators-in-human-behavior-simulations-a-case-study-in-luxembourgish-next-like-prediction-120243.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07735>)

Author: Nils Schwager, Simon M\\"unker, Alistair Plum, Achim Rettinger

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [agents](<https://devfeed.tech/tags/agents.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [case-study](<https://devfeed.tech/tags/case-study.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [data](<https://devfeed.tech/tags/data.md>), [digital](<https://devfeed.tech/tags/digital.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [learning](<https://devfeed.tech/tags/learning.md>), [like](<https://devfeed.tech/tags/like.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llms](<https://devfeed.tech/tags/llms.md>), [next](<https://devfeed.tech/tags/next.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [simulations](<https://devfeed.tech/tags/simulations.md>), [study](<https://devfeed.tech/tags/study.md>), [token](<https://devfeed.tech/tags/token.md>)

### AI overview

In Luxembourgish RTL.lu Next Like Prediction, Ministral 3 models using in-context learning were overconfident and scored below a simple statistical baseline on both accuracy and calibration. Accuracy and confidence calibration diverged: moving from 3B to 8B improved accuracy but worsened calibration, while longer behavioral histories improved accuracy as confidence declined. Task-specific supervised fine-tuning reduced Expected Calibration Error from 0.2 to 0.006 and raised accuracy from 0.536 to 0.758. The case study therefore supports evaluating confidence calibration alongside accuracy when assessing LLM simulations of behavior on noisy digital traces.

### Source excerpt

arXiv:2611.07735v1 Announce Type: cross Abstract: As large language models (LLMs) are increasingly used to simulate human behavior, their operational validity is typically assessed through accuracy alone, implicitly treating human behavior as deterministic. We argue that on noisy digital trace data, a model's performance depends not only on predicting the correct outcome but on its calibration. We present a case study testing whether the token probabilities of open-weight LLMs can serve as confidence estimates in a Next Like Prediction task: predicting how a user of the Luxembourgish news platform RTL.lu rates a comment, conditioned on their prior interactions. Evaluating Ministral 3 (3B, 8B), we find that, with in-context learning only, the tested models are overconfident and are outperformed on accuracy and calibration by a simple statistical baseline. Accuracy and calibration do not move together: scaling from 3B to 8B improves accuracy while degrading calibration, and longer behavioral histories raise accuracy while confidence declines. Task-specific supervised fine-tuning acts as an effective calibration intervention, reducing Expected Calibration Error from 0.2 to 0.006 while improving accuracy from 0.536 to 0.758. We conclude that confidence estimation should be evaluated alongside accuracy to improve validity when using LLM-based agents.

## Two-stage residual learning for calibrated uncertainty in solar irradiance forecasts

DevFeed: [Two-stage residual learning for calibrated uncertainty in solar irradiance forecasts](<https://devfeed.tech/articles/two-stage-residual-learning-for-calibrated-uncertainty-in-solar-irradiance-forecasts-120614.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07444>)

Author: RR Flanagan, E Kasha, PW Talbot, AS Epiny, MR Deinert

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Predictive AI](<https://devfeed.tech/topics/predictive-ai.md>)

Tags: [calibrated-uncertainty](<https://devfeed.tech/tags/calibrated-uncertainty.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-sy](<https://devfeed.tech/tags/cs-sy.md>), [eess-sy](<https://devfeed.tech/tags/eess-sy.md>), [irradiance](<https://devfeed.tech/tags/irradiance.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [solar](<https://devfeed.tech/tags/solar.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>)

### AI overview

The method estimates solar irradiance uncertainty by separating forecast error from expected irradiance: one LSTM predicts expected global horizontal irradiance, and a second models the forecast-error distribution, which is combined with the mean to produce forecasts and uncertainty. Evaluated on historical National Solar Radiation Database weather data using a probability integral transform, it reduced the gap between predicted and historical standard deviation by 34.6% for a standalone LSTM and 56.5% for an LSTM coupled with Gaussian process regression, the most reliable tested configuration. The authors say this captures total forecast error, whereas dropout-based approaches estimate only the model contribution.

### Source excerpt

arXiv:2611.07444v1 Announce Type: new Abstract: Accurate forecasting of renewable generation requires not only point predictions but also reliable estimates of the uncertainty attached to them, particularly at extended horizons. Machine learning models are increasingly used to predict solar output, but they typically report a single value and capture the associated uncertainty poorly. The standard remedy is to train a model to predict the parameters of an assumed probability distribution directly. Here we develop an alternative two-stage method, motivated by synthetic solar signal generation, in which the forecast error is modeled separately from the expected irradiance. A first long short-term memory (LSTM) network predicts the expected global horizontal irradiance, and a second network models the distribution of the forecast error, which is added back to the mean to give both the forecast and its uncertainty. Using historical weather data from the National Solar Radiation Database and a probability integral transform evaluation, we show that explicitly modeling the error brings the predicted standard deviation markedly closer to the historical value, reducing the deviation between the two by 34.6 percent for a standalone LSTM and by 56.5 percent when the LSTM is coupled to Gaussian process regression, the most reliable of the configurations tested. Because the method models the total forecast error directly, it captures the full predictive uncertainty that matters for decisions, rather than only the model contribution that dropout-based approaches estimate.

## ForecastCompass: Guiding Agentic Forecasting with Adaptive Factor Memory

DevFeed: [ForecastCompass: Guiding Agentic Forecasting with Adaptive Factor Memory](<https://devfeed.tech/articles/forecastcompass-guiding-agentic-forecasting-with-adaptive-factor-memory-120438.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.30858>)

Author: Yurui Chang, Yongkang Du, Yuanpu Cao, Jinghui Chen, Lu Lin

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Predictive AI](<https://devfeed.tech/topics/predictive-ai.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Agentic AI](<https://devfeed.tech/topics/what-is-agentic-ai.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [agent-memory](<https://devfeed.tech/tags/agent-memory.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [factor](<https://devfeed.tech/tags/factor.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [framework](<https://devfeed.tech/tags/framework.md>), [future](<https://devfeed.tech/tags/future.md>), [gemini](<https://devfeed.tech/tags/gemini.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [gpt-5](<https://devfeed.tech/tags/gpt-5.md>), [handling](<https://devfeed.tech/tags/handling.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [learning](<https://devfeed.tech/tags/learning.md>), [mechanism](<https://devfeed.tech/tags/mechanism.md>), [memory](<https://devfeed.tech/tags/memory.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [retrospective](<https://devfeed.tech/tags/retrospective.md>), [revision](<https://devfeed.tech/tags/revision.md>)

### AI overview

ForecastCompass (FoCo) aims to improve agentic forecasting by turning resolved forecasts into reusable, task-relevant memory. A hierarchical taxonomy guides retrieval; factor memory stores predictive dimensions, while reasoning memory captures probability updating, uncertainty handling and calibration principles. The system uses retrospective analyses to revise these memories over time. Experiments on Prophet Arena and FutureX with GPT-5-mini and Gemini-2.5-Flash report gains in probabilistic accuracy and calibration. The supplied text does not quantify those gains or describe the evaluation setup.

### Source excerpt

arXiv:2605.30858v2 Announce Type: replace Abstract: Agentic forecasting is important for decision-making in dynamic environments, but it remains challenging because agents must reason from incomplete, time-limited evidence and produce calibrated probabilities before outcomes are resolved. Memory provides a natural mechanism for transferring experience from resolved forecasts to future prediction tasks. However, existing agent-memory methods are not tailored to forecasting, as they typically store past interactions, reflections, or factual associations without explicitly representing reusable predictive factors or calibration knowledge. We propose ForecastCompass (FoCo), an adaptive factor-based memory framework for agentic forecasting. FoCo organizes forecasting experience with a hierarchical forecasting-task taxonomy, enabling retrieval task-relevant forecasting knowledge. It maintains two complementary memory components: factor memory, which captures reusable predictive dimensions, and reasoning memory, which encodes probability updating, uncertainty handling, and calibration principles. Using retrospective analyses as learning signals, FoCo iteratively revises memory through a verbalized memory-revision procedure, enabling the agent to accumulate transferable forecasting knowledge over time. Experiments on Prophet Arena and FutureX with GPT-5-mini and Gemini-2.5-Flash show that FoCo improves both probabilistic accuracy and calibration.

[Next page](<https://devfeed.tech/topics/calibrated-uncertainty.md?cursor=WyIyMDI2LTExLTA2VDA1OjAwOjAwKzAwOjAwIiwgIjViY2I4YmZjLTY0MGEtNDNjYy04NzU3LTU2NTc4NzNhZmFiOSJd>)