# Performance metrics in machine learning: a scenario-based guide from the confusion matrix to calibration, regression and uncertainty

DevFeed: [Performance metrics in machine learning: a scenario-based guide from the confusion matrix to calibration, regression and uncertainty](<https://devfeed.tech/articles/performance-metrics-in-machine-learning-a-scenario-based-guide-from-the-confusion-matrix-to-calibration-regression-and-uncertainty-120696.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.08048>)

Author: Mehmet Baygin, Sengul Dogan, Turker Tuncer

Published: 2026-11-06T05:00:00Z

Content type: tutorial

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [class](<https://devfeed.tech/tags/class.md>), [classification](<https://devfeed.tech/tags/classification.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [discrimination](<https://devfeed.tech/tags/discrimination.md>), [guide](<https://devfeed.tech/tags/guide.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [performance-metrics](<https://devfeed.tech/tags/performance-metrics.md>), [precision](<https://devfeed.tech/tags/precision.md>), [probability-calibration](<https://devfeed.tech/tags/probability-calibration.md>), [regression](<https://devfeed.tech/tags/regression.md>), [reporting](<https://devfeed.tech/tags/reporting.md>), [roc-auc](<https://devfeed.tech/tags/roc-auc.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>)

## AI overview

The tutorial shows how prevalence, thresholds, aggregation rules and software conventions can change conclusions drawn from the same predictions. In its controlled scenarios, an always-negative rule reached 0.990 accuracy at 1% prevalence while balanced accuracy was 0.500; with sensitivity and specificity fixed, precision fell from 0.889 to 0.008 as prevalence dropped from 0.5 to 0.001. Three probability variants had identical ROC-AUC (0.825) but calibration slopes of 2.08, 1.04 and 0.52. A 10-participant example reports participant-majority accuracy of 0.700 with a wide 95% interval (0.348-0.933), and another shows how an inappropriate pooled-test-set model can falsely flag fold-averaged scores as inconsistent. The guide organizes 22 confusion-matrix metrics, presents 15 scenarios across classification, calibration, regression and uncertainty, and provides a goal-based metric table and reporting checklist. These scenarios illustrate mechanisms, not their prevalence in published studies; the proposed literature audit and real-data demonstration were not conducted.

## Source excerpt

arXiv:2611.08048v1 Announce Type: new Abstract: Performance metrics can yield different conclusions from the same predictions. This methodological tutorial explains the effects of class prevalence, decision thresholds, aggregation rules and software conventions through controlled scenarios. Twenty-two confusion-matrix metrics are organized in a prevalence-coded map. Fifteen scenarios cover binary and multi-class classification, score discrimination, probability calibration, regression, uncertainty and consistency checks. Inputs are fixed in released files, and one factor is varied in each scenario. All 205 verification checks passed; 202 were independent and 3 were internal. An always-negative rule achieved an accuracy of 0.990 at a prevalence of 0.010, with balanced accuracy of 0.500. At fixed sensitivity and specificity, precision fell from 0.889 to 0.008 as prevalence fell from 0.5 to 0.001. Three probability variants shared a ROC-AUC of 0.825 but had calibration slopes of 2.08, 1.04 and 0.52. With 10 participants, participant-majority accuracy of 0.700 had a 95% interval of 0.348-0.933. Fold-averaged scores were falsely flagged as inconsistent under an inappropriate pooled-test-set model. A goal-based metric table and a reporting checklist are provided. The scenarios illustrate mechanisms and do not estimate their frequency in published studies. A literature audit and a real-data demonstration are specified as future protocols and were not executed.