# Calibrated selective prediction using deep ensembles for thyroid nodule ultrasound classification under dataset shift

DevFeed: [Calibrated selective prediction using deep ensembles for thyroid nodule ultrasound classification under dataset shift](<https://devfeed.tech/articles/calibrated-selective-prediction-using-deep-ensembles-for-thyroid-nodule-ultrasound-classification-under-dataset-shift-131950.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.12075>)

Author: Md. Sadibul Hasan Sadib, Md. Mohayminul Mukit, Rahmatul Kabir Rasel Sarker, Tahmid Alam Tamim, Md. Nazmul Hasan, Md. Monir Hossain Shimul

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [Image processing](<https://devfeed.tech/topics/image-processing.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>)

Tags: [classification](<https://devfeed.tech/tags/classification.md>), [cross-validation](<https://devfeed.tech/tags/cross-validation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [deep](<https://devfeed.tech/tags/deep.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [eess-iv](<https://devfeed.tech/tags/eess-iv.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fna](<https://devfeed.tech/tags/fna.md>), [framework](<https://devfeed.tech/tags/framework.md>), [image](<https://devfeed.tech/tags/image.md>), [msp](<https://devfeed.tech/tags/msp.md>), [performance](<https://devfeed.tech/tags/performance.md>), [policy](<https://devfeed.tech/tags/policy.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [risk](<https://devfeed.tech/tags/risk.md>), [roc](<https://devfeed.tech/tags/roc.md>), [selection](<https://devfeed.tech/tags/selection.md>), [shift](<https://devfeed.tech/tags/shift.md>), [site](<https://devfeed.tech/tags/site.md>), [test](<https://devfeed.tech/tags/test.md>), [ultrasound](<https://devfeed.tech/tags/ultrasound.md>)

## AI overview

A five-member ConvNeXt-Tiny ensemble with uncertainty-aware selective triage performed well on TN5000 but showed substantial dataset-shift degradation on TN3K, limiting its transportability. On TN5000, out-of-fold AUC-ROC was 0.9377 and expected calibration error 0.0138. The primary mutual-information policy assigned 8.86% of images a No-FNA suggestion, 60.00% an FNA recommendation, and 31.14% for radiologist review; 0.39% of malignant images received No-FNA. Predictive variance and maximum-softmax probability ranked errors better than mutual information, while no uncertainty measure optimized every triage outcome. Applied to TN3K without recalibration, AUC-ROC fell to 0.7870, calibration error rose to 0.1899, and 83.7% of images were sent for review. The authors call for multicentre validation, site-specific recalibration, threshold validation, and prospective evaluation before clinical use.

## Source excerpt

arXiv:2607.12075v3 Announce Type: replace-cross Abstract: Background: Deep learning can classify thyroid nodules on ultrasound, but discrimination alone does not establish clinical reliability. We developed a calibrated deep-ensemble framework with uncertainty-aware selective prediction and conservative image-based triage. Methods: Nested five-fold cross-validation with three-fold inner architecture selection chose ConvNeXt-Tiny with a squeeze-and-excitation block, used to build a deterministic five-member ensemble. Training, early-stopping, calibration, and threshold subsets were separated within each fold; outer-test data remained isolated. Mutual information (MI), predictive variance, and maximum-softmax-probability (MSP) uncertainty were compared by risk-coverage and triage analyses, and predictive values across assumed malignancy prevalences of 5%-70%. The frozen TN5000 ensemble was applied to TN3K without recalibration. Results: On TN5000, pooled out-of-fold AUC-ROC was 0.9377, average precision 0.9708, expected calibration error 0.0138, and Brier score 0.0827. Under the primary MI-guided policy, 8.86% of images received a No-FNA suggestion, 60.00% an FNA recommendation, and 31.14% radiologist review; of 3574 malignant images, 14 (0.39%) received No-FNA and 708 (19.81%) review. Predictive variance and MSP ranked errors better than MI (AURC, AUGRC); no measure optimized all triage outcomes. Across prevalences of 5%-70%, NPV ranged 99.95%-97.93% and PPV 29.28%-94.75%. On TN3K, AUC-ROC was 0.7870, average precision 0.7254, expected calibration error 0.1899, and radiologist-review rate 83.7%. Conclusion: The ensemble showed strong internal discrimination and calibration, but performance deteriorated substantially on TN3K, indicating limited transportability under dataset shift. Multicentre validation, site-specific recalibration, threshold validation, and prospective evaluation are needed before clinical use.