# Model Annealing: Data-Efficient Learning through Review and Consolidation

DevFeed: [Model Annealing: Data-Efficient Learning through Review and Consolidation](<https://devfeed.tech/articles/model-annealing-data-efficient-learning-through-review-and-consolidation-135075.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17524>)

Author: Ali Mohammadi Ruzbahani, Rezvan Abouzadeh

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [machine learning overfitting](<https://devfeed.tech/topics/machine-learning-overfitting.md>), [generalization in machine learning](<https://devfeed.tech/topics/generalization-in-machine-learning.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [Inference](<https://devfeed.tech/topics/inference.md>)

Tags: [1-3](<https://devfeed.tech/tags/1-3.md>), [adversarial](<https://devfeed.tech/tags/adversarial.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [budget](<https://devfeed.tech/tags/budget.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [dreams](<https://devfeed.tech/tags/dreams.md>), [inference](<https://devfeed.tech/tags/inference.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [learning](<https://devfeed.tech/tags/learning.md>), [memory](<https://devfeed.tech/tags/memory.md>), [mixer](<https://devfeed.tech/tags/mixer.md>), [mlp](<https://devfeed.tech/tags/mlp.md>), [model](<https://devfeed.tech/tags/model.md>), [network](<https://devfeed.tech/tags/network.md>), [predictions](<https://devfeed.tech/tags/predictions.md>), [raises](<https://devfeed.tech/tags/raises.md>), [review](<https://devfeed.tech/tags/review.md>)

## AI overview

Model annealing uses the remaining training budget for repeated review: after each learning-rate reheating and cooldown, it stores that model's predictions, then trains the next student on labels, softened memory predictions, and adversarial examples found where student and memory disagree. The final student is deployed, so the prediction memory adds no inference cost. Across the tested CIFAR-100, Tiny-ImageNet, and Imagewoof settings, AdamW-trained ConvNeXt, ViT, and MLP-Mixer models beat cross-entropy top-1 accuracy in 28 of 33 configurations, averaging 1.3 points, and matched snapshot distillation in accuracy while improving likelihood, calibration, and robustness to corruptions and small adversarial perturbations. Results differ for the SGD-trained ResNet-18: accuracy fell, though small-perturbation robustness improved. Ablations attribute gains to relearning and losses to reheating; memory and adversarial dreams also curb the overconfidence that snapshot-only relearning compounds with little data.

## Source excerpt

arXiv:2611.17524v1 Announce Type: new Abstract: A small labelled dataset is memorized long before the training budget runs out, and the one-hot labels that remain to be fitted then mostly sharpen confidence. Model annealing spends the rest of the budget on review and consolidation, relearning from the network's own knowledge. Training is divided into episodes whose learning rate is raised again and cooled, every cooled state joins a memory of past selves, and each new episode learns from the labels, from the tempered predictions of this memory, and from adversarial dreams sought where student and memory disagree most. The memory averages predictions rather than weights, and the deployed network is the last student, so inference costs nothing extra. Across CIFAR-100, Tiny-ImageNet and Imagewoof, four architecture families and 5\% to 100\% of the labels, model annealing raises the top-1 accuracy of the AdamW-trained ConvNeXt, ViT and MLP-Mixer over cross-entropy training in 28 of 33 configurations, by 1.3 points on average, and is on par in accuracy with snapshot distillation, the closest competitor. On these families it is better than both in likelihood, calibration and robustness to corruptions and small adversarial perturbations. For an SGD-trained ResNet-18 it lowers accuracy but still improves robustness to small adversarial perturbations. Separating the reheated schedule from relearning shows that relearning helps every family and that the losses come from reheats, which undo most of this network's fit to its training set. How early the accuracy of a cross-entropy run saturates largely separates the AdamW families from ResNet-18 but does not predict the gain within them. With little data, relearning from the latest snapshot without dreams, as snapshot distillation does, compounds overconfidence in the AdamW families from episode to episode. The memory and the dreams each curb most of this drift, and together they remove it.