# Does On-Policy Distillation Need to Complete Every Rollout?

DevFeed: [Does On-Policy Distillation Need to Complete Every Rollout?](<https://devfeed.tech/articles/does-on-policy-distillation-need-to-complete-every-rollout-94193.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.21337>)

Author: Zhengxin Zhang, Maosen Tang, Ning Wang, Changhao Li, Jingren Zhou, Bo Dai, Sainyam Galhotra, Alex Townsend, Claire Cardie

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Language models](<https://devfeed.tech/topics/language-models.md>)

Tags: [bias](<https://devfeed.tech/tags/bias.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [muon](<https://devfeed.tech/tags/muon.md>), [policy](<https://devfeed.tech/tags/policy.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [student](<https://devfeed.tech/tags/student.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

STOPD reduces the cost of on-policy distillation by randomly choosing a rollout stopping position before generation, then weighting feedback on generated tokens with blockwise inverse-probability weights. The weights compensate for later tokens being less likely to be retained while controlling their magnitude; the authors prove the resulting gradient sum matches Vanilla OPD in expectation and give one-step update-error bounds for SGD, Adam, and Muon. Across the evaluated model pairs, task domains, and optimizers, STOPD reduced generated tokens by 25-48%, with accuracy comparable to Vanilla OPD and training speedups of 1.3-2.2x. These are abstract-level reported results; the supplied text does not detail experimental settings.

## Source excerpt

arXiv:2610.21337v3 Announce Type: replace Abstract: On-policy distillation (OPD) transfers teacher knowledge to a student through feedback on student-generated rollouts, but generating complete rollouts and computing per-token teacher feedback are computationally expensive. Existing truncation methods save computation by stopping rollouts early. However, they do not compensate for truncated token gradients, which biases the gradient estimates and changes Vanilla OPD's original training objective. Furthermore, their truncation decisions rely on assumptions about supervision quality or criteria that may not generalize across settings. We introduce Stochastic Truncated On-Policy Distillation (STOPD), which accelerates OPD by randomly truncating student rollouts and correcting the resulting sampling bias with blockwise inverse-probability weighting (IPW). At each training step, we sample a stopping position for each rollout before generation and let the student generate until that position or natural termination. We obtain teacher feedback on every generated token and use blockwise IPW to compensate for the lower probability of retaining later tokens while controlling the weights. We prove that this weighting preserves Vanilla OPD's gradient sum in expectation and derive one-step update-error bounds for SGD, Adam, and Muon. We evaluate STOPD across multiple teacher-student model pairs, task domains, and optimizers. STOPD reduces generated tokens by 25-48% while maintaining accuracy comparable to Vanilla OPD, achieving 1.3-2.2x training speedups.