# Beyond the Teacher: Understanding and Preserving Distillation Overshoot in Agentic On-Policy Distillation

DevFeed: [Beyond the Teacher: Understanding and Preserving Distillation Overshoot in Agentic On-Policy Distillation](<https://devfeed.tech/articles/beyond-the-teacher-understanding-and-preserving-distillation-overshoot-in-agentic-on-policy-distillation-93410.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.25034>)

Author: Jike Zhong, Ming Li, Yuxiang Lai, Xi Xiao, Sheng Zhang, Ritwick Chaudhry, Tianle Chen, Jingyu Xie, Zheda Mai, Shao-Yuan Lo, Konstantinos Psounis, Yuanyuan Lei

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-tasks](<https://devfeed.tech/tags/agentic-tasks.md>), [agents](<https://devfeed.tech/tags/agents.md>), [all](<https://devfeed.tech/tags/all.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [best](<https://devfeed.tech/tags/best.md>), [but](<https://devfeed.tech/tags/but.md>), [can](<https://devfeed.tech/tags/can.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [effects](<https://devfeed.tech/tags/effects.md>), [guidance](<https://devfeed.tech/tags/guidance.md>), [https](<https://devfeed.tech/tags/https.md>), [intermediate](<https://devfeed.tech/tags/intermediate.md>), [model](<https://devfeed.tech/tags/model.md>), [points](<https://devfeed.tech/tags/points.md>), [policy](<https://devfeed.tech/tags/policy.md>), [repair](<https://devfeed.tech/tags/repair.md>)

## AI overview

On-policy distillation can let a weaker student outperform its teacher, then lose those gains as training continues. The authors call this "distillation overshoot": teacher guidance repairs weak decisions early, but can later erode successful student choices the teacher disfavors. Their Success-Preserving On-Policy Distillation (SPOD) method masks conflicting teacher supervision on successful choices and applies a soft probability floor, while retaining guidance elsewhere. It uses binary episode success and model probabilities already available during OPD. Across two model families and three agentic benchmarks, SPOD reportedly improved final success by up to 15.2 percentage points, exceeded teacher performance by up to 20.0 points, and beat ordinary OPD's best intermediate checkpoint in all six settings.

## Source excerpt

arXiv:2610.25034v1 Announce Type: new Abstract: On-policy distillation (OPD) trains agents on their own trajectories using supervision from a stronger teacher. We uncover a counterintuitive phenomenon in multi-turn agentic tasks: a smaller, initially weaker student can substantially outperform its teacher during training, only to regress as distillation continues. We term this phenomenon distillation overshoot. Through decision- and task-level analyses, we trace overshoot to two competing effects of OPD: repairing weak student decisions and eroding successful student choices that the teacher disfavors. Early in training, repair outpaces erosion, allowing the student to combine the teacher's corrections with its own strengths and surpass the teacher. As training continues, erosion overtakes repair, causing these gains to vanish. Motivated by this finding, we propose Success-Preserving On-Policy Distillation (SPOD), which selectively masks conflicting teacher supervision on successful student choices and introduces a soft probability floor to counteract their erosion, while retaining teacher guidance elsewhere. SPOD requires only binary episode success and model probabilities already available during OPD. Across two model families and three agentic benchmarks, SPOD consistently outperforms ordinary OPD and evaluated variants, improving final success by up to 15.2 percentage points and surpassing the teacher by up to 20.0 points. Moreover, SPOD's final checkpoints outperform even ordinary OPD's best intermediate checkpoints in all six settings, demonstrating that beyond-teacher gains can be not only preserved but further extended. Code: https://github.com/Jike338/SPOD