# Training-Free On-Policy Distillation

DevFeed: [Training-Free On-Policy Distillation](<https://devfeed.tech/articles/training-free-on-policy-distillation-94196.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.22236>)

Author: Bangji Yang, Jiajun Fan, Zhaoyu Zhu, Hongbo Ma, Ge Liu

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [linear](<https://devfeed.tech/tags/linear.md>), [mathematics](<https://devfeed.tech/tags/mathematics.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

Training-free on-policy distillation combines separately tuned specialists into a single model using forward passes, without training steps. For specialists sharing a base model, each fused linear layer is solved by least squares to match specialists' outputs on inputs from their own rollouts; the required input second moments can be collected in one pass over a few hundred rollouts. Rollout quality matters: repetition-loop truncations make up few mathematics rollouts but many tokens, and removing them raises AIME25 performance from task arithmetic's level to the specialist's at matched precision. On Olmo-3-7B, a few GPU-hours of forward computation per specialist produced a generalist retaining most specialist gains across seven mathematics and code benchmarks.

## Source excerpt

arXiv:2610.22236v2 Announce Type: replace Abstract: On-policy distillation consolidates specialist models into one student at the price of rollouts, teacher passes and gradient steps for every teacher. We show that the consolidation does not need training: with forward passes only, the capabilities of separately tuned specialists can be aggregated into one model that is, on average, more accurate than either specialist alone. When specialists share a base model, distillation has an exact layer-wise solution: each linear layer of the fused model must reproduce every specialist's output on the inputs that specialist receives while reading its own answers, a least-squares problem whose solution needs only the second moments of those inputs. One forward pass over a few hundred rollouts provides them. We call this training-free on-policy distillation. What the rollouts contain matters more than the solver: answers truncated in repetition loops are few among a mathematics specialist's rollouts but carry a large share of their tokens, and removing them lifts the fused model on AIME25 from task arithmetic's level to the specialist's at matched precision. On Olmo-3-7B, a few GPU-hours of forward computation per specialist yield a generalist that keeps most of each specialist's gain on each of seven mathematics and code benchmarks.