# Structural Rationale Distillation via Reasoning Space Compression

DevFeed: [Structural Rationale Distillation via Reasoning Space Compression](<https://devfeed.tech/articles/structural-rationale-distillation-via-reasoning-space-compression-118214.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.07139>)

Author: Jialin Yang, Jiankun Wang, Jiajun Wu, Henry Leung, Jiayu Zhou, Steve Drew

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [bayes](<https://devfeed.tech/tags/bayes.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [compression](<https://devfeed.tech/tags/compression.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [diversity](<https://devfeed.tech/tags/diversity.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [math](<https://devfeed.tech/tags/math.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [rpc](<https://devfeed.tech/tags/rpc.md>), [size](<https://devfeed.tech/tags/size.md>), [strategy](<https://devfeed.tech/tags/strategy.md>), [structure](<https://devfeed.tech/tags/structure.md>), [student](<https://devfeed.tech/tags/student.md>), [supervision](<https://devfeed.tech/tags/supervision.md>), [template](<https://devfeed.tech/tags/template.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

D-RPC aims to make teacher rationales easier for smaller language models to learn by reducing structural inconsistency across similar problems. It retrieves a relevant high-level reasoning path from a compact, dynamically maintained bank, then conditions the teacher to use that path while retaining variety across problem types. The paper's PAC-Bayes analysis describes a bank-size tradeoff: smaller banks restrict path diversity and can leave coverage gaps; ablations favor an intermediate size. On five math and commonsense reasoning benchmarks with two student models, D-RPC had the highest average accuracy among evaluated distillation methods and used fewer tokens than template-heavy alternatives.

## Source excerpt

arXiv:2605.07139v2 Announce Type: replace-cross Abstract: When distilling reasoning from large language models (LLMs) into smaller ones, teacher rationales for similar problems often vary wildly in structure and strategy. Like a chef who makes the same dish differently each time, this inconsistency burdens the student with noisy supervision that is hard to internalize. We propose Distillation through Reasoning Path Compression (D-RPC), which constrains the teacher to follow a compact, dynamically maintained bank of reusable high-level reasoning paths. For each training question, D-RPC retrieves the most relevant path and conditions the teacher to follow it, producing rationales that are consistent across similar problems yet diverse enough to cover different problem types. A PAC-Bayes analysis formalizes a trade-off between bank size and coverage, with smaller banks limiting path diversity at the risk of coverage gaps. Our ablations favor an intermediate bank size, consistent with this trade-off. Across five math and commonsense reasoning benchmarks with two student models, D-RPC achieves the highest average accuracy among the evaluated distillation methods, while using fewer tokens than template-heavy alternatives.