# How Much Does the Hint Matter? Scope Conditions for Hindsight Self-Distillation in Financial Reasoning

DevFeed: [How Much Does the Hint Matter? Scope Conditions for Hindsight Self-Distillation in Financial Reasoning](<https://devfeed.tech/articles/how-much-does-the-hint-matter-scope-conditions-for-hindsight-self-distillation-in-financial-reasoning-103082.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.36082>)

Author: Johnny Yu, Cyril Gorlla

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Finance](<https://devfeed.tech/topics/finance.md>), [gpt-oss](<https://devfeed.tech/topics/gpt-oss.md>)

Tags: [accounting](<https://devfeed.tech/tags/accounting.md>), [audit](<https://devfeed.tech/tags/audit.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data-residency](<https://devfeed.tech/tags/data-residency.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [error](<https://devfeed.tech/tags/error.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [financial](<https://devfeed.tech/tags/financial.md>), [financial-institutions](<https://devfeed.tech/tags/financial-institutions.md>), [gpt-oss](<https://devfeed.tech/tags/gpt-oss.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [hint](<https://devfeed.tech/tags/hint.md>), [hints](<https://devfeed.tech/tags/hints.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [open-weight-models](<https://devfeed.tech/tags/open-weight-models.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [regression](<https://devfeed.tech/tags/regression.md>), [student](<https://devfeed.tech/tags/student.md>)

## AI overview

Fin-SD improves multi-step financial reasoning by using a model's own failed attempt to focus distillation on the point just before its first error. The student critiques that step; a frozen copy conditions on the preceding reasoning and a hint, then trains the student over a short rollout with reverse KL, without an external model contributing gradients. On GPT-OSS-20B, generic feedback produced the same 3.03 percentage-point gain on FinanceReasoning Hard as the critique, while hints from GPT-OSS-120B offered no material benefit over self-written hints. Fin-SD matched full-reference-solution self-distillation in accuracy with 9.5% fewer tokens. The reported MMLU-Pro and 14-metric FinTrust checks found no material regression.

## Source excerpt

arXiv:2610.36082v1 Announce Type: cross Abstract: Financial institutions operating under data-residency constraints are restricted to open-weight models small enough to locally host, which trail frontier systems on multi-step quantitative reasoning. On-policy distillation narrows such gaps by supervising a student on its own rollouts, but traditional OPD requires a strong external teacher, and on-policy self-distillation (OPSD) conditions on a full reference solution. We ask how much of that signal is necessary. We present Fin-SD, a localized hindsight self-distillation recipe for multi-step reasoning. Given a problem the student fails, the student itself identifies the first incorrect reasoning step and writes a short critique of that step. We truncate the student's trajectory immediately before the error (aiming to obtain a correct prefix), condition a frozen copy of the student on the same prefix plus the hint, and distill that hint-conditioned teacher into the student over a short rollout under reverse KL. No external model contributes a gradient. We apply Fin-SD to GPT-OSS-20B on a custom multi-step financial reasoning benchmark that requires deriving formula inputs from realistic scenarios via accounting conventions. We find: (i) replacing the critique with a generic-feedback control matches Fin-SD's entire 3.03 pp accuracy gain over the base model on the FinanceReasoning Hard subset, suggesting the gain comes from where supervision is applied, not hint content; (ii) the model writing the hint does not need to be stronger than the model receiving it: hints from same-family GPT-OSS-120B, at roughly 6x the parameters, yield no materially greater benefits than self-written hints; and (iii) Fin-SD matches on-policy self-distillation from full reference solutions on accuracy while generating 9.5% fewer tokens. With Fin-SD, general reasoning on MMLU-Pro and a fourteen-metric compliance audit on FinTrust show no material regression.