# The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodology

DevFeed: [The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodology](<https://devfeed.tech/articles/the-model-organism-lottery-model-organism-interpretability-strongly-depends-on-training-methodology-108517.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.01033>)

Author: Andrzej Szablewski, Gabriel Konar-Steenberg, Raffaello Fornasiere, Nikita Menon, Stefan Heimersheim

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [activation-steering](<https://devfeed.tech/tags/activation-steering.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [interpretability](<https://devfeed.tech/tags/interpretability.md>), [lens](<https://devfeed.tech/tags/lens.md>), [methodology](<https://devfeed.tech/tags/methodology.md>), [oracles](<https://devfeed.tech/tags/oracles.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [sft](<https://devfeed.tech/tags/sft.md>), [training](<https://devfeed.tech/tags/training.md>), [variance](<https://devfeed.tech/tags/variance.md>), [white-box](<https://devfeed.tech/tags/white-box.md>)

## AI overview

Across 61 model organisms based on OLMo2-1B and gemma-3-1b-it, plus 96 knowledge-distilled variants, interpretability varied substantially with training objective, target behaviour, architecture and data-generation pipeline. The authors benchmarked activation oracles, activation steering, the logit lens and the Jacobian lens; variation persisted even when target-behaviour strength was controlled. More realistic integration into OLMo's post-training DPO phase and knowledge distillation often produced less interpretable models than standard post-hoc SFT. The findings raise doubts about using conventional post-hoc-trained model organisms as reliable proxies for interpretability, and motivate more robust benchmarks.

## Source excerpt

arXiv:2607.01033v2 Announce Type: replace Abstract: Model organisms (MOs) - language models trained to exhibit undesired or unnatural behaviours - are frequently used as testbeds for evaluating white-box interpretability techniques. Current MOs are typically constructed via post-hoc supervised fine-tuning (SFT). Prior research has shown that interpretability methods can easily identify hidden behaviours in these MOs. However, recent work suggests that such post-hoc training methods may make interpretability unrealistically easy. We investigate this claim by constructing a suite of 61 $\verb|OLMo2-1B|$- and $\verb|gemma-3-1b-it|$-based MOs trained with seven different techniques, including standard post-hoc SFT, post-hoc DPO, and more realistic integration of MO data into the OLMo post-training DPO phase. We also use these to train 96 further variants via knowledge distillation. We use these MO variants to benchmark activation oracles, activation steering, the logit lens, and the Jacobian lens. Our findings show that (i) MO interpretability depends strongly on training objective, target behaviour, model architecture, and training data generation pipeline; (ii) substantial variance remains even after controlling for differences in the strength of target behaviour expression; and (iii) both our more realistic $\textit{integrated training}$ and knowledge distillation often yield less interpretable MOs than standard post-hoc methods. Our results cast substantial doubt on the validity of current MOs as interpretability proxies and suggest actionable steps towards more robust benchmarks.