# Calibrated concept profiles: making the reading of a model's neurons a measurement

DevFeed: [Calibrated concept profiles: making the reading of a model's neurons a measurement](<https://devfeed.tech/articles/calibrated-concept-profiles-making-the-reading-of-a-model-s-neurons-a-measurement-123370.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09196>)

Author: Piotr Wilam

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [prompt](<https://devfeed.tech/topics/prompt.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [checkpoint](<https://devfeed.tech/tags/checkpoint.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [interpretability](<https://devfeed.tech/tags/interpretability.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [neuron](<https://devfeed.tech/tags/neuron.md>), [profile](<https://devfeed.tech/tags/profile.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [python](<https://devfeed.tech/tags/python.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

The paper presents calibrated concept profiles for measuring how individual model neurons respond to a target construct. It treats activation as a distribution over prompts drawn from a declared stimulus population, with surrounding factors sampled at rates from a reference corpus. Two prompt-battery draws provide a noise unit for each neuron's score, which is expressed relative to that neuron's spread on ordinary input. Fixed checks assess whether the battery is suitable for the model, and the method measures the smallest change detectable across training checkpoints. The authors implement it for Python on OpenCoder-1.5B across checkpoints spanning two trillion training tokens, reporting per-score units, chance levels for comparisons, and which neurons carry a construct over time. They say the battery, code, and result data are released.

## Source excerpt

arXiv:2611.09196v1 Announce Type: cross Abstract: Interpretability has largely moved from reading a model's neurons directly to reading a second model trained on the first: a probe, a sparse autoencoder, a crosscoder. The move answered real faults -- mixed meanings, thresholds without a unit, readings that did not transfer from one study to the next -- but gave up what a direct reading alone provides: nothing trained in between, the neurons themselves named, and a reading that can be repeated and compared. Here the direct reading is made a measurement. A neuron's activation varies with the prompt, so its reading is a distribution over a population of stimuli. We declare that population: every factor around a target construct is drawn at its rate in a reference corpus and logged. From the declared distribution we generate a battery of prompts, and draw it twice. From the battery we compute the calibrated concept profile, one number per neuron, stated in units of the neuron's own spread on ordinary input; the two draws give every number a noise unit. Before the profile is read, a fixed set of checks licenses the battery on the model, and the smallest change the profile can resolve across training checkpoints is measured. Realised for Python on OpenCoder-1.5B and followed across checkpoints spanning two trillion training tokens, the profile gives what the alternatives do not report: a unit on every number, a chance level for every comparison, and which neurons carry a construct at every checkpoint. The battery, the code and the data behind every result are released.