# A balloon effect: ban one channel and a hidden trait finds another

DevFeed: [A balloon effect: ban one channel and a hidden trait finds another](<https://devfeed.tech/articles/a-balloon-effect-ban-one-channel-and-a-hidden-trait-finds-another-115320.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.02509>)

Author: Keuntae Kim, Yong Suk Choi

Published: 2026-11-03T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>)

Tags: [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [ban](<https://devfeed.tech/tags/ban.md>), [channel](<https://devfeed.tech/tags/channel.md>), [circuit](<https://devfeed.tech/tags/circuit.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [format](<https://devfeed.tech/tags/format.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [leak](<https://devfeed.tech/tags/leak.md>), [learning](<https://devfeed.tech/tags/learning.md>), [patching](<https://devfeed.tech/tags/patching.md>), [random](<https://devfeed.tech/tags/random.md>), [reduction](<https://devfeed.tech/tags/reduction.md>), [show](<https://devfeed.tech/tags/show.md>), [stop](<https://devfeed.tech/tags/stop.md>), [stress](<https://devfeed.tech/tags/stress.md>), [student](<https://devfeed.tech/tags/student.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [trace](<https://devfeed.tech/tags/trace.md>)

## AI overview

Across two models, six output formats and 16 traits, banning visible ways a trait appeared did not eliminate it: the trait shifted to channels the format still allowed, an effect the authors call the balloon effect. The study traced this transfer to circuits that write traits into outputs. Patching that circuit while a teacher model generated numbers stopped the tested chains, sharply reduced overt name leakage and halved the trait passed to a student; patching the same number of random heads produced no detectable reduction. The findings suggest that surface-token bans alone may not prevent subliminal trait transfer, while circuit-level intervention reduced it in these tests.

## Source excerpt

arXiv:2611.02509v1 Announce Type: new Abstract: Language models trained on the outputs of other models can acquire traits that those outputs never mention, such as a preference for an animal carried by lists of numbers. Such subliminal learning can pass on unwanted traits. The intuitive defence is to find and forbid the tokens through which a trait shows, but how a trait's trace behaves under such bans, and what of it a student learns, has not been resolved. We closed a trait's outlets one by one in two stress tests, a preference chain and a name chain, across two models, six output formats and 16 traits. Bans did not remove the trait. Much as squeezing a balloon in one place makes it bulge in another, the trait moved to wherever the format still allowed, an effect we call the balloon effect. Decomposing this transfer, which bans on surface features cannot stop, down to single heads and neurons, we show that it can be cut at the circuit that writes the trait into the output. Patching that circuit while the teacher writes numbers stopped the chains, sharply reduced the overt name leak and halved the trait passed to a student, whereas the same number of random heads produced no detectable reduction. These results give a mechanistic account of how unwanted hidden traits are transferred and a route to preventing it.