# Feature Superposition in Neural Networks: From Theory to Practice

DevFeed: [Feature Superposition in Neural Networks: From Theory to Practice](<https://devfeed.tech/articles/feature-superposition-in-neural-networks-from-theory-to-practice-94754.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.06862>)

Author: Dai Shi, Xiaoyu Li, Andi Han, Jos\\'e Miguel Hern\\'andez-Lobato

Published: 2026-10-22T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [adversarial](<https://devfeed.tech/tags/adversarial.md>), [applications](<https://devfeed.tech/tags/applications.md>), [autoencoders](<https://devfeed.tech/tags/autoencoders.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoder](<https://devfeed.tech/tags/decoder.md>), [lower-bounds](<https://devfeed.tech/tags/lower-bounds.md>), [neural-networks](<https://devfeed.tech/tags/neural-networks.md>), [new-features](<https://devfeed.tech/tags/new-features.md>), [open-problems](<https://devfeed.tech/tags/open-problems.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [overlap](<https://devfeed.tech/tags/overlap.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>), [superposition](<https://devfeed.tech/tags/superposition.md>), [theory](<https://devfeed.tech/tags/theory.md>), [toy](<https://devfeed.tech/tags/toy.md>)

## AI overview

Superposition lets neural networks represent more features than they have dimensions, with interference limiting how well each feature can be read back. The abstract contrasts theoretical decoding requirements: a single linear map that recovers every nonzero feature combination needs nearly quadratic width, while sparse optimization can work at linear width. For trained networks, features are unknown and must be estimated from activations or weights, using methods such as sparse autoencoders and transcoders. The article organizes theory by decoder class and practical recovery methods by the strength of evidence for claims about features. It reviews overlap bounds, dimension sharing in toy models, and the distinction between storing and computing features; it also groups recovery methods into four families and compares their evaluations, failure modes, and applications. This overview is based on the supplied abstract, not the full article.

## Source excerpt

arXiv:2609.06862v2 Announce Type: replace Abstract: Superposition refers to neural networks representing more features than they have dimensions. It was proposed to explain polysemantic neurons, which respond to several features, and also other phenomena, such as neural scaling laws and adversarial vulnerability. A feature stored in superposition is useful to the extent that its value can be read back despite interference from the other features. The theory of superposition mainly asks how wide a layer must be for the features to be read back, and by which decoder. In theoretical models, a single linear map that reads back every combination of nonzero features needs a width nearly quadratic in their number, while sparse optimization can succeed with a width linear in that number. In practice, one may hope to interpret a trained network by recovering its features in the same way. However, its features are usually unknown, and thus they are estimated from its activations or weights by recovery methods, such as sparse autoencoders and transcoders. In this work, we organize the theory of superposition by the decoder class and its practice by the strength of the claims about a feature supported by each method's results. On the theory side, we review the lower bounds on the overlap between feature directions, the conditions under which trained toy models share dimensions among features, and the separation between storing features and computing new features. On the practice side, we categorize recovery methods into four families, from probes for a named feature to decompositions of the trained weights. We then compare their evaluations, failure modes, and applications, and grade the evidence on one scale of claims about a feature. Finally, we assess previously stated open problems and identify remaining theoretical and empirical questions about superposition in trained networks. Project page: https://eethanshi.github.io/awesome-superposition/