# A Circuit for In-Context Learning

DevFeed: [A Circuit for In-Context Learning](<https://devfeed.tech/articles/a-circuit-for-in-context-learning-117058.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05344>)

Author: Andrew Gao, William Gao

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [few-shot](<https://devfeed.tech/topics/few-shot.md>), [few-shot-learning](<https://devfeed.tech/topics/few-shot-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [circuit](<https://devfeed.tech/tags/circuit.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [few-shot](<https://devfeed.tech/tags/few-shot.md>), [gate](<https://devfeed.tech/tags/gate.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [learning](<https://devfeed.tech/tags/learning.md>), [lookup](<https://devfeed.tech/tags/lookup.md>), [memory](<https://devfeed.tech/tags/memory.md>), [patching](<https://devfeed.tech/tags/patching.md>)

## AI overview

In few-shot sentence classification with arbitrary labels, the studied transformers perform in-context learning through a gated associative lookup during a single forward pass. Middle-layer attention heads write each demonstration's class into label-token positions; upper-layer heads match the query's class to those stored keys and retrieve the corresponding labels. An early-layer signal gates this readout when a label has repeated, while separate heads suppress the most recently seen label. The authors report causal evidence from activation patching, ablations and steering, and say the computation recurs across three 3-4B models, a model trained from scratch, and one model family from 0.6B to 8B, surviving instruction tuning. The circuit also transfers to semantic concepts and meaningful labels. These findings concern the settings and models described in the abstract.

## Source excerpt

arXiv:2611.05344v1 Announce Type: cross Abstract: In-context learning (ICL) lets a pretrained language model perform a task specified by demonstrations in its prompt, with no parameter updates. With arbitrary labels the model must acquire a mapping it cannot have memorized, so the learning happens within a single forward pass. Prior work found candidate parts but not how they compose. We give the composition for few-shot sentence classification with arbitrary labels in three 3-4B-parameter transformers. The model implements a key-value memory over label-token positions: each label token's residual stream encodes the label's identity, and middle-layer attention heads write into it the class of the preceding demonstration as its key. At the query, upper-layer heads match the query's class against the stored keys and copy out the matching labels. An early-layer signal encoding whether any label has repeated gates this readout; in its absence the readers attend to the start of the prompt and no label is favored. A separate set of heads suppresses the most recently seen label. Each step is established causally by activation patching, head ablations against random-head baselines, and steering with the model's own signals. A softmax model of the readout with two constants per head, fit on one prompt distribution, predicts another. The computation is conserved across the three models, the named heads alone reproduce the behaviour, and a transformer trained from scratch grows the same parts. The circuit transfers to semantic concepts and to labels with meaning, and the same parts, with a causal gate, appear from 0.6B to 8B in one family and survive instruction tuning. ICL in this setting is a gated associative lookup built from a few identifiable component types, traced causally from demonstrations to prediction.