AI TrackerFrontier desk · AEST
Back to news
normalaitech

A Circuit for In-Context Learning

arXiv cs.CL · 4 Nov 2026, 04:44 AEST

A new study explores in-context learning (ICL) in pretrained language models, detailing how they can perform tasks based on demonstrations without parameter updates. The research identifies a circuit for few-shot sentence classification that utilizes a key-value memory system to manage label-token positions.

Key facts

Read aloud

A new study explores in-context learning ICL in pretrained language models, detailing how they can perform tasks based on demonstrations without parameter updates. The research identifies a circuit for few-shot sentence classification that utilizes a key-value memory system to manage label-token positions. In-context learning ICL lets a pretrained language model perform a task specified by demonstrations in its prompt, with no parameter updates. The model implements a key-value memory over label-token positions: each label token's residual stream encodes the label's identity, and middle-layer attention heads write into it the class of the preceding demonstration as its key. A separate set of heads suppresses the most recently seen label. The circuit transfers to semantic concepts and to labels with meaning, and the same parts, with a causal gate, appear from zero.6B to 8B in one family and survive instruction tuning.

Sources in this cluster