Towards interpreting any residual stream direction by scaling activation inversion
arXiv cs.AI · 6 Nov 2026, 04:54 AEST
A new approach called Maximally-Activating Example Metamodels (MAEMs) aims to interpret directions in a language model's residual stream. This method shows improved performance over traditional corpus search and natural-language autoencoders in generating informative text exemplars.
Key facts
- To interpret a direction in a language model's residual stream, we can look for text that strongly activates it.
- We develop Maximally-Activating Example Metamodels (MAEMs), language models which map residual stream directions from a subject model to text exemplars that strongly activate those directions.
- Empirically, we find the resultant metamodel produces informative exemplars: a Qwen3.6-27B MAEM inverts held-out activations better than corpus search or a natural-language autoencoder (NLA).
- It also identifies concepts behind steering vectors about as well as NLAs, recovers the payloads of LoRA backdoors more often than corpus search, and can even verbalise some model workspace inner monologue.
Read aloud
A new approach called Maximally-Activating Example Metamodels MAEMs aims to interpret directions in a language model's residual stream. This method shows improved performance over traditional corpus search and natural-language autoencoders in generating informative text exemplars. To interpret a direction in a language model's residual stream, we can look for text that strongly activates it. We develop Maximally-Activating Example Metamodels MAEMs, language models which map residual stream directions from a subject model to text exemplars that strongly activate those directions. Empirically, we find the resultant metamodel produces informative exemplars: a Qwen3.six-27B MAEM inverts held-out activations better than corpus search or a natural-language autoencoder NLA. It also identifies concepts behind steering vectors about as well as NLAs, recovers the payloads of LoRA backdoors more often than corpus search, and can even verbalise some model workspace inner monologue.
Sources in this cluster
- Towards interpreting any residual stream direction by scaling activation inversion
arXiv cs.AI · 6 Nov 2026, 04:54 AEST