Context--Answer Metamodels: Forecasting LLM Activations Before the First Answer Token
arXiv cs.AI · 5 Nov 2026, 03:53 AEST
Research on context--answer metamodels reveals that activations from a language model's context can forecast its answer activations. This approach allows for scalable predictions of model behaviors prior to generating the first answer token.
Key facts
- Under stochastic decoding, a single context can elicit many different answers from a language model.
- We train context--answer metamodels that map a context activation sequence to point or distribution forecasts of mean-pooled answer activations.
- We find that the four metamodel families we study forecast mean-pooled answer activations with high fidelity, and the distribution metamodels also capture rollout variability.
- Performance improves as the training set grows to 500,000 prompts, and the metamodels generalize zero-shot to out-of-domain datasets.
Read aloud
Research on context--answer metamodels reveals that activations from a language model's context can forecast its answer activations. This approach allows for scalable predictions of model behaviors prior to generating the first answer token. Under stochastic decoding, a single context can elicit many different answers from a language model. We train context--answer metamodels that map a context activation sequence to point or distribution forecasts of mean-pooled answer activations. We find that the four metamodel families we study forecast mean-pooled answer activations with high fidelity, and the distribution metamodels also capture rollout variability. Performance improves as the training set grows to five hundred thousand prompts, and the metamodels generalize zero-shot to out-of-domain datasets.
Sources in this cluster
- Context--Answer Metamodels: Forecasting LLM Activations Before the First Answer Token
arXiv cs.AI · 5 Nov 2026, 03:53 AEST