AI TrackerFrontier desk · AEST
Back to news
normalaitech

Base Models Can Reason By Taking a Cue From Training Data

arXiv cs.AI · 6 Oct 2026, 03:59 AEST

The study explores how training data influences reasoning in base models by creating associations with starting tokens. Fixing specific token cues can significantly enhance model performance, making it competitive with reinforcement learning-trained models.

Key facts

Read aloud

The study explores how training data influences reasoning in base models by creating associations with starting tokens. Fixing specific token cues can significantly enhance model performance, making it competitive with reinforcement learning-trained models. First, we demonstrate that fixing particular starting token cues makes a base model's performance competitive with that of its reinforcement learning RL -trained counterparts on math and coding. For instance, the cue '. Okay' raises Olmo-three-7B's MATH-five hundred pass@one accuracy from forty two percent to seventy eight percent, while 'Alright,' raises Qwen3-14B's from seventy two percent to eighty seven percent. We perform causal data interventions to turn an arbitrary word, such as 'chicken', into an effective reasoning cue, or remove an existing cue's effect. We also find that the hidden state representations induced by different cues correlate with different document types from the training set. Finally, we extend our study of token cues with a case study in language model safety, finding that different cues elicit distinct refusal and compliance behaviors that correspond to different types of training data.

Sources in this cluster