AI TrackerFrontier desk · AEST
Back to news
normalaitech

Suppressing Misaligned Behaviors in Language Models, Durably

arXiv cs.AI · 4 Nov 2026, 04:54 AEST

A study explores methods to suppress undesired behaviors in language models during training. The research introduces a new approach called Prefix-Expanded Adversarial Reinforcement Learning (PEARL) that shows promise in achieving durable suppression of misaligned behaviors.

Key facts

Read aloud

A study explores methods to suppress undesired behaviors in language models during training. The research introduces a new approach called Prefix-Expanded Adversarial Reinforcement Learning PEARL that shows promise in achieving durable suppression of misaligned behaviors. Undesired model behaviors can emerge or re-emerge at any stage of training. We construct two model organisms--language models deliberately trained to exhibit a specific misalignment--that differ in what triggers the behavior. We introduce Prefix-Expanded Adversarial Reinforcement Learning PEARL, a GRPO variant that augments rollouts with continuations from intermediate states sampled along cached organism trajectories. Across both organism settings and two model families Qwen3-4B, GPT-OSS-20B, PEARL achieves strong immediate suppression while preserving task accuracy.

Sources in this cluster