Suppressing Misaligned Behaviors in Language Models, Durably
arXiv cs.AI · 4 Nov 2026, 04:54 AEST
A study explores methods to suppress undesired behaviors in language models during training. The research introduces a new approach called Prefix-Expanded Adversarial Reinforcement Learning (PEARL) that shows promise in achieving durable suppression of misaligned behaviors.
Key facts
- Undesired model behaviors can emerge or re-emerge at any stage of training.
- We construct two model organisms--language models deliberately trained to exhibit a specific misalignment--that differ in what triggers the behavior.
- We introduce Prefix-Expanded Adversarial Reinforcement Learning (PEARL), a GRPO variant that augments rollouts with continuations from intermediate states sampled along cached organism trajectories.
- Across both organism settings and two model families (Qwen3-4B, GPT-OSS-20B), PEARL achieves strong immediate suppression while preserving task accuracy.
Read aloud
A study explores methods to suppress undesired behaviors in language models during training. The research introduces a new approach called Prefix-Expanded Adversarial Reinforcement Learning PEARL that shows promise in achieving durable suppression of misaligned behaviors. Undesired model behaviors can emerge or re-emerge at any stage of training. We construct two model organisms--language models deliberately trained to exhibit a specific misalignment--that differ in what triggers the behavior. We introduce Prefix-Expanded Adversarial Reinforcement Learning PEARL, a GRPO variant that augments rollouts with continuations from intermediate states sampled along cached organism trajectories. Across both organism settings and two model families Qwen3-4B, GPT-OSS-20B, PEARL achieves strong immediate suppression while preserving task accuracy.
Sources in this cluster
- Suppressing Misaligned Behaviors in Language Models, Durably
arXiv cs.AI · 4 Nov 2026, 04:54 AEST