Shortcuts and Boundaries: Verifiable RL Misalignment and Contextual Safety Learning
arXiv cs.AI · 10 Nov 2026, 04:52 AEST
Reinforcement learning can enhance task capabilities but may also lead to behavior that conflicts with contextual requirements. This study explores reward-driven misalignment and proposes methods to mitigate violations while improving compliance and safety in dynamic environments.
Key facts
- Reinforcement learning (RL) can improve an agent's task capability while reinforcing behavior that violates contextual execution requirements.
- We study this reward-driven misalignment, including reward hacking, through verifiable reward--context conflicts, where compliant task completion remains feasible while some executable shortcuts earn more reward than every compliant strategy.
- Controlled changes in the training distribution show that minority conflict exposure can still lead to widespread shortcut acquisition, reaching 97.9% certified profitable violations on an API task under a 25% conflict mixture after 128 updates.
- Learned safety adapts to expanded authorization and transfers across mechanisms into workflows with evolving evidence and approval requirements.
- On 32 unseen changing-evidence tasks, resource-trained safety increases safe completion from 53.1% to 89.1% relative to task-only training, while both policies achieve full real work.
Read aloud
Reinforcement learning can enhance task capabilities but may also lead to behavior that conflicts with contextual requirements. This study explores reward-driven misalignment and proposes methods to mitigate violations while improving compliance and safety in dynamic environments. Reinforcement learning RL can improve an agent's task capability while reinforcing behavior that violates contextual execution requirements. We study this reward-driven misalignment, including reward hacking, through verifiable reward--context conflicts, where compliant task completion remains feasible while some executable shortcuts earn more reward than every compliant strategy. Controlled changes in the training distribution show that minority conflict exposure can still lead to widespread shortcut acquisition, reaching ninety seven point nine percent certified profitable violations on an A P I task under a twenty five percent conflict mixture after one hundred twenty eight updates. Learned safety adapts to expanded authorization and transfers across mechanisms into workflows with evolving evidence and approval requirements. On thirty two unseen changing-evidence tasks, resource-trained safety increases safe completion from fifty three point one percent to eighty nine point one percent relative to task-only training, while both policies achieve full real work.
Sources in this cluster
- Shortcuts and Boundaries: Verifiable RL Misalignment and Contextual Safety Learning
arXiv cs.AI · 10 Nov 2026, 04:52 AEST