Questioning Instrumental Convergence in RLVR: Checkpoint Evidence from OLMo-3
arXiv cs.AI · 23 Oct 2026, 03:44 AEST
A study questions the concept of instrumental convergence in reinforcement learning with verifiable rewards (RLVR) using the OLMo-3 model. The findings indicate that RLVR does not consistently increase instrumental behavior, with rates rising initially before declining below baseline levels.
Key facts
- Instrumental convergence is often invoked as a general reason to expect capable agents to acquire or express power-seeking subgoals: self-preservation, resource acquisition, deception, or resistance to oversight which poses an important threat to the society.
- Using InstrumentalEval, we evaluate OLMo-3-7B-Think across SFT, DPO, 13 intermediate RLVR checkpoints, and the final RLVR model.
- The trajectory is nonmonotonic: instrumental rate rises from 25.0% at SFT to 46.1% near RLVR step 200, then falls to 11.8% at the final checkpoint.
- Together with preliminary 32B endpoint results (35.53% SFT to 22.37% RLVR final), these findings indicate that RLVR does not monotonically increase instrumental behavior in this setting: the rate rises early, then falls below the SFT baseline.
Read aloud
A study questions the concept of instrumental convergence in reinforcement learning with verifiable rewards RLVR using the OLMo-three model. The findings indicate that RLVR does not consistently increase instrumental behavior, with rates rising initially before declining below baseline levels. Instrumental convergence is often invoked as a general reason to expect capable agents to acquire or express power-seeking subgoals: self-preservation, resource acquisition, deception, or resistance to oversight which poses an important threat to the society. Using InstrumentalEval, we evaluate OLMo-three-7B-Think across SFT, DPO, thirteen intermediate RLVR checkpoints, and the final RLVR model. The trajectory is nonmonotonic: instrumental rate rises from twenty five point zero percent at SFT to forty six point one percent near RLVR step two hundred, then falls to eleven point eight percent at the final checkpoint. Together with preliminary 32B endpoint results thirty five point five three percent SFT to twenty two point three seven percent RLVR final, these findings indicate that RLVR does not monotonically increase instrumental behavior in this setting: the rate rises early, then falls below the SFT baseline.
Sources in this cluster
- Questioning Instrumental Convergence in RLVR: Checkpoint Evidence from OLMo-3
arXiv cs.AI · 23 Oct 2026, 03:44 AEST