Reinforcement Learning with Visually Attributable On-Policy Self-Distillation
arXiv cs.AI · 20 Oct 2026, 03:59 AEST
A new approach called reinforcement learning with visually attributable on-policy self-distillation aims to improve visual reasoning in reinforcement learning. This method addresses issues of error localization and contamination in token-level corrections by grounding updates in visual evidence.
Key facts
- Multimodal reinforcement learning with verifiable rewards (RLVR) provides reliable task-level supervision, yet offers limited guidance for localizing errors in visual reasoning.
- On-policy self-distillation (OPSD) complements this supervision with dense token-level corrections from a privileged teacher equipped with an evidence-centered visual view.
- We formalize this phenomenon as privileged-correction contamination: a token-level correction is contaminated when its update direction cannot be consistently attributed to the privileged visual evidence.
- To address this issue, we propose reinforcement learning with visually attributable on-policy self-distillation, an evidence-consistent paradigm that augments task-level RLVR supervision with privileged token-level corrections grounded in visual evidence.
- Experiments across six visual reasoning benchmarks show that VAG consistently outperforms GRPO and OPSD methods, establishing new state-of-the-art average performance.
Read aloud
A new approach called reinforcement learning with visually attributable on-policy self-distillation aims to improve visual reasoning in reinforcement learning. This method addresses issues of error localization and contamination in token-level corrections by grounding updates in visual evidence. Multimodal reinforcement learning with verifiable rewards RLVR provides reliable task-level supervision, yet offers limited guidance for localizing errors in visual reasoning. On-policy self-distillation OPSD complements this supervision with dense token-level corrections from a privileged teacher equipped with an evidence-centered visual view. We formalize this phenomenon as privileged-correction contamination: a token-level correction is contaminated when its update direction cannot be consistently attributed to the privileged visual evidence. To address this issue, we propose reinforcement learning with visually attributable on-policy self-distillation, an evidence-consistent paradigm that augments task-level RLVR supervision with privileged token-level corrections grounded in visual evidence. Experiments across six visual reasoning benchmarks show that VAG consistently outperforms GRPO and OPSD methods, establishing new state-of-the-art average performance.
Sources in this cluster
- Reinforcement Learning with Visually Attributable On-Policy Self-Distillation
arXiv cs.AI · 20 Oct 2026, 03:59 AEST