On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning
arXiv cs.AI · 7 Nov 2026, 04:49 AEST
The study examines the reliability of majority-vote pseudo-labels in test-time reinforcement learning. It introduces a new method, Prompt-Conditional Distributionally Robust DPO, to improve the adaptation process.
Key facts
- Test-time reinforcement learning (TTRL) adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, whose reliability remains poorly understood.
- We analyze majority-vote supervision for TTRL with online Direct Preference Optimization (DPO) and decompose its error into population-majority bias and prompt-dependent finite-rollout variation.
- We propose Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO), which conditions within-prompt robust reweighting on observed vote reliability to make updates from sampling-sensitive prompts more conservative.
- Across six mathematical and knowledge-intensive reasoning benchmarks and five model backbones, PC-Dr.DPO matches or outperforms the strongest TTRL baseline in 24 of 30 settings.
Read aloud
The study examines the reliability of majority-vote pseudo-labels in test-time reinforcement learning. It introduces a new method, Prompt-Conditional Distributionally Robust DPO, to improve the adaptation process. Test-time reinforcement learning TTRL adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, whose reliability remains poorly understood. We analyze majority-vote supervision for TTRL with online Direct Preference Optimization DPO and decompose its error into population-majority bias and prompt-dependent finite-rollout variation. We propose Prompt-Conditional Distributionally Robust DPO PC-Dr.DPO, which conditions within-prompt robust reweighting on observed vote reliability to make updates from sampling-sensitive prompts more conservative. Across six mathematical and knowledge-intensive reasoning benchmarks and five model backbones, PC-Dr.DPO matches or outperforms the strongest TTRL baseline in twenty four of thirty settings.
Sources in this cluster
- On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning
arXiv cs.AI · 7 Nov 2026, 04:49 AEST