Differentially Private Sparse Reward Estimation with Preference Feedback
arXiv cs.AI · 22 Oct 2026, 03:45 AEST
A new theoretical framework for differentially private sparse reward estimation in reinforcement learning has been introduced. This framework addresses the limitations of existing methods that rely on dense reward parameterizations, which can lead to significant estimation errors.
Key facts
- Reinforcement Learning with Human Feedback (RLHF) has become a central paradigm for aligning Large Language Models with human values.
- Existing private alignment methods typically assume dense reward parameterizations, resulting in estimation error bounds that scale polynomially with the ambient model size.
- We provide the first comprehensive theoretical framework for differentially private sparse reward estimation from pairwise preference feedback under the Bradley-Terry-Luce model.
- Empirically, our framework demonstrates superior efficacy over dense baselines across sentiment generation and safety alignment benchmarks.
Read aloud
A new theoretical framework for differentially private sparse reward estimation in reinforcement learning has been introduced. This framework addresses the limitations of existing methods that rely on dense reward parameterizations, which can lead to significant estimation errors. Reinforcement Learning with Human Feedback RLHF has become a central paradigm for aligning Large Language Models with human values. Existing private alignment methods typically assume dense reward parameterizations, resulting in estimation error bounds that scale polynomially with the ambient model size. We provide the first comprehensive theoretical framework for differentially private sparse reward estimation from pairwise preference feedback under the Bradley-Terry-Luce model. Empirically, our framework demonstrates superior efficacy over dense baselines across sentiment generation and safety alignment benchmarks.
Sources in this cluster
- Differentially Private Sparse Reward Estimation with Preference Feedback
arXiv cs.AI · 22 Oct 2026, 03:45 AEST