PORT: Preference Optimization via Robust Token-Level Reweighting
arXiv cs.AI · 10 Nov 2026, 03:38 AEST
Preference optimization is crucial for aligning large language models with human values, but it often suffers from noisy preference annotations. The proposed PORT framework offers a fine-grained approach to improve robustness by performing token-level reweighting.
Key facts
- Preference optimization has become a central approach for aligning large language models with human values, but its effectiveness depends heavily on the quality of preference annotations.
- In practice, preference data is often noisy due to annotation errors and ambiguity.
- We propose Preference Optimization via Robust Token-level reweighting (PORT), a fine-grained framework for robust alignment under noisy preferences.
- PORT performs token-level reweighting using the empirical cumulative distribution function (CDF) of token log-probabilities, yielding an efficient forward-pass-only proxy to gradient-norm penalization.
- Extensive experiments across diverse noise settings demonstrate that PORT consistently improves robustness and outperforms existing sequence-level baselines.
Read aloud
Preference optimization is crucial for aligning large language models with human values, but it often suffers from noisy preference annotations. The proposed PORT framework offers a fine-grained approach to improve robustness by performing token-level reweighting. Preference optimization has become a central approach for aligning large language models with human values, but its effectiveness depends heavily on the quality of preference annotations. In practice, preference data is often noisy due to annotation errors and ambiguity. We propose Preference Optimization via Robust Token-level reweighting PORT, a fine-grained framework for robust alignment under noisy preferences. PORT performs token-level reweighting using the empirical cumulative distribution function CDF of token log-probabilities, yielding an efficient forward-pass-only proxy to gradient-norm penalization. Extensive experiments across diverse noise settings demonstrate that PORT consistently improves robustness and outperforms existing sequence-level baselines.
Sources in this cluster
- PORT: Preference Optimization via Robust Token-Level Reweighting
arXiv cs.AI · 10 Nov 2026, 03:38 AEST