AI TrackerFrontier desk · AEST
Back to news
normalaitech

Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization

arXiv cs.AI · 4 Nov 2026, 04:59 AEST

Direct Nash Optimization (DNO) aligns a language model to a general preference function by approximating the Nash equilibrium of a two-player preference game. The study re-implements DNO-Prct and evaluates its performance on two open-weight instruct models, showing significant improvements in win rates against GPT-4o.

Key facts

Read aloud

Direct Nash Optimization DNO aligns a language model to a general preference function by approximating the Nash equilibrium of a two-player preference game. The study re-implements DNO-Prct and evaluates its performance on two open-weight instruct models, showing significant improvements in win rates against G P T four oh. Its practical variant, DNO-Prct, iterates on-policy sampling, L L M-based rating of the samples together with a teacher's gold response, large-margin preference-pair selection and a DPO-style contrastive update. On AlpacaEval two point zero with G P T four oh as judge, three iterations raise the length-controlled LC win rate against G P T four oh from thirty one point seven percent to forty five point three percent for Llama and from six point eight percent to eighteen point four percent for Mistral. With same-family open-weight annotators, the runs retain sixty two percent Llama and eighty two percent Mistral of the LC gain achieved with G P T four oh as annotator, so the rating step does not require a proprietary model. In a separate ablation configuration, variants that remove the gold responses, remove the data look-back, or accept only teacher-preferred pairs score six to ten LC points below the full procedure.

Sources in this cluster