# Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization

DevFeed: [Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization](<https://devfeed.tech/articles/aligning-language-models-with-general-preferences-an-empirical-study-of-direct-nash-optimization-117064.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05373>)

Author: Austin Tsai, Saurav Nanda, Ganapathy Parthasarathy, Bhuvnesh Kumar, Parivesh Choudhary

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [llama](<https://devfeed.tech/topics/llama.md>), [open-llm-leaderboard](<https://devfeed.tech/topics/open-llm-leaderboard.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [llama-3-1](<https://devfeed.tech/tags/llama-3-1.md>), [margin](<https://devfeed.tech/tags/margin.md>), [mistral](<https://devfeed.tech/tags/mistral.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [weight](<https://devfeed.tech/tags/weight.md>)

## AI overview

DNO-Prct's iterative preference-training procedure improved AlpacaEval 2.0 length-controlled win rates after three iterations: Llama 3.1 8B Instruct rose from 31.7% to 45.3% against GPT-4o, and Mistral 7B Instruct v0.1 from 6.8% to 18.4%. The procedure samples model responses on-policy, has an LLM rate them alongside a teacher's gold response, selects large-margin preference pairs, then applies a DPO-style update. Same-family open-weight annotators retained 62% of Llama's and 82% of Mistral's gain relative to GPT-4o annotation. Ablations removing gold responses, data look-back, or non-teacher-preferred pairs scored 6-10 points lower; changing sample count or margin had smaller reported effects. These are single-run results, with GPT-4o serving as both teacher and judge.

## Source excerpt

arXiv:2611.05373v1 Announce Type: new Abstract: Direct Nash Optimization (DNO) aligns a language model to a general preference function by approximating the Nash equilibrium of a two-player preference game. Its practical variant, DNO-Prct, iterates on-policy sampling, LLM-based rating of the samples together with a teacher's gold response, large-margin preference-pair selection and a DPO-style contrastive update. We re-implement DNO-Prct independently and study it on two open-weight instruct models, Llama 3.1 8B Instruct and Mistral 7B Instruct v0.1, with GPT-4o or a larger open-weight model of the same family as the annotator. On AlpacaEval 2.0 with GPT-4o as judge, three iterations raise the length-controlled (LC) win rate against GPT-4o from 31.7% to 45.3% for Llama and from 6.8% to 18.4% for Mistral, and the tuned 8B model scores above Llama 3.1 70B Instruct on every reported win rate. With same-family open-weight annotators, the runs retain 62% (Llama) and 82% (Mistral) of the LC gain achieved with GPT-4o as annotator, so the rating step does not require a proprietary model. In a separate ablation configuration, variants that remove the gold responses, remove the data look-back, or accept only teacher-preferred pairs score 6 to 10 LC points below the full procedure, whereas varying the number of on-policy samples per prompt (1 to 7) moves the LC win rate by at most 2.1 points, and a margin sweep (1 to 5) run in its own pipeline by at most 2.4 points. We also report the cost of the pipeline and where the practical algorithm departs from the theory it approximates. All results are single runs, and GPT-4o is both the teacher and the evaluation judge; we discuss what this implies for the reported gains.