# Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning

DevFeed: [Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning](<https://devfeed.tech/articles/reliability-weighted-direct-preference-optimization-penalized-weights-for-robust-preference-learning-111601.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.41588>)

Author: Bin Luo, Param Venkat Vivek Kesireddy, Anil Vallepu

Published: 2026-11-02T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [annotation](<https://devfeed.tech/tags/annotation.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [error](<https://devfeed.tech/tags/error.md>), [estimator](<https://devfeed.tech/tags/estimator.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [learning](<https://devfeed.tech/tags/learning.md>), [linear](<https://devfeed.tech/tags/linear.md>), [margin](<https://devfeed.tech/tags/margin.md>), [margins](<https://devfeed.tech/tags/margins.md>), [minimax](<https://devfeed.tech/tags/minimax.md>), [model](<https://devfeed.tech/tags/model.md>), [noise](<https://devfeed.tech/tags/noise.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [policy](<https://devfeed.tech/tags/policy.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [robust](<https://devfeed.tech/tags/robust.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

## AI overview

RW-DPO jointly learns a preference policy and per-comparison reliability weights, addressing noisy or flipped labels that standard DPO treats as correct. A minimax-concave penalty on the negative log weight leaves comparisons the model agrees with at weight 1 and downweights those it contradicts; closed-form updates add negligible cost over DPO. The paper proves flipped comparisons cannot all be recovered from one annotation each without margins of order log n, and gives consistency and recovery results in a linear-index model under a correctly directed initial fit, including when a fixed fraction of labels is flipped. Simulations report lower average estimation error than DPO and other compared robust methods, with more precise flags than label-correction methods. In language-model fine-tuning with 10-30% flipped labels, reported win rate does not decline as the flip rate rises and exceeds DPO trained on clean labels, matching the strongest compared method.

## Source excerpt

arXiv:2610.41588v1 Announce Type: cross Abstract: Direct Preference Optimization (DPO) treats every reported comparison as correct, yet preference data contain reversed, tied or otherwise noisy comparisons. RW-DPO gives each comparison an unknown reliability weight $w_i\in[0,1]$ that enters the likelihood linearly -- DPO is $w\equiv1$, a reversed comparison $w_i=0$ -- and estimates the weights jointly with the policy under a minimax-concave penalty on $-\log w_i$. The penalty keeps every comparison the model agrees with at exactly $w_i=1$ and acts only on those it contradicts, through closed-form weight updates at negligible cost over DPO. We prove that recovering all flipped comparisons from a single annotation per comparison requires margins of order $\log n$. Furthermore, in a linear-index model, starting from an initial fit pointing in the correct direction, our estimator jointly consistently recovers the preference direction and weights, as well as identifies the flipped comparisons above this margin scale, even when a fixed fraction of comparisons is flipped. In simulations RW-DPO has the lowest average estimation error among DPO, noise-rate corrections and recent robust methods, with far more precise flags than recent label-correction methods. In DPO fine-tuning of a language model with 10-30% flipped labels, its win rate does not fall as the flip rate grows and exceeds that of DPO trained on clean labels, matching the strongest recent method with corrections concentrated on the flipped labels.