# When Scores Conflict with Preferences: Pre-Alignment Diagnosis and Adaptive Correction of Constraint Drift in DPO

DevFeed: [When Scores Conflict with Preferences: Pre-Alignment Diagnosis and Adaptive Correction of Constraint Drift in DPO](<https://devfeed.tech/articles/when-scores-conflict-with-preferences-pre-alignment-diagnosis-and-adaptive-correction-of-constraint-drift-in-dpo-120607.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07375>)

Author: Ping Liu, Yan Yan

Published: 2026-11-06T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [constraint](<https://devfeed.tech/tags/constraint.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diagnosis](<https://devfeed.tech/tags/diagnosis.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>)

## AI overview

In standard DPO, preference pairs whose preferred response scores worse on a pointwise criterion can increase constraint cost, while matching pairs can reduce it; the net effect depends on their balance. The paper introduces a dataset-statistics diagnostic, ρ*eq, to predict whether a dataset is self-correcting or needs intervention. Its Conflict-aware DPO (CoDPO) method uses that diagnostic to set conflict-pair down-weighting and adds an adaptive margin shift to address remaining drift. In the reported experiments, the diagnostic's intervention decisions matched observed drift across all domains. Across four main-comparison domains, CoDPO reduced drift while largely preserving vanilla DPO's preference learning, with a more favorable drift-preference trade-off than hard filtering and the constrained and multi-objective baselines evaluated.

## Source excerpt

arXiv:2611.07375v1 Announce Type: new Abstract: Direct Preference Optimization (DPO) is increasingly used in settings where pairwise preferences coexist with pointwise criteria such as response length, answer correctness, or safety scores. When the two disagree, standard DPO treats pairs whose preferred output scores worse on the pointwise criterion (conflict pairs) in the same way as the remaining (matching) pairs. We characterize a systematic failure mode, constraint drift: for sufficiently small updates, conflict pairs tend to increase constraint cost while matching pairs tend to repair it, and the net change depends on their balance. This balance yields a closed-form pre-alignment diagnostic, $\rho^{*}_{\mathrm{eq}}$, computable from dataset statistics, that indicates whether a dataset is self-correcting or requires intervention. Building on this diagnostic, we propose Conflict-aware DPO (CoDPO), which down-weights conflict pairs with a strength set from $\rho^{*}_{\mathrm{eq}}$ rather than tuned, and adds an adaptive margin shift that addresses residual drift during training. Empirically, drift grows monotonically with conflict exposure on the conflict-dominated domains, and the intervention decision given by $\rho^{*}_{\mathrm{eq}}$ matches the observed drift on every domain, including one that it identifies as self-correcting. On all four domains of the main comparison, CoDPO reduces drift while largely preserving the preference learning of vanilla DPO. Compared with hard filtering and recent constrained and multi-objective baselines, CoDPO also achieves a more favorable drift-preference trade-off.