# Proximal Preference Optimization: A Unified Framework for Stable and Reward-Free Alignment

DevFeed: [Proximal Preference Optimization: A Unified Framework for Stable and Reward-Free Alignment](<https://devfeed.tech/articles/proximal-preference-optimization-a-unified-framework-for-stable-and-reward-free-alignment-125254.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10785>)

Author: Qisai Liu, Zhanhong Jiang, Zahid Hasan, Soumik Sarkar

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [post-training](<https://devfeed.tech/topics/post-training.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [better](<https://devfeed.tech/tags/better.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [effective](<https://devfeed.tech/tags/effective.md>), [framework](<https://devfeed.tech/tags/framework.md>), [free](<https://devfeed.tech/tags/free.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [improvement](<https://devfeed.tech/tags/improvement.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [reward](<https://devfeed.tech/tags/reward.md>), [stable](<https://devfeed.tech/tags/stable.md>)

## AI overview

P²O adds a clipped importance ratio to DPO's preference margin, aiming to enable reward-model-free training with safer dataset reuse and bounded policy divergence. PKTO extends it with KTO-style utility margins, an adaptive baseline that tightens the effective trust region as the policy drifts, and asymmetric weighting for imbalanced preferences. The paper reports KL trust-region bounds, a policy-improvement lower bound, and convergence guarantees for multi-epoch updates. Across multiple LLMs and three preference benchmarks, the authors report that both methods match or exceed DPO and KTO in alignment quality and are more stable under repeated dataset reuse.

## Source excerpt

arXiv:2611.10785v1 Announce Type: new Abstract: We introduce a proximal preference optimization framework that unifies the implicit reward formulation of Direct Preference Optimization (DPO) with the trust-region stability of Proximal Policy Optimization (PPO). Our framework yields two complementary algorithms. P$^2$O (Proximal Preference Optimization) introduces a clipped importance ratio directly into the DPO preference margin, producing a reward-model-free algorithm that supports safe dataset reuse, bounded policy divergence, and monotonic preference improvement guarantees. PKTO (Proximal Kahneman-Tversky Optimization) extends P$^2$O by replacing the DPO margin with KTO's prospect-theoretic utility margins, adding a self-regulating adaptive baseline that tightens the effective trust region as the policy drifts, and an asymmetric weighting that handles imbalanced or asymmetric preference datasets. We provide a unified theoretical analysis, including Kullback Leibler (KL) trust-region bounds, a policy-improvement lower bound, and convergence guarantees under multi-epoch proximal updates. Empirically, P$^2$O and PKTO match or exceed DPO and KTO in alignment quality while maintaining better stability under repeated dataset reuse across multiple LLMs on three preference benchmarks. The code base is available here.