# Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment

DevFeed: [Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment](<https://devfeed.tech/articles/learning-where-it-matters-geometric-anchoring-for-robust-preference-alignment-108464.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.04909>)

Author: Youngjae Cho, Jongsuk Kim, Ji-Hoon Kim

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [adversarial](<https://devfeed.tech/tags/adversarial.md>), [align](<https://devfeed.tech/tags/align.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [anchor](<https://devfeed.tech/tags/anchor.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [learning](<https://devfeed.tech/tags/learning.md>), [llm](<https://devfeed.tech/tags/llm.md>), [local](<https://devfeed.tech/tags/local.md>), [margin](<https://devfeed.tech/tags/margin.md>), [noise](<https://devfeed.tech/tags/noise.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [optimizing](<https://devfeed.tech/tags/optimizing.md>), [pair](<https://devfeed.tech/tags/pair.md>), [performance](<https://devfeed.tech/tags/performance.md>), [policy](<https://devfeed.tech/tags/policy.md>), [radius](<https://devfeed.tech/tags/radius.md>), [robust](<https://devfeed.tech/tags/robust.md>)

## AI overview

GAPO replaces DPO's fixed reference policy with a small, adversarial local perturbation of the current policy as a pessimistic anchor. It uses the reward difference between policy and anchor--the Anchor Gap--to reweight preference pairs by local sensitivity, reducing the influence of geometrically brittle examples and emphasizing robust signals. Under smoothness conditions, the gap approximates worst-case local margin degradation. The abstract reports improved robustness across diverse noise settings while matching or improving performance on standard LLM alignment and reasoning benchmarks; it does not provide quantitative results or benchmark details.

## Source excerpt

arXiv:2602.04909v5 Announce Type: replace Abstract: Direct Preference Optimization (DPO) and related methods align large language models from pairwise preferences by regularizing updates against a fixed reference policy. As the policy drifts, a static reference, however, can become increasingly miscalibrated, leading to distributional mismatch and amplifying spurious preference signals under noisy supervision. Conversely, reference-free variants avoid mismatch but often suffer from unconstrained reward drift. We propose Geometric Anchor Preference Optimization (GAPO), which replaces the fixed reference with a dynamic, geometry-aware anchor: an adversarial local perturbation of the current policy within a small radius that serves as a pessimistic baseline. This anchor enables an adaptive reweighting mechanism, modulating the importance of each preference pair based on its local sensitivity. We further introduce the Anchor Gap, the reward discrepancy between the policy and its anchor, and show under smoothness conditions that it approximates worst-case local margin degradation. Optimizing a logistic objective weighted by this gap downweights geometrically brittle instances while emphasizing robust preference signals. Across diverse noise settings, GAPO consistently improves robustness while matching or improving performance on standard LLM alignment and reasoning benchmarks.