# Objective-Conditioned Active Preference Learning for Multi-Objective Alignment

DevFeed: [Objective-Conditioned Active Preference Learning for Multi-Objective Alignment](<https://devfeed.tech/articles/objective-conditioned-active-preference-learning-for-multi-objective-alignment-118331.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05506>)

Author: Duc Anh Vu, Thong Nguyen, Hoang Pham, Ponhvoan Srey, Luu Anh Tuan

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [annotation](<https://devfeed.tech/tags/annotation.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [can](<https://devfeed.tech/tags/can.md>), [change](<https://devfeed.tech/tags/change.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [cost](<https://devfeed.tech/tags/cost.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [effectiveness](<https://devfeed.tech/tags/effectiveness.md>), [either](<https://devfeed.tech/tags/either.md>), [existing](<https://devfeed.tech/tags/existing.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [external](<https://devfeed.tech/tags/external.md>), [framework](<https://devfeed.tech/tags/framework.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [learning](<https://devfeed.tech/tags/learning.md>)

## AI overview

OGC-DPO reduces preference-labeling costs for multi-objective LLM alignment by treating each response-pair/objective combination as a query and selecting queries that expand gradient coverage for each objective. It estimates informativeness from the policy's implicit gradient, without an external reward model. In experiments across varying numbers of objectives, it outperformed existing active preference-learning methods at the same annotation cost. The authors also report that informative conflicts between preferences can provide a useful training signal, rather than being uniformly harmful.

## Source excerpt

arXiv:2611.05506v1 Announce Type: cross Abstract: Aligning large language models (LLMs) with several objectives, such as helpfulness, harmlessness, and honesty, typically relies on preference judgment for each objective, growing the annotation cost linearly with the number of objectives. Active preference learning can reduce this cost by selecting the comparisons whose labels are expected to change the policy most. Existing remedies either lack an objective-aware query selection mechanism for the multi-objective setting or regard conflicting preferences as harmful, therefore excluding them from the training data. We introduce OGC-DPO (Objective-conditioned Gradient Coverage DPO), an active learning framework that treats each (response pair, objective) combination as a separate query and measures its informativeness through the policy's own implicit gradient, without relying on an external reward model. OGC-DPO then selects the queries that most expand each objective's gradient coverage. Empirical experiments across varying numbers of objectives demonstrate the effectiveness of OGC-DPO over existing active preference learning methods under the same annotation cost. Further analysis demonstrates that informative conflicts, rather than being uniformly harmful, provide a useful policy training signal.