# Progress-Guided Preference Generation for Aligning Generative Control Policies

DevFeed: [Progress-Guided Preference Generation for Aligning Generative Control Policies](<https://devfeed.tech/articles/progress-guided-preference-generation-for-aligning-generative-control-policies-93531.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.25692>)

Author: Tung M. Luu, Tri Ton, Duc M. Nguyen, Xuan Thanh Nguyen, Nguyen Gia Binh, Quang Tan Nguyen, Vu Duong, Daniel Sonntag, Duy M. H. Nguyen, Vien Anh Ngo, Chang D. Yoo

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [annotations](<https://devfeed.tech/tags/annotations.md>), [automatic](<https://devfeed.tech/tags/automatic.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [flow-matching](<https://devfeed.tech/tags/flow-matching.md>), [generative](<https://devfeed.tech/tags/generative.md>), [imitation](<https://devfeed.tech/tags/imitation.md>), [manipulation](<https://devfeed.tech/tags/manipulation.md>), [performance](<https://devfeed.tech/tags/performance.md>), [policies](<https://devfeed.tech/tags/policies.md>), [reward](<https://devfeed.tech/tags/reward.md>)

## AI overview

PGPG generates preference labels for aligning pretrained generative control policies without human annotations or manually designed reward functions. It measures sequential progress in autonomously collected rollouts against a small set of expert demonstrations using ordered coverage alignment, then uses those pseudo-preferences for fine-tuning. The abstract reports better performance than standard supervised fine-tuning baselines on challenging long-horizon manipulation tasks and applicability to both diffusion and flow-matching policies. The supplied text does not provide quantitative results or implementation details.

## Source excerpt

arXiv:2610.25692v1 Announce Type: new Abstract: Generative policies have achieved strong performance in imitation learning tasks, but their success is often constrained by the limited availability of expert demonstrations. Although preference-based fine-tuning can leverage suboptimal rollouts to improve performance, existing approaches heavily rely on expensive human annotations. In this work, we introduce Progress-Guided Preference Generation (PGPG), an unsupervised preference alignment framework that eliminates the need for human-in-the-loop supervision. Our method leverages ordered coverage alignment to measure sequential progress relative to a small set of expert demonstrations, enabling the automatic extraction of pseudo-preference labels from autonomously collected rollouts. Furthermore, we introduce key enhancements to improve the reliability of these progress-guided preferences, enabling effective fine-tuning of pretrained generative policies without requiring manually designed reward functions. Extensive experiments across a suite of challenging long-horizon manipulation tasks demonstrate that PGPG outperforms standard supervised fine-tuning baselines. Moreover, our conceptually simple framework is broadly applicable across multiple classes of generative policies, demonstrating strong versatility when integrated with both diffusion and flow-matching policies.