# Exploring the Limits of the Delta Learning Hypothesis: Single-Pair Generator Biases in Direct Preference Optimization

DevFeed: [Exploring the Limits of the Delta Learning Hypothesis: Single-Pair Generator Biases in Direct Preference Optimization](<https://devfeed.tech/articles/exploring-the-limits-of-the-delta-learning-hypothesis-single-pair-generator-biases-in-direct-preference-optimization-103226.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.37058>)

Author: Giuseppe Curci, Stefano Simonazzi, Andrea Molinari, Andrea Zugarini

Published: 2026-10-28T04:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [diversity](<https://devfeed.tech/tags/diversity.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [generator](<https://devfeed.tech/tags/generator.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>)

## AI overview

The study finds that DPO trained on preference data from a fixed pair of models underperforms training on data from multiple models, even when the larger model produces higher-quality responses. Across model families and scales, fixed pairs introduce generator-specific biases that predict preference labels independently of response quality; the authors link the performance gap to training dynamics consistent with reward hacking on those biases. Adding more generators partially mitigates the effect, even without an external judge. The findings suggest generator diversity matters when constructing preference data, while the abstract does not establish that it fully removes the bias.

## Source excerpt

arXiv:2610.37058v1 Announce Type: cross Abstract: Direct Preference Optimization (DPO) typically relies on preference pairs generated from diverse model pools and ranked by an external LLM judge. The recently proposed delta-learning hypothesis suggests that comparable supervision can instead be obtained from a single pair of models by treating the larger model's response as preferred. We investigate the limits of this approach across model families and scales. We show, both theoretically and empirically, that fixed model pairs introduce systematic, generator-dependent biases that are predictive of the preference label independent of response quality. Consequently, DPO trained on single-pair data consistently underperforms multi-model preference data, despite the larger generator producing higher-quality responses, and we trace this gap to training dynamics consistent with reward hacking on these biases. We further show that simply increasing the number of generators, even without an external judge, is enough to partially mitigate this effect. Our results highlight generator diversity as a critical factor in preference-data construction.