# Human or Synthetic? What a Fixed Budget Buys for Post-Training

DevFeed: [Human or Synthetic? What a Fixed Budget Buys for Post-Training](<https://devfeed.tech/articles/human-or-synthetic-what-a-fixed-budget-buys-for-post-training-133156.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16895>)

Author: Shourik Banerjee, Ethan Mendes, Mohit Raghavendra, Wei Xu, Alan Ritter

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [post-training](<https://devfeed.tech/topics/post-training.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [human feedback](<https://devfeed.tech/topics/human-feedback.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [annotation](<https://devfeed.tech/tags/annotation.md>), [budget](<https://devfeed.tech/tags/budget.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [deduplication](<https://devfeed.tech/tags/deduplication.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [human-data](<https://devfeed.tech/tags/human-data.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [performance](<https://devfeed.tech/tags/performance.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [sft](<https://devfeed.tech/tags/sft.md>), [supervised-fine-tuning](<https://devfeed.tech/tags/supervised-fine-tuning.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [synthetic-data](<https://devfeed.tech/tags/synthetic-data.md>)

## AI overview

Under a dollar-matched annotation budget, synthetic preference data was more cost-efficient than human-annotated data at low to moderate spending in the tested post-training experiments. It achieved high win rates with $200-$400; on helpfulness, human data closed the gap only at budgets roughly two orders of magnitude larger, while on summarization it did not close the gap at any tested budget. Results also varied between human preference sources: one widely used source underperformed another, and the difference persisted after domain restriction and deduplication. The abstract reports evaluations across multiple base model families and tasks but does not provide further methodological detail.

## Source excerpt

arXiv:2611.16895v1 Announce Type: cross Abstract: Post-training of large language models typically combines supervised fine-tuning (SFT) with preference-based fine-tuning such as Direct Preference Optimization (DPO). Human-annotated and synthetically generated preference data differ substantially in per-sample cost, but prior work typically holds sample count fixed when comparing the two. In this work, we explicitly study the interplay between cost and performance by comparing the effect of post-training with human and synthetic preference data under a fixed, dollar-matched annotation budget. We train and evaluate multiple base model families across helpfulness and summarization tasks. We find that synthetic data is far more cost-efficient at low-to-moderate budgets, achieving high win rates with as little as \$200--\$400. On helpfulness, human data closes the gap only at budgets roughly two orders of magnitude larger; on summarization, it does not close the gap at any tested budget. We also find that human-annotated data varies in reliability: one widely used human preference source substantially underperforms another, and the gap persists even after standard remedies such as domain restriction and deduplication.