# The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention

DevFeed: [The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention](<https://devfeed.tech/articles/the-alignment-tax-is-graded-task-selective-capability-loss-under-length-normalized-dpo-with-sft-retention-120535.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.06476>)

Author: Ye Yuan

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [lora](<https://devfeed.tech/topics/lora.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [characterization](<https://devfeed.tech/tags/characterization.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [financial](<https://devfeed.tech/tags/financial.md>), [format](<https://devfeed.tech/tags/format.md>), [json](<https://devfeed.tech/tags/json.md>), [lora](<https://devfeed.tech/tags/lora.md>), [margin](<https://devfeed.tech/tags/margin.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [model](<https://devfeed.tech/tags/model.md>), [objectives](<https://devfeed.tech/tags/objectives.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [retention](<https://devfeed.tech/tags/retention.md>), [sft](<https://devfeed.tech/tags/sft.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [tax](<https://devfeed.tech/tags/tax.md>), [testing](<https://devfeed.tech/tags/testing.md>)

## AI overview

In a study of Qwen3.5-4B on three structured financial-filing tasks, increasing the implicit reward margin under length-normalized DPO with an SFT retention term produced task-selective capability loss. Across 8 seeds, extraction was preserved within a ±5-point equivalence margin (−0.55 points); JSON performance fell 3.52 points, with equivalence not established; and router performance fell 30 points, mainly in a margin 1-3 damage window that included a transient format-break spike. At least 96% of margin growth came from suppressing rejected completions while chosen reward stayed approximately flat; the study does not assign the behavioral damage to a single gradient channel. Measured reward margins differed by up to 3.4x between nf4 training and fp16 evaluation on the unquantized base. These findings are limited to one 4B model, sparse LoRA, easily separable synthetic preferences, and the tested objective; the authors leave a no-retention control and measured format-alignment metric for future work.

## Source excerpt

arXiv:2611.06476v2 Announce Type: replace Abstract: Setting. This is an empirical characterization study; we propose no new method and run no comparisons against alternative objectives. We fine-tune Qwen3.5-4B with sparse LoRA and length-normalized DPO plus an SFT retention term (the RPO objective) on structured financial-filing tasks, and track downstream capability as the implicit reward margin grows to $\sim 7$ $\beta$-scaled units (nf4 training; $\sim 3$ under fp16 measurement on the unquantized base). Question. Does margin growth degrade capability uniformly across tasks, or selectively? Findings. In a pre-registered study (8 seeds, 6 unseen at freeze) with artifact-aware scoring and TOST equivalence testing, the alignment tax is graded and task-selective: extraction is preserved ($\Delta = -0.55$ pts, equivalent at $\pm5$ pts), json loses little ($\Delta = -3.52$ pts, equivalence fails on a clustered item-level interval), and router degrades severely ($\Delta = -30$ pts). Router's loss concentrates in a margin 1-3 damage window containing a transient format-break spike. The margin is accounted for almost entirely ($\geq 96\%$) by suppression of rejected completions, with chosen reward approximately flat, consistent with the retention term anchoring it; we do not attribute the behavioral damage to any single gradient channel. Gold-logp deflation is task-selective (extraction $-0.15$, json $-1.04$, router $-1.47$ nats/token under fp16), with gold selectively protected relative to incorrect controls on extraction (8/8 seeds). QLoRA reward margins differ up to $3.4\times$ between nf4 training and unquantized fp16 measurement, with an axis-dependent gap. Scope. Length-normalized rewards with SFT retention, sparse LoRA on 8/32 layers, one 4B model, easily-separable synthetic preferences, three structured tasks. A no-retention control and a measured format-alignment metric are left to future work.