# dpo

Direct Preference Optimization (DPO) is a language-model alignment and fine-tuning technique that adjusts model weights using human preference data without fitting an explicit reward model.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Goal-Guided Reinforcement Learning via Fine-Tuning Vision-Language Models

DevFeed: [Goal-Guided Reinforcement Learning via Fine-Tuning Vision-Language Models](<https://devfeed.tech/articles/goal-guided-reinforcement-learning-via-fine-tuning-vision-language-models-145097.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25742>)

Author: Yajie Qi, Wei Wei, Lin Li, Lijun Zhang, Jianye Hao

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [kl-regularization](<https://devfeed.tech/tags/kl-regularization.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>)

### AI overview

The proposed framework fine-tunes a vision-language model on CCGP-2M, a multimodal goal-preference dataset built from Craftax-Classic interaction trajectories, then uses the adapted model to guide online reinforcement learning. Offline fine-tuning uses KLTDPO, combining token-level KL regularization with distributionally robust reweighting. In Craftax-Classic experiments, the method collected diamonds earlier than state-of-the-art baselines, achieved the highest score and reward, and had the highest success rates on most achievements. The source does not provide details about the baselines or quantitative effect sizes.

### Source excerpt

arXiv:2611.25742v1 Announce Type: new Abstract: Exploration remains a major challenge for reinforcement learning (RL) agents in Craftax-style environments due to sparse rewards and long horizons. Large language models (LLMs) can guide exploration by generating high-level goals, but methods using models without explicit task adaptation often rely on carefully designed prompts and may produce inconsistent or contextually inappropriate goals. We introduce CCGP-2M, a multimodal goal-preference dataset constructed from interaction trajectories in Craftax-Classic, and propose a vision-language model (VLM)-guided RL framework that learns environment-specific goal guidance from this dataset. The framework consists of two stages: offline VLM fine-tuning and online goal-guided RL. In the offline stage, we align the VLM with goal preferences using Kullback-Leibler Token-level Direct Preference Optimization (KLTDPO), which combines token-level KL regularization with distributionally robust reweighting. Experiments on Craftax-Classic show that our method collects diamonds earlier than state-of-the-art baselines, achieves the highest score and reward, and attains the highest success rates on most achievements.

## Swiss-Knife: A Framework for Reconfigurable Externalised Multi-Objective Alignment at Decode Time

DevFeed: [Swiss-Knife: A Framework for Reconfigurable Externalised Multi-Objective Alignment at Decode Time](<https://devfeed.tech/articles/swiss-knife-a-framework-for-reconfigurable-externalised-multi-objective-alignment-at-decode-time-143311.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.22226>)

Author: Agnibh Karmakar, Mayur Parvatikar, Shreyash Dhoot, Amit Dhanda, Aman Chadha, Kapil Wanaskar, Vinija Jain, Amitava Das

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [peft](<https://devfeed.tech/topics/peft.md>), [lora](<https://devfeed.tech/topics/lora.md>), [post-training](<https://devfeed.tech/topics/post-training.md>)

Tags: [adversarial](<https://devfeed.tech/tags/adversarial.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [arm](<https://devfeed.tech/tags/arm.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [batch](<https://devfeed.tech/tags/batch.md>), [best](<https://devfeed.tech/tags/best.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [contamination](<https://devfeed.tech/tags/contamination.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decode](<https://devfeed.tech/tags/decode.md>), [design](<https://devfeed.tech/tags/design.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [external](<https://devfeed.tech/tags/external.md>), [f](<https://devfeed.tech/tags/f.md>), [family](<https://devfeed.tech/tags/family.md>), [form](<https://devfeed.tech/tags/form.md>), [framework](<https://devfeed.tech/tags/framework.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

Swiss-Knife treats decode-time alignment as a reconfigurable runtime specification combining hot-swappable scoring models, batch normalization, pairwise aggregation and a selection rule. Its representation theorem describes admissible aggregators as a two-parameter family that includes probit, logistic and pointwise argmax rules. The authors report that pairwise aggregation is stable under adversarial reward contamination, whereas argmax is not; Candidate-Batch Normalization (CBN) makes objective weights invariant to reward-model rescaling. In their DPO-LoRA and uncertainty-aware tournament instantiation, the system achieved harmonic F1 0.797 versus 0.750 for the strongest of six methods, with the lowest refusal rate and highest helpfulness reported. Objective reconfiguration took 0.05 ms without gradient computation. Removing CBN reduced F1 by 0.217; the authors attribute the collapse to a low-variance blade retaining only 9% of its nominal influence. These are claims from the supplied abstract excerpt.

### Source excerpt

arXiv:2609.22226v2 Announce Type: replace-cross Abstract: Decode-time alignment methods steer a frozen language model by scoring candidate continuations with an external reward and selecting the maximiser. We argue that this shared design is a single degenerate point in a much larger space. We introduce Swiss-Knife, a framework for externalised multi-objective alignment in which the alignment specification is a first-class runtime object: hot-swappable scoring blades, a batch normaliser, a pairwise aggregation operator, and a selection rule. Six axioms characterise the admissible aggregation operators, and we prove a representation theorem: every operator satisfying them has the form $R_i = \sum_{j \neq i} g((\mu_i - \mu_j)/s(\sigma_i,\sigma_j))$, a two-parameter family containing probit and logistic comparison rules and pointwise argmax as named coordinates. Within it, pairwise aggregation is Lipschitz-stable under adversarial reward contamination while argmax is not, and Candidate-Batch Normalization (CBN) makes the weight simplex invariant to the rescalings under which reward models are only ever identified. Our reference instantiation pairs DPO-LoRA blades with an uncertainty-aware pairwise tournament. Sweeping the helpfulness/honesty/harmlessness simplex, it attains the best balanced frontier of six decode-time methods (harmonic $F_1$ 0.797 vs. 0.750 for the strongest baseline, $p < 10^{-14}$) with the lowest refusal rate and highest helpfulness of any arm, and reconfigures its objectives in 0.05 ms with no gradient computation. Ablating CBN costs 0.217 $F_1$, and the metadata confirms the predicted mechanism: the lowest-variance blade retains 9% of its nominal 33% influence, and the collapse follows the predicted ordering.

## Margin Dynamics for Large Language Model Alignment

DevFeed: [Margin Dynamics for Large Language Model Alignment](<https://devfeed.tech/articles/margin-dynamics-for-large-language-model-alignment-139904.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21766>)

Author: Xingzi Xu, Mehmet Saygin Seyfioglu, Karim Bouyarmane

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [design](<https://devfeed.tech/tags/design.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [gradient-flow](<https://devfeed.tech/tags/gradient-flow.md>), [ipo](<https://devfeed.tech/tags/ipo.md>), [label-smoothing](<https://devfeed.tech/tags/label-smoothing.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [llm](<https://devfeed.tech/tags/llm.md>), [margin](<https://devfeed.tech/tags/margin.md>), [model](<https://devfeed.tech/tags/model.md>), [model-alignment](<https://devfeed.tech/tags/model-alignment.md>), [models](<https://devfeed.tech/tags/models.md>), [noise](<https://devfeed.tech/tags/noise.md>), [ode](<https://devfeed.tech/tags/ode.md>), [prediction](<https://devfeed.tech/tags/prediction.md>)

### AI overview

The paper models pairwise alignment methods as flows over preferred and dispreferred completion probabilities, reducing each pair to a margin equation with a closed-form solution. It reports that DPO margins grow logarithmically without bound under gradient flow, while IPO margins approach 1/(2τ); label noise changes DPO from unbounded to bounded growth but affects IPO quantitatively. The authors also find that asymmetric weighting can concentrate gradients and affect trainability. For SimPO, any positive label smoothing ε yields a finite margin attractor, whereas standard SimPO (ε=0) has none. Tests across five model families and four benchmarks support the first predictions; in three 8B-9B models, TPO-S with ε=0.05 was competitive with strong baselines and had the highest mean IFEval score among compared methods on each. The authors describe it as a usable operating point, not a new state-of-the-art loss.

### Source excerpt

arXiv:2611.21766v1 Announce Type: cross Abstract: Practitioners choose among LLM alignment methods largely by trial and error. We provide a theoretical foundation: each pairwise method defines a vector field on the probability simplex that governs how the model redistributes mass between preferred and dispreferred completions. Projecting onto a single preferred/dispreferred pair reduces this to a one-dimensional margin ODE with a closed-form solution. These solutions reveal that DPO margins grow without bound (logarithmically as $O(\log t)$ in the gradient-flow limit) while IPO margins converge to a finite $1/(2\tau)$; that label noise induces a qualitative phase transition in DPO (unbounded $\to$ bounded) but affects IPO only quantitatively; and that gradient concentration under asymmetric weighting governs trainability at scale. We confirm all three predictions across five model families spanning four benchmarks. The closed-form solutions also enable principled method design: we prove that adding any label smoothing $\varepsilon > 0$ to SimPO creates a finite margin attractor at $\Delta^* = \beta^{-1}\log((1-\varepsilon)/\varepsilon)$, while $\varepsilon = 0$ (standard SimPO) has no attractor. We test this across three models (8B-9B) and confirm the prediction: the theory-derived configuration (TPO-S, $\varepsilon = 0.05$) is competitive with the strongest baselines, attaining the highest mean IFEval score among the compared methods on all three, validating that the predicted finite attractor is a usable operating point rather than proposing a new state-of-the-art loss.

## Sample-efficient Personalization of Large Language Model Alignment with Multidimensional User Preference Data

DevFeed: [Sample-efficient Personalization of Large Language Model Alignment with Multidimensional User Preference Data](<https://devfeed.tech/articles/sample-efficient-personalization-of-large-language-model-alignment-with-multidimensional-user-preference-data-141748.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22210>)

Author: Zhen-Yu Zhang, Jerry Huang, Gang Niu, Takashi Ishida, Masashi Sugiyama

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [automated](<https://devfeed.tech/tags/automated.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [human-feedback](<https://devfeed.tech/tags/human-feedback.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [online](<https://devfeed.tech/tags/online.md>), [personalization](<https://devfeed.tech/tags/personalization.md>), [queries](<https://devfeed.tech/tags/queries.md>), [user](<https://devfeed.tech/tags/user.md>)

### AI overview

MPG-DPO aims to reduce costly human preference queries by using low-cost multidimensional evaluations from automated verifiers on a user's past comparisons to estimate personalized preferences. It removes responses confidently dominated under that estimate, then asks the user to compare two retained responses that are competitive under the preference cone and show the strongest unresolved disagreement. Theoretical analysis says that, with high probability, queries spent on cone-dominated responses decline toward zero as learning progresses. Experiments across multiple LLM families report better performance than strong contenders at matched query budgets. The supplied text is an abstract and does not provide implementation details or experiment results.

### Source excerpt

arXiv:2611.22210v1 Announce Type: new Abstract: Preference alignment has become a central tool to align large language models (LLMs) with human values and desired behavior, and its online variant can further advance their capabilities beyond what static data alone offer by repeatedly generating and labeling new responses with human feedback. But costly, time-consuming annotations limit its scalability, making sample efficiency essential, and existing methods typically seek to improve it by expanding coverage of the response space or selecting pairs with the highest uncertainty for comparison. Although they are effective, these methods quantify uncertainty from the perspective of the LLM, which may differ from the uncertainty perceived by the user: a pair can appear highly uncertain to the LLM even when past comparisons from the user already imply a clear ordering. In this paper, we propose multidimensional preference-guided online direct preference optimization (MPG-DPO), which leverages low-cost multidimensional evaluations from automated verifiers on the user historical comparisons to construct a personalized preference cone guiding response selection for online preference alignment. Specifically, it first discards responses that are confidently dominated under the estimated user preference, then selects two responses that are competitive under the retained preference cone yet exhibit the strongest unresolved disagreement. Theoretical analysis shows that, with high probability, the fraction of queries spent on cone-dominated responses vanishes as learning progresses. Experiments across multiple LLM families show that MPG-DPO outperforms strong contenders under matched query budgets.

## Say Less, Hallucinate Less? Diagnosing and Restoring Object Coverage in Preference-Optimized Vision-Language Models

DevFeed: [Say Less, Hallucinate Less? Diagnosing and Restoring Object Coverage in Preference-Optimized Vision-Language Models](<https://devfeed.tech/articles/say-less-hallucinate-less-diagnosing-and-restoring-object-coverage-in-preference-optimized-vision-language-models-140127.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23179>)

Author: Hansang Lee

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [dpo](<https://devfeed.tech/topics/dpo.md>)

Tags: [computer-vision](<https://devfeed.tech/tags/computer-vision.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>)

### AI overview

Four public preference-optimized LLaVA-1.5-7B checkpoints name fewer object categories, and their lower hallucination rates do not significantly improve set-level object F1 on CHAIR. On two disjoint COCO sets, increasing the base model's EOS logit bias to stop earlier reproduces 58-86% of the CHAIRi gain for three checkpoints. Longer captions do not restore missed categories; under greedy decoding, negative EOS bias recovers only 2-9% of lost CHAIR recall, mostly by repeating categories already mentioned. The proposed training-free method appends high-confidence categories found in the base model's caption but absent from the PO caption. With thresholds fixed on development data, it brings three checkpoints within one point of base recall on held-out COCO while lowering distinct-object hallucination versus the base model. Gains over recall-matched early stopping hold for RLAIF-V near base recall and for RLAIF-V and CHAIR-DPO at intermediate recall.

### Source excerpt

arXiv:2611.23179v1 Announce Type: cross Abstract: Preference optimization (PO) is the standard way to reduce object hallucination in large vision-language models. We find that four public PO checkpoints of LLaVA-1.5-7B also lower object coverage. They name fewer object categories, and their lower hallucination rates bring no significant gain in set-level object F1 on CHAIR. We therefore ask how much of the PO gain the base model can reproduce by stopping earlier. A positive end-of-sequence (EOS) logit bias lets the base model reach the recall of each PO model, and this early stopping alone reproduces 58 to 86 percent of the CHAIRi gain of three checkpoints on two disjoint COCO sets. Writing longer does not bring the lost categories back. Under greedy decoding, a negative EOS bias recovers only 2 to 9 percent of the lost CHAIR recall of these checkpoints, and 94 to 96 percent of the added object mentions repeat categories already named. On CHAIR, the base caption of the same image still names about half of the missed object categories. We therefore propose base-guided object-category restoration, which appends high-confidence base-only categories to the PO caption without training. With thresholds fixed on a development set, it brings three checkpoints within one point of the base recall on a held-out COCO set with a lower distinct-object hallucination rate than the base model. Against the recall-matched base stopping curve, its gain holds for RLAIF-V near the base recall and for RLAIF-V and CHAIR-DPO at an intermediate recall. We will release the code of the recall-controlled evaluation.

## Arabic Safety Alignment as Selective Refusal: An Empirical Study of SFT, DPO, and Guard Calibration

DevFeed: [Arabic Safety Alignment as Selective Refusal: An Empirical Study of SFT, DPO, and Guard Calibration](<https://devfeed.tech/articles/arabic-safety-alignment-as-selective-refusal-an-empirical-study-of-sft-dpo-and-guard-calibration-139811.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.29378>)

Author: Mohamad Zbib, Mohamad Bazzi, Ammar Mohanna

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [dpo](<https://devfeed.tech/topics/dpo.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [arabic](<https://devfeed.tech/tags/arabic.md>), [computer-science](<https://devfeed.tech/tags/computer-science.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [guard](<https://devfeed.tech/tags/guard.md>), [guards](<https://devfeed.tech/tags/guards.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [safety](<https://devfeed.tech/tags/safety.md>), [sft](<https://devfeed.tech/tags/sft.md>), [study](<https://devfeed.tech/tags/study.md>)

### AI overview

The study finds that Arabic safety alignment requires choosing a model-specific balance between refusing harmful prompts (H) and avoiding refusals of benign prompts (B). Across five models and 130 runs on AraSafe, refusal-only SFT tended toward blanket refusal, while selected mixed-SFT settings achieved H of 90-93% with B of 14-23%; four settings cleared the H>=90% target in all three runs, and Fanar did so in two. DPO and inference guards shifted B and H differently across models, so they were not consistent upgrades. On Arabizi, selected SFT raised H for all five models, but none reached 90%, indicating partial transfer from Modern Standard Arabic. In a 300-response audit, Qwen3Guard and Aya Expanse 32B scored 88.7% and 91.0% accuracy, with no conclusive paired difference. The abstract recommends retaining interventions only when they meet a defined deployment target.

### Source excerpt

arXiv:2608.29378v2 Announce Type: replace-cross Abstract: Arabic large language models must refuse harmful prompts without over-refusing benign or sensitive prompts, yet a single refusal rate hides this trade-off. We evaluate it using benign refusal B and harmful-prompt refusal H, where H measures refusal rather than harmful compliance. Across five Arabic-capable models and 130 runs on the full human-written AraSafe set, refusal-only supervised fine-tuning (SFT) collapses toward blanket refusal, whereas selected mixed-SFT configurations reach H = 90% to 93% at B = 14% to 23%; four selected configurations exceed the H = 90% target in all three runs, while Fanar does so in two of three. Direct Preference Optimization (DPO) and inference guards change B and H differently across models rather than acting as uniform upgrades. In a blinded 300-response audit, annotator binary-refusal agreement is 89.0% (kappa = 0.78); Qwen3Guard and Aya Expanse 32B reach 88.7% and 91.0% accuracy, respectively, with no conclusive paired difference. Selected SFT raises H on Arabizi for all five models, but none reaches 90%, showing only partial transfer from Modern Standard Arabic. Overall, the results support model-specific operating-point selection: set a deployment target and retain only interventions that improve it.

## Policy Optimization With Multiple Verifiers via Pareto Rankings

DevFeed: [Policy Optimization With Multiple Verifiers via Pareto Rankings](<https://devfeed.tech/articles/policy-optimization-with-multiple-verifiers-via-pareto-rankings-137387.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19064>)

Author: Kai Yamashita, Shohei Taniguchi, Hiroki Furuta, Masahiro Suzuki, Yutaka Matsuo

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [rlvr](<https://devfeed.tech/topics/rlvr.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [dpo](<https://devfeed.tech/topics/dpo.md>)

Tags: [code-generation](<https://devfeed.tech/tags/code-generation.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [improvements](<https://devfeed.tech/tags/improvements.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [learning](<https://devfeed.tech/tags/learning.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [policy](<https://devfeed.tech/tags/policy.md>), [probability](<https://devfeed.tech/tags/probability.md>), [rankings](<https://devfeed.tech/tags/rankings.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>), [rlvr](<https://devfeed.tech/tags/rlvr.md>), [supervision](<https://devfeed.tech/tags/supervision.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

POLPO trains language models from multiple verifier outcomes without collapsing them into a scalar reward. It uses Pareto dominance to order responses that satisfy additional requirements while preserving already satisfied ones, leaving conflicting responses incomparable. Its objective maximizes the probability of complete rankings consistent with these partial orders, extending the listwise likelihood underlying Direct Preference Optimization. The abstract reports improved learning efficiency and higher complete-requirement satisfaction than scalar-reward training on code generation and instruction-following experiments.

### Source excerpt

arXiv:2611.19064v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has become an important approach to improving the reasoning capabilities of large language models. In tasks such as code generation and instruction following, multiple verifiers evaluate whether a response satisfies different requirements. Their outcomes provide detailed feedback about partial success, yet this feedback is commonly aggregated into a scalar reward for policy optimization. Rewarding only complete success discards distinctions among partial solutions, while counts and weighted sums impose comparisons between responses that satisfy different subsets of requirements. We introduce Partial-Order Likelihood Policy Optimization (POLPO), which learns from the comparisons established by verification without first constructing an aggregate reward. Pareto dominance identifies responses that satisfy additional requirements without losing any already satisfied, while leaving responses with conflicting outcomes incomparable. POLPO extends the listwise likelihood underlying Direct Preference Optimization by maximizing the total probability of all complete rankings consistent with these partial orders. This objective provides supervision from verified improvements even before complete solutions are available, without requiring a prescribed ordering of incomparable responses. Experiments on code generation and instruction following demonstrate improved learning efficiency and higher rates of complete requirement satisfaction compared with scalar-reward training. These results support partial-order likelihood as an effective approach to learning from multiple verifiers.

## Contrastive Activation-Guided Zeroth-Order Preference Optimisation for Memory-Efficient LLM Alignment

DevFeed: [Contrastive Activation-Guided Zeroth-Order Preference Optimisation for Memory-Efficient LLM Alignment](<https://devfeed.tech/articles/contrastive-activation-guided-zeroth-order-preference-optimisation-for-memory-efficient-llm-alignment-139077.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20635>)

Author: Weiyun Huang, Alessio Galatolo

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [backpropagation](<https://devfeed.tech/tags/backpropagation.md>), [behaviour](<https://devfeed.tech/tags/behaviour.md>), [ca](<https://devfeed.tech/tags/ca.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-memory](<https://devfeed.tech/tags/gpu-memory.md>), [hh](<https://devfeed.tech/tags/hh.md>), [human-feedback](<https://devfeed.tech/tags/human-feedback.md>), [learning](<https://devfeed.tech/tags/learning.md>), [llm](<https://devfeed.tech/tags/llm.md>), [memory](<https://devfeed.tech/tags/memory.md>), [momentum](<https://devfeed.tech/tags/momentum.md>), [optimisation](<https://devfeed.tech/tags/optimisation.md>), [order](<https://devfeed.tech/tags/order.md>)

### AI overview

CA-ZODPO reduces activation-memory demands for preference alignment by restricting zeroth-order perturbations to a low-rank subspace built from activation differences between chosen and rejected responses, and keeping momentum in those low-rank coordinates. On Qwen2.5-1.5B with Anthropic HH-RLHF, it outperforms common zeroth-order baselines and is competitive with first-order DPO in pairwise preference evaluation. The abstract reports about 59% lower peak GPU memory at sequence length 2048, while retaining zeroth-order training's sequence-length scaling behavior.

### Source excerpt

arXiv:2611.20635v1 Announce Type: new Abstract: Direct Preference Optimisation (DPO) simplifies reinforcement learning from human feedback, but still relies on traditional backpropagation and its associated activation-memory cost. Zeroth-order (ZO) optimisation avoids backpropagation, yet isotropic perturbations can produce noisy updates for preference objectives defined by differences between chosen and rejected responses. We introduce Contrastive Activation-guided Zeroth-Order Direct Preference Optimisation (CA-ZODPO), which constructs a low-rank search subspace from the activation discrepancy between the chosen and rejected responses in each preference pair. Perturbations are restricted to this subspace, while momentum is maintained in low-rank coordinates to preserve the memory advantages of ZO training. On Qwen2.5-1.5B with Anthropic HH-RLHF, CA-ZODPO outperforms common ZO baselines and remains competitive with first-order DPO in pairwise preference evaluation. It also reduces peak GPU memory by about 59\% at a sequence length of 2048 while retaining the sequence-length scaling behaviour of zeroth-order training.

## Human or Synthetic? What a Fixed Budget Buys for Post-Training

DevFeed: [Human or Synthetic? What a Fixed Budget Buys for Post-Training](<https://devfeed.tech/articles/human-or-synthetic-what-a-fixed-budget-buys-for-post-training-133156.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16895>)

Author: Shourik Banerjee, Ethan Mendes, Mohit Raghavendra, Wei Xu, Alan Ritter

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [post-training](<https://devfeed.tech/topics/post-training.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [human feedback](<https://devfeed.tech/topics/human-feedback.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [annotation](<https://devfeed.tech/tags/annotation.md>), [budget](<https://devfeed.tech/tags/budget.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [deduplication](<https://devfeed.tech/tags/deduplication.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [human-data](<https://devfeed.tech/tags/human-data.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [performance](<https://devfeed.tech/tags/performance.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [sft](<https://devfeed.tech/tags/sft.md>), [supervised-fine-tuning](<https://devfeed.tech/tags/supervised-fine-tuning.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [synthetic-data](<https://devfeed.tech/tags/synthetic-data.md>)

### AI overview

Under a dollar-matched annotation budget, synthetic preference data was more cost-efficient than human-annotated data at low to moderate spending in the tested post-training experiments. It achieved high win rates with $200-$400; on helpfulness, human data closed the gap only at budgets roughly two orders of magnitude larger, while on summarization it did not close the gap at any tested budget. Results also varied between human preference sources: one widely used source underperformed another, and the difference persisted after domain restriction and deduplication. The abstract reports evaluations across multiple base model families and tasks but does not provide further methodological detail.

### Source excerpt

arXiv:2611.16895v1 Announce Type: cross Abstract: Post-training of large language models typically combines supervised fine-tuning (SFT) with preference-based fine-tuning such as Direct Preference Optimization (DPO). Human-annotated and synthetically generated preference data differ substantially in per-sample cost, but prior work typically holds sample count fixed when comparing the two. In this work, we explicitly study the interplay between cost and performance by comparing the effect of post-training with human and synthetic preference data under a fixed, dollar-matched annotation budget. We train and evaluate multiple base model families across helpfulness and summarization tasks. We find that synthetic data is far more cost-efficient at low-to-moderate budgets, achieving high win rates with as little as \$200--\$400. On helpfulness, human data closes the gap only at budgets roughly two orders of magnitude larger; on summarization, it does not close the gap at any tested budget. We also find that human-annotated data varies in reliability: one widely used human preference source substantially underperforms another, and the gap persists even after standard remedies such as domain restriction and deduplication.

## How Post-Training Influences Verbalized Algorithmic Computation in Language Models

DevFeed: [How Post-Training Influences Verbalized Algorithmic Computation in Language Models](<https://devfeed.tech/articles/how-post-training-influences-verbalized-algorithmic-computation-in-language-models-128731.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13552>)

Author: Niall Dalton

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [post-training](<https://devfeed.tech/topics/post-training.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [rlvr](<https://devfeed.tech/topics/rlvr.md>), [Chain-of-thought](<https://devfeed.tech/topics/chain-of-thought.md>), [dpo](<https://devfeed.tech/topics/dpo.md>)

Tags: [addition](<https://devfeed.tech/tags/addition.md>), [chain-of-thought](<https://devfeed.tech/tags/chain-of-thought.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [rlvr](<https://devfeed.tech/tags/rlvr.md>), [sft](<https://devfeed.tech/tags/sft.md>)

### AI overview

On multi-digit addition with Olmo-3-7B-Instruct checkpoints, RLVR-produced chains of thought led to higher accuracy across receiving checkpoints than chains from SFT or DPO. Probes recovered carry information from internal activations at similar accuracy in SFT and RLVR, but manipulating that information did not reliably control outputs. After RLVR, carry-related statements received higher importance scores than matched random controls. DPO showed similar sensitivity to carry-related statements without comparable accuracy gains. The results suggest post-training can change how useful written reasoning is and how strongly models respond to its content; greater sensitivity alone did not ensure better performance.

### Source excerpt

arXiv:2611.13552v1 Announce Type: cross Abstract: Post-training improves language-model reasoning, but how it changes the use of internal information and verbalized chain of thought (CoT) remains unclear. We study successive SFT, DPO, and RLVR checkpoints of Olmo-3-7B-Instruct on multi-digit addition, using carry values as known intermediate state. Probes recover carry information from internal activations with similar accuracy in SFT and RLVR, although manipulating this information does not reliably control model outputs. In contrast, the usefulness of verbalized reasoning differs: CoTs produced by RLVR yield higher accuracy across receiving checkpoints than those produced by SFT or DPO. We also show that carry-related statements receive higher importance scores after RLVR than matched random controls. DPO shows similar sensitivity to carry-related statements but without comparable accuracy gains. These results suggest that post-training changes the usefulness of written reasoning and sensitivity to its content in meaningful ways, although greater sensitivity does not necessarily coincide with better problem performance.

## From Counterfactual Edits to Human Narratives: LLM-Driven Interpretations for Recommender System Explanations

DevFeed: [From Counterfactual Edits to Human Narratives: LLM-Driven Interpretations for Recommender System Explanations](<https://devfeed.tech/articles/from-counterfactual-edits-to-human-narratives-llm-driven-interpretations-for-recommender-system-explanations-128369.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13914>)

Author: Vojtech Cahlik, Amir Reza Mohammadi, Rodrigo Alves, Pavel Kordik, Eva Zangerle

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [human feedback](<https://devfeed.tech/topics/human-feedback.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [edits](<https://devfeed.tech/tags/edits.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [patterns](<https://devfeed.tech/tags/patterns.md>), [quality](<https://devfeed.tech/tags/quality.md>), [recommender-system](<https://devfeed.tech/tags/recommender-system.md>)

### AI overview

The study uses large language models to turn lengthy counterfactual recommendation edit lists into concise explanations of higher-level user preference patterns. The mediators are preference-optimized for correctness, informativeness, and linguistic quality, then evaluated with human-calibrated metrics based on separate human-labeled datasets. On MovieLens-1M, summaries show high overall linguistic quality across multiple counterfactual generators and language models. Strongest semantic results come from high-success generators such as ACCENT and SPINRec, and Direct Preference Optimization improves semantic metrics in almost all evaluated settings.

### Source excerpt

arXiv:2611.13914v1 Announce Type: new Abstract: Recommender systems increasingly influence what users discover and consume online, yet the decision logic behind their outputs often remains opaque. Counterfactual explanation (CFX) methods provide an effective way to show which changes to a user's interaction history would alter a recommendation. In recommendation settings, however, CFXs often require many edits, producing long, low-level lists that are hard for users to interpret. Moreover, most evaluations emphasize edit counts while largely overlooking whether the resulting CFXs are coherent and actionable for users. We argue that this gap between generating valid CFXs and communicating model behavior stems from the lack of user-centered abstraction and limited qualitative evaluation. To bridge this gap, we investigate large language models as mediators that summarize raw CFX interaction edit lists into concise natural language explanations describing higher-level preference patterns. We optimize these mediators for correctness, informativeness, and linguistic quality through preference optimization, and evaluate the resulting explanations with human-calibrated metrics derived from separate human-labeled datasets. On the MovieLens-1M dataset, across multiple counterfactual generators and language models, the resulting summaries achieve high overall linguistic quality, while the strongest semantic results are obtained with high-success CFX methods such as ACCENT and SPINRec. Direct Preference Optimization improves the semantic metrics in almost all evaluated settings.

## Proximal Preference Optimization: A Unified Framework for Stable and Reward-Free Alignment

DevFeed: [Proximal Preference Optimization: A Unified Framework for Stable and Reward-Free Alignment](<https://devfeed.tech/articles/proximal-preference-optimization-a-unified-framework-for-stable-and-reward-free-alignment-125254.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10785>)

Author: Qisai Liu, Zhanhong Jiang, Zahid Hasan, Soumik Sarkar

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [post-training](<https://devfeed.tech/topics/post-training.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [better](<https://devfeed.tech/tags/better.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [effective](<https://devfeed.tech/tags/effective.md>), [framework](<https://devfeed.tech/tags/framework.md>), [free](<https://devfeed.tech/tags/free.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [improvement](<https://devfeed.tech/tags/improvement.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [reward](<https://devfeed.tech/tags/reward.md>), [stable](<https://devfeed.tech/tags/stable.md>)

### AI overview

P²O adds a clipped importance ratio to DPO's preference margin, aiming to enable reward-model-free training with safer dataset reuse and bounded policy divergence. PKTO extends it with KTO-style utility margins, an adaptive baseline that tightens the effective trust region as the policy drifts, and asymmetric weighting for imbalanced preferences. The paper reports KL trust-region bounds, a policy-improvement lower bound, and convergence guarantees for multi-epoch updates. Across multiple LLMs and three preference benchmarks, the authors report that both methods match or exceed DPO and KTO in alignment quality and are more stable under repeated dataset reuse.

### Source excerpt

arXiv:2611.10785v1 Announce Type: new Abstract: We introduce a proximal preference optimization framework that unifies the implicit reward formulation of Direct Preference Optimization (DPO) with the trust-region stability of Proximal Policy Optimization (PPO). Our framework yields two complementary algorithms. P$^2$O (Proximal Preference Optimization) introduces a clipped importance ratio directly into the DPO preference margin, producing a reward-model-free algorithm that supports safe dataset reuse, bounded policy divergence, and monotonic preference improvement guarantees. PKTO (Proximal Kahneman-Tversky Optimization) extends P$^2$O by replacing the DPO margin with KTO's prospect-theoretic utility margins, adding a self-regulating adaptive baseline that tightens the effective trust region as the policy drifts, and an asymmetric weighting that handles imbalanced or asymmetric preference datasets. We provide a unified theoretical analysis, including Kullback Leibler (KL) trust-region bounds, a policy-improvement lower bound, and convergence guarantees under multi-epoch proximal updates. Empirically, P$^2$O and PKTO match or exceed DPO and KTO in alignment quality while maintaining better stability under repeated dataset reuse across multiple LLMs on three preference benchmarks. The code base is available here.

## Causal Signal Credit Assignment for Verilog Code Generation via Counterfactual Hardware Verification

DevFeed: [Causal Signal Credit Assignment for Verilog Code Generation via Counterfactual Hardware Verification](<https://devfeed.tech/articles/causal-signal-credit-assignment-for-verilog-code-generation-via-counterfactual-hardware-verification-126192.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11848>)

Author: Yali Du, Ming Li

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Verilog](<https://devfeed.tech/topics/verilog.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [Automated reasoning](<https://devfeed.tech/topics/automated-reasoning.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [credit-assignment](<https://devfeed.tech/tags/credit-assignment.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [signal](<https://devfeed.tech/tags/signal.md>), [verification](<https://devfeed.tech/tags/verification.md>), [verilog](<https://devfeed.tech/tags/verilog.md>)

### AI overview

Causal Signal Credit Assignment (CSCA) addresses the coarse pass/fail feedback of module-level simulation by assigning continuous credit to RTL statements. It verifies sampled candidates at signal level, identifies output-driving assignments, and applies five interventions while holding the rest of the module and testbench fixed. Changes in signal-level errors and behavioral agreement determine statement credits, which are mapped to response-token weights for strict weighted direct preference optimization. The method is designed to reinforce locally supported statements without uniformly rewarding an otherwise incorrect response. Built with Icarus Verilog and a CodeV-derived corpus, it was evaluated on code-completion and specification-to-RTL tasks in VerilogEval 2.0 and RTLLM v2.0. With Qwen2.5-Coder-7B-Instruct, CSCA led the evaluated 7B-scale models and was reported as comparable to the larger GPT-5.4-mini on these benchmarks.

### Source excerpt

arXiv:2611.11848v1 Announce Type: new Abstract: Large language models have improved Verilog generation, yet functional correctness remains difficult to optimize because simulation commonly provides a coarse module-level pass/fail outcome. Such feedback discards useful evidence from partially correct RTL modules and cannot determine whether a source statement is functionally responsible for an observed output behavior. We present a new method named Causal Signal Credit Assignment (CSCA), which is a counterfactual preference-optimization framework for Verilog generation. CSCA first performs continuous signal-level behavioral verification on sampled candidate modules. It then locates output-driving assignment spans and applies five interventions while holding the remaining module and testbench fixed. The resulting changes in signal-level errors are combined with behavioral agreement to form a continuous statement credit, which is mapped to response-token weights for strict weighted direct preference optimization. This enables CSCA to learn from locally supported statements without uniformly reinforcing an otherwise incorrect response. Built on Icarus Verilog and a CodeV-derived corpus, CSCA is evaluated on VerilogEval~2.0 and RTLLM~v2.0, including both code completion and specification-to-RTL tasks. With a Qwen2.5-Coder-7B-Instruct backbone, CSCA leads the evaluated 7B-scale models and remains comparable to the much larger GPT-5.4-mini on the reported Verilog-generation benchmarks.

## CRUSTPairs: A Cross-Language Dataset of Paired C and Rust Solutions

DevFeed: [CRUSTPairs: A Cross-Language Dataset of Paired C and Rust Solutions](<https://devfeed.tech/articles/crustpairs-a-cross-language-dataset-of-paired-c-and-rust-solutions-125437.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11711>)

Author: Vojtech Hejlek, Leonardo Jesus Almeida, Alneu de Andrade Lopes

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Rust](<https://devfeed.tech/topics/rust.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [cosine similarity search](<https://devfeed.tech/topics/cosine-similarity-search.md>)

Tags: [c](<https://devfeed.tech/tags/c.md>), [c-to-rust](<https://devfeed.tech/tags/c-to-rust.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [migration](<https://devfeed.tech/tags/migration.md>), [model](<https://devfeed.tech/tags/model.md>), [mrr](<https://devfeed.tech/tags/mrr.md>), [pairs](<https://devfeed.tech/tags/pairs.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [rust](<https://devfeed.tech/tags/rust.md>), [similarity](<https://devfeed.tech/tags/similarity.md>), [tf](<https://devfeed.tech/tags/tf.md>)

### AI overview

CRUSTPairs provides 1,857 paired C and Rust solutions to shared programming problems, drawn from competitive programming archives and normalized, deduplicated, tiered by zero-shot embedding similarity, and accompanied by provenance and licensing. In cross-language retrieval, TF-IDF over shared identifiers achieves MRR 0.21, slightly above a pretrained code model's 0.19; that model performs at 1.7 times chance on the hardest quarter. Fine-tuning reaches 0.79 overall and improves hardest-quarter performance to 10 times chance. When user identifiers are renamed, fine-tuned models retain 96-97% of their accuracy, compared with 66% for the pretrained model. On structurally divergent pairs, fine-tuned models do better without identifiers than with them.

### Source excerpt

arXiv:2611.11711v1 Announce Type: cross Abstract: C-to-Rust migration is an active agenda, but no dataset pairs the two languages at the level of whole solutions to a shared problem. We present CRUSTPairs, 1857 C/Rust implementation pairs drawn from competitive programming archives, normalised, deduplicated, and stratified into difficulty tiers by zero-shot cross-language embedding similarity, with perrecord provenance and licensing. Cross-language retrieval is hard on this data, and harder than a single baseline would suggest: TF-IDF over shared identifiers reaches MRR 0.21 and so beats a pretrained code model at 0.19, which on the hardest quarter of the data scores 1.7x chance. Finetuning reaches 0.79, and lifts that hardest quarter 10x. Renaming every user identifier settles what is being learned: the finetuned models keep 96-97% of their accuracy without them, while the pretrained model keeps 66\% - less than the lexical baseline does - and on structurally divergent pairs the finetuned models are better without identifiers than with them.

## TLRS-DPO: Token-Level Risk Scoring for Better Direct Preference Optimization

DevFeed: [TLRS-DPO: Token-Level Risk Scoring for Better Direct Preference Optimization](<https://devfeed.tech/articles/tlrs-dpo-token-level-risk-scoring-for-better-direct-preference-optimization-125344.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11232>)

Author: Suxin Ji, Hungtao Wan

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [drop-in](<https://devfeed.tech/tags/drop-in.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llms](<https://devfeed.tech/tags/llms.md>), [offline](<https://devfeed.tech/tags/offline.md>), [quality](<https://devfeed.tech/tags/quality.md>), [reference](<https://devfeed.tech/tags/reference.md>), [risk](<https://devfeed.tech/tags/risk.md>), [safety](<https://devfeed.tech/tags/safety.md>), [schedule](<https://devfeed.tech/tags/schedule.md>), [token](<https://devfeed.tech/tags/token.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

TLRS-DPO adds token-level risk control to offline, reference-based DPO, addressing its sequence-level objective's inability to localize risky patterns within otherwise acceptable responses. It derives localized risk scores from policy-reference log-ratios on rejected responses, gradually increases their compensation during training, and selectively penalizes high-risk tokens rather than entire sequences. The authors report improved alignment and safety-relevant behavior with minimal impact on helpfulness versus strong DPO baselines; the method is described as a drop-in modification to standard DPO trainers.

### Source excerpt

arXiv:2611.11232v1 Announce Type: new Abstract: Direct Preference Optimization (DPO) is a simple and widely used alternative to RLHF for aligning large language models (LLMs), but its sequence-level objective cannot precisely locate or suppress the local patterns that make an otherwise acceptable response unsafe or undesirable. We propose TLRS-DPO (Token-Level Risk Scoring for Better Direct Preference Optimization), which augments DPO with fine-grained token-level control while remaining fully offline and reference-based. TLRS-DPO introduces a token-level robust compensation term that converts the policy-reference log-ratio on rejected responses into localized risk scores, a curriculum-style schedule that gradually increases the strength of this compensation during training, and a risk-adaptive screening mechanism that selectively penalizes only high-risk tokens instead of entire sequences. This design bridges sequence-level preference fitting and token-level behavior control, and can be implemented as a drop-in modification to standard DPO trainers. Empirically, TLRS-DPO improves alignment quality and safety-relevant behavior with minimal impact on helpfulness compared to strong DPO baselines.

## Personalized Alignment of Language Models with Human Value Profiles

DevFeed: [Personalized Alignment of Language Models with Human Value Profiles](<https://devfeed.tech/articles/personalized-alignment-of-language-models-with-human-value-profiles-125840.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09919>)

Author: Rashon Poole, Farnaz Jahanbakhsh

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [personalization](<https://devfeed.tech/topics/personalization.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [human feedback](<https://devfeed.tech/topics/human-feedback.md>)

Tags: [activation-steering](<https://devfeed.tech/tags/activation-steering.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-hc](<https://devfeed.tech/tags/cs-hc.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [personalization](<https://devfeed.tech/tags/personalization.md>), [profile](<https://devfeed.tech/tags/profile.md>), [prompting](<https://devfeed.tech/tags/prompting.md>), [survey](<https://devfeed.tech/tags/survey.md>), [value](<https://devfeed.tech/tags/value.md>)

### AI overview

The paper proposes personalizing language model responses around users' stable Schwartz value profiles. For each query, it selects relevant values, then applies targeted prompting, activation steering, or LoRA-DPO. In tests using 1,501 value-tension scenarios and more than 152 value profiles sampled from a nationally representative survey, value-targeted methods outperformed no-personalization, demographic, and full-profile prompting baselines. Steering gains varied across users' positions in the value space. Combining targeted prompting with activation steering or LoRA-DPO produced the strongest personalization while maintaining logical quality within an adequate range.

### Source excerpt

arXiv:2611.09919v1 Announce Type: cross Abstract: Large language models are increasingly used for subjective tasks such as seeking advice, where there is often no single correct answer. Yet dominant alignment methods learn a single behavior policy that applies uniformly across users, limiting their ability to reflect individual value priorities. We propose personalized value alignment, a framework for adapting model responses to users' stable value profiles rather than to personas, demographics, or task-specific preferences used in prior personalization work. Our approach represents users through Schwartz value profiles and selects which of their values are relevant to each query before applying targeted prompting, activation steering, or LoRA-DPO. We create a dataset of 1,501 value-tension scenarios and evaluate over 152 diverse value profiles sampled from a nationally representative survey. Value-targeted methods outperform no-personalization, demographic, and full-profile prompting baselines, indicating that gains come from focused conditioning on query-specific value priorities rather than mere exposure. We further find that steering effects are not uniform across users: alignment gains vary by where a user's priorities fall in the value space. Combining targeted prompting with activation steering or LoRA-DPO yields the strongest personalization while keeping logical quality within adequate range.

## On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning

DevFeed: [On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning](<https://devfeed.tech/articles/on-the-reliability-of-majority-vote-pseudo-labels-in-test-time-reinforcement-learning-124738.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09872>)

Author: Haoyu Wang, Ru Wang, Yuanyuan Wang, Jiaxian Guo, Mingming Gong

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [Uncertainty quantification LLMs](<https://devfeed.tech/topics/uncertainty-quantification-llms.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Self-supervised learning](<https://devfeed.tech/topics/self-supervised-learning.md>)

Tags: [bias](<https://devfeed.tech/tags/bias.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [test](<https://devfeed.tech/tags/test.md>), [vote](<https://devfeed.tech/tags/vote.md>)

### AI overview

The paper analyzes reliability errors in majority-vote pseudo-labels used for test-time reinforcement learning with online Direct Preference Optimization (DPO), separating population-majority bias from prompt-dependent finite-rollout variation. Since population-majority bias is generally unidentifiable without external supervision, the proposed Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO) uses observed vote reliability to make updates more conservative for sampling-sensitive prompts. A cache-based mechanism is also introduced to reduce self-supervised forgetting during adaptation. Across six reasoning benchmarks and five model backbones, PC-Dr.DPO matches or outperforms the strongest TTRL baseline in 24 of 30 settings.

### Source excerpt

arXiv:2611.09872v1 Announce Type: new Abstract: Test-time reinforcement learning (TTRL) adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, whose reliability remains poorly understood. We analyze majority-vote supervision for TTRL with online Direct Preference Optimization (DPO) and decompose its error into population-majority bias and prompt-dependent finite-rollout variation. Because population-majority bias is generally unidentifiable without external supervision, we focus on mitigating the downstream effects of finite-rollout variation. We propose \emph{Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO)}, which conditions within-prompt robust reweighting on observed vote reliability to make updates from sampling-sensitive prompts more conservative. We further introduce a cache-based mechanism to reduce self-supervised forgetting during online adaptation. Across six mathematical and knowledge-intensive reasoning benchmarks and five model backbones, PC-Dr.DPO matches or outperforms the strongest TTRL baseline in 24 of 30 settings.

## SteadyThought: Mitigating LLM Under-Thinking via Thought-Level Preference Optimization

DevFeed: [SteadyThought: Mitigating LLM Under-Thinking via Thought-Level Preference Optimization](<https://devfeed.tech/articles/steadythought-mitigating-llm-under-thinking-via-thought-level-preference-optimization-124492.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.25045>)

Author: Zhenyue Peng, Lin Yang, Jie Wang, Xiaoqi Ni, Hanzhu Chen, Xize Liang, Yanjun Wu, Rujing Wang, Kaiwen Zhou, Jianye Hao

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [languages](<https://devfeed.tech/tags/languages.md>), [llm](<https://devfeed.tech/tags/llm.md>), [models](<https://devfeed.tech/tags/models.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reasoning-models](<https://devfeed.tech/tags/reasoning-models.md>), [reduce](<https://devfeed.tech/tags/reduce.md>), [switching](<https://devfeed.tech/tags/switching.md>), [thinking](<https://devfeed.tech/tags/thinking.md>), [thought](<https://devfeed.tech/tags/thought.md>), [thoughts](<https://devfeed.tech/tags/thoughts.md>), [token](<https://devfeed.tech/tags/token.md>), [token-consumption](<https://devfeed.tech/tags/token-consumption.md>)

### AI overview

Steady Thought (ST) treats under-thinking as a preference problem at reasoning-trajectory switching points. It identifies points where a single-path continuation can produce high-quality reasoning, then optimizes preferences by labeling that new continuation as preferred and the original response as dispreferred. The abstract reports token-use reductions of up to 63.8% while maintaining strong reasoning accuracy across multiple models and datasets; further analysis suggests the reduced redundant switching generalizes across domains and languages. The supplied text is an abstract, so it does not provide experimental details or establish how broadly the results apply.

### Source excerpt

arXiv:2610.25045v2 Announce Type: replace Abstract: Flexible switching between reasoning trajectories (i.e., thought switching) has significantly enhanced the reasoning capabilities of Large Reasoning Models (LRMs). However, existing models often switch excessively yet fail to sustain promising reasoning thoughts, a phenomenon termed "under-thinking". While recent efforts suppress switching to mitigate this, such over-correction may discard valuable trajectories. To address this challenge, we propose Steady Thought (ST), a novel thought-level preference optimization framework. ST formalizes under-thinking as a preference issue at switching points, forcing single-path continuations from identified points to yield high-quality reasoning. Then, ST performs thought-level preference optimization by treating the newly generated response as preferred and the original one as dis-preferred. Experiments across multiple models and datasets show that ST effectively reduces token consumption, achieving reductions of up to 63.8%, while maintaining strong reasoning accuracy. Further analysis suggests that ST helps models acquire a generalizable ability to reduce redundant switching across domains and languages. Code is available at https://github.com/MIRALab-USTC/SteadyThought.

## When Scores Conflict with Preferences: Pre-Alignment Diagnosis and Adaptive Correction of Constraint Drift in DPO

DevFeed: [When Scores Conflict with Preferences: Pre-Alignment Diagnosis and Adaptive Correction of Constraint Drift in DPO](<https://devfeed.tech/articles/when-scores-conflict-with-preferences-pre-alignment-diagnosis-and-adaptive-correction-of-constraint-drift-in-dpo-120607.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07375>)

Author: Ping Liu, Yan Yan

Published: 2026-11-06T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [constraint](<https://devfeed.tech/tags/constraint.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diagnosis](<https://devfeed.tech/tags/diagnosis.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>)

### AI overview

In standard DPO, preference pairs whose preferred response scores worse on a pointwise criterion can increase constraint cost, while matching pairs can reduce it; the net effect depends on their balance. The paper introduces a dataset-statistics diagnostic, ρ*eq, to predict whether a dataset is self-correcting or needs intervention. Its Conflict-aware DPO (CoDPO) method uses that diagnostic to set conflict-pair down-weighting and adds an adaptive margin shift to address remaining drift. In the reported experiments, the diagnostic's intervention decisions matched observed drift across all domains. Across four main-comparison domains, CoDPO reduced drift while largely preserving vanilla DPO's preference learning, with a more favorable drift-preference trade-off than hard filtering and the constrained and multi-objective baselines evaluated.

### Source excerpt

arXiv:2611.07375v1 Announce Type: new Abstract: Direct Preference Optimization (DPO) is increasingly used in settings where pairwise preferences coexist with pointwise criteria such as response length, answer correctness, or safety scores. When the two disagree, standard DPO treats pairs whose preferred output scores worse on the pointwise criterion (conflict pairs) in the same way as the remaining (matching) pairs. We characterize a systematic failure mode, constraint drift: for sufficiently small updates, conflict pairs tend to increase constraint cost while matching pairs tend to repair it, and the net change depends on their balance. This balance yields a closed-form pre-alignment diagnostic, $\rho^{*}_{\mathrm{eq}}$, computable from dataset statistics, that indicates whether a dataset is self-correcting or requires intervention. Building on this diagnostic, we propose Conflict-aware DPO (CoDPO), which down-weights conflict pairs with a strength set from $\rho^{*}_{\mathrm{eq}}$ rather than tuned, and adds an adaptive margin shift that addresses residual drift during training. Empirically, drift grows monotonically with conflict exposure on the conflict-dominated domains, and the intervention decision given by $\rho^{*}_{\mathrm{eq}}$ matches the observed drift on every domain, including one that it identifies as self-correcting. On all four domains of the main comparison, CoDPO reduces drift while largely preserving the preference learning of vanilla DPO. Compared with hard filtering and recent constrained and multi-objective baselines, CoDPO also achieves a more favorable drift-preference trade-off.

## The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention

DevFeed: [The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention](<https://devfeed.tech/articles/the-alignment-tax-is-graded-task-selective-capability-loss-under-length-normalized-dpo-with-sft-retention-120535.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.06476>)

Author: Ye Yuan

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [lora](<https://devfeed.tech/topics/lora.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [characterization](<https://devfeed.tech/tags/characterization.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [financial](<https://devfeed.tech/tags/financial.md>), [format](<https://devfeed.tech/tags/format.md>), [json](<https://devfeed.tech/tags/json.md>), [lora](<https://devfeed.tech/tags/lora.md>), [margin](<https://devfeed.tech/tags/margin.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [model](<https://devfeed.tech/tags/model.md>), [objectives](<https://devfeed.tech/tags/objectives.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [retention](<https://devfeed.tech/tags/retention.md>), [sft](<https://devfeed.tech/tags/sft.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [tax](<https://devfeed.tech/tags/tax.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

In a study of Qwen3.5-4B on three structured financial-filing tasks, increasing the implicit reward margin under length-normalized DPO with an SFT retention term produced task-selective capability loss. Across 8 seeds, extraction was preserved within a ±5-point equivalence margin (−0.55 points); JSON performance fell 3.52 points, with equivalence not established; and router performance fell 30 points, mainly in a margin 1-3 damage window that included a transient format-break spike. At least 96% of margin growth came from suppressing rejected completions while chosen reward stayed approximately flat; the study does not assign the behavioral damage to a single gradient channel. Measured reward margins differed by up to 3.4x between nf4 training and fp16 evaluation on the unquantized base. These findings are limited to one 4B model, sparse LoRA, easily separable synthetic preferences, and the tested objective; the authors leave a no-retention control and measured format-alignment metric for future work.

### Source excerpt

arXiv:2611.06476v2 Announce Type: replace Abstract: Setting. This is an empirical characterization study; we propose no new method and run no comparisons against alternative objectives. We fine-tune Qwen3.5-4B with sparse LoRA and length-normalized DPO plus an SFT retention term (the RPO objective) on structured financial-filing tasks, and track downstream capability as the implicit reward margin grows to $\sim 7$ $\beta$-scaled units (nf4 training; $\sim 3$ under fp16 measurement on the unquantized base). Question. Does margin growth degrade capability uniformly across tasks, or selectively? Findings. In a pre-registered study (8 seeds, 6 unseen at freeze) with artifact-aware scoring and TOST equivalence testing, the alignment tax is graded and task-selective: extraction is preserved ($\Delta = -0.55$ pts, equivalent at $\pm5$ pts), json loses little ($\Delta = -3.52$ pts, equivalence fails on a clustered item-level interval), and router degrades severely ($\Delta = -30$ pts). Router's loss concentrates in a margin 1-3 damage window containing a transient format-break spike. The margin is accounted for almost entirely ($\geq 96\%$) by suppression of rejected completions, with chosen reward approximately flat, consistent with the retention term anchoring it; we do not attribute the behavioral damage to any single gradient channel. Gold-logp deflation is task-selective (extraction $-0.15$, json $-1.04$, router $-1.47$ nats/token under fp16), with gold selectively protected relative to incorrect controls on extraction (8/8 seeds). QLoRA reward margins differ up to $3.4\times$ between nf4 training and unquantized fp16 measurement, with an axis-dependent gap. Scope. Length-normalized rewards with SFT retention, sparse LoRA on 8/32 layers, one 4B model, easily-separable synthetic preferences, three structured tasks. A no-retention control and a measured format-alignment metric are left to future work.

## Vision Paper: Why Distributed LLM Post-Training and Decentralized Alignment Are Now Essential

DevFeed: [Vision Paper: Why Distributed LLM Post-Training and Decentralized Alignment Are Now Essential](<https://devfeed.tech/articles/vision-paper-why-distributed-llm-post-training-and-decentralized-alignment-are-now-essential-118389.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05841>)

Author: Zhanhong Jiang, Qisai Liu, Soumik Sarkar

Published: 2026-11-05T05:00:00Z

Content type: opinion

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [post-training](<https://devfeed.tech/topics/post-training.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Federated Learning](<https://devfeed.tech/topics/federated-learning.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [data-sovereignty](<https://devfeed.tech/tags/data-sovereignty.md>), [decentralized](<https://devfeed.tech/tags/decentralized.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [equity](<https://devfeed.tech/tags/equity.md>), [federated](<https://devfeed.tech/tags/federated.md>), [healthcare](<https://devfeed.tech/tags/healthcare.md>), [llm](<https://devfeed.tech/tags/llm.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [research](<https://devfeed.tech/tags/research.md>), [robustness](<https://devfeed.tech/tags/robustness.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [scale](<https://devfeed.tech/tags/scale.md>), [unique](<https://devfeed.tech/tags/unique.md>), [utility](<https://devfeed.tech/tags/utility.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

The paper argues that LLM post-training should support federated and decentralized learning as a core approach, since preference data in fields such as healthcare, law, finance, and multilingual communities may not be suitable for centralization. It grounds the case in privacy and data sovereignty, pluralism and representational equity, and robustness and scalability. It identifies reference drift, heterogeneous preferences, and the privacy-versus-utility trade-off for contrastive losses as technical challenges, and proposes algorithmic directions and a research agenda. These details are described only at abstract level in the supplied text.

### Source excerpt

arXiv:2611.05841v1 Announce Type: new Abstract: This vision paper argues that distributed learning, including federated and decentralized, should become a first-class paradigm in LLM post-training, not merely a privacy add-on. Current methods like RLHF and DPO rest on an invisible assumption: preference data should be centralized. We contend this assumption is scientifically unjustified, practically unsustainable, and ethically problematic. As LLMs deploy into healthcare, law, finance, and multilingual communities, preference data originates from populations that cannot or will not centralize. We develop a three-pillar argument grounded in (1) privacy and data sovereignty, (2) pluralism and representational equity, and (3) robustness and scalability. We formalize key technical challenges unique to this intersection, reference drift, preference heterogeneity, and the DP-utility trade-off for contrastive losses, and propose concrete algorithmic directions, and sketch a research agenda. Our analysis shows distributed post-training is not only necessary but tractable, and ignoring it risks AI that is legally noncompliant, representationally narrow, and brittle at scale.

## Objective-Conditioned Active Preference Learning for Multi-Objective Alignment

DevFeed: [Objective-Conditioned Active Preference Learning for Multi-Objective Alignment](<https://devfeed.tech/articles/objective-conditioned-active-preference-learning-for-multi-objective-alignment-118331.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05506>)

Author: Duc Anh Vu, Thong Nguyen, Hoang Pham, Ponhvoan Srey, Luu Anh Tuan

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [annotation](<https://devfeed.tech/tags/annotation.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [can](<https://devfeed.tech/tags/can.md>), [change](<https://devfeed.tech/tags/change.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [cost](<https://devfeed.tech/tags/cost.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [effectiveness](<https://devfeed.tech/tags/effectiveness.md>), [either](<https://devfeed.tech/tags/either.md>), [existing](<https://devfeed.tech/tags/existing.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [external](<https://devfeed.tech/tags/external.md>), [framework](<https://devfeed.tech/tags/framework.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [learning](<https://devfeed.tech/tags/learning.md>)

### AI overview

OGC-DPO reduces preference-labeling costs for multi-objective LLM alignment by treating each response-pair/objective combination as a query and selecting queries that expand gradient coverage for each objective. It estimates informativeness from the policy's implicit gradient, without an external reward model. In experiments across varying numbers of objectives, it outperformed existing active preference-learning methods at the same annotation cost. The authors also report that informative conflicts between preferences can provide a useful training signal, rather than being uniformly harmful.

### Source excerpt

arXiv:2611.05506v1 Announce Type: cross Abstract: Aligning large language models (LLMs) with several objectives, such as helpfulness, harmlessness, and honesty, typically relies on preference judgment for each objective, growing the annotation cost linearly with the number of objectives. Active preference learning can reduce this cost by selecting the comparisons whose labels are expected to change the policy most. Existing remedies either lack an objective-aware query selection mechanism for the multi-objective setting or regard conflicting preferences as harmful, therefore excluding them from the training data. We introduce OGC-DPO (Objective-conditioned Gradient Coverage DPO), an active learning framework that treats each (response pair, objective) combination as a separate query and measures its informativeness through the policy's own implicit gradient, without relying on an external reward model. OGC-DPO then selects the queries that most expand each objective's gradient coverage. Empirical experiments across varying numbers of objectives demonstrate the effectiveness of OGC-DPO over existing active preference learning methods under the same annotation cost. Further analysis demonstrates that informative conflicts, rather than being uniformly harmful, provide a useful policy training signal.

## Teaching a 4B Model to Think Like a Policy Expert

DevFeed: [Teaching a 4B Model to Think Like a Policy Expert](<https://devfeed.tech/articles/teaching-a-4b-model-to-think-like-a-policy-expert-117705.md>)

Original publisher: [Read original article](<https://www.atlassian.com/blog/how-we-build/teaching-a-4b-model-to-think-like-a-policy-expert>)

Author: dkoudsi@atlassian.com

Published: 2026-11-04T16:55:00Z

Content type: article

Language: en

Sources: [How We Build Archives - Inside Atlassian](<https://devfeed.tech/sources/how-we-build-archives-inside-atlassian.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [Inference](<https://devfeed.tech/topics/inference.md>)

Tags: [acceptable-use](<https://devfeed.tech/tags/acceptable-use.md>), [adversarial](<https://devfeed.tech/tags/adversarial.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [atlassian](<https://devfeed.tech/tags/atlassian.md>), [bridge](<https://devfeed.tech/tags/bridge.md>), [content-moderation](<https://devfeed.tech/tags/content-moderation.md>), [data](<https://devfeed.tech/tags/data.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [how-we-build](<https://devfeed.tech/tags/how-we-build.md>), [latency](<https://devfeed.tech/tags/latency.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [rovo](<https://devfeed.tech/tags/rovo.md>), [search](<https://devfeed.tech/tags/search.md>), [user](<https://devfeed.tech/tags/user.md>)

### AI overview

Atlassian fine-tuned NVIDIA Nemotron 3 Nano 4B as a fast first-pass filter for Acceptable Use Policy violations, routing suspicious cases to a larger 20B model. Prompt engineering on the 20B model had plateaued at 50.9% recall, and its inference-time reasoning added latency. The team trained the 4B model with supervised examples containing labels and reasoning traces, then used Direct Preference Optimization to refine difficult cases. Synthetic training data included benign, violating, and adversarial examples; evaluation used balanced and adversarial sets. The article reports that the fine-tuned model reached 87.0% precision and 75.5% recall, while the overall moderation system became 53% faster and supported 42% more throughput. AI agents helped run over 300 experiments. The reported results are Atlassian's internal evaluation and production claims.

### Source excerpt

Content moderation at Atlassian's scale means making millions of real-time decisions every message, prompt, and query that flows through Rovo Chat, Rovo Search, and more has to be checked before a response is served. That puts moderation squarely on the critical path of every user request. Latency isn't a nice-to-have. It's a product requirement. For [...] The post Teaching a 4B Model to Think Like a Policy Expert appeared first on Inside Atlassian.

## Think Less, Locally: Local Preference Learning for Reasoning Compression

DevFeed: [Think Less, Locally: Local Preference Learning for Reasoning Compression](<https://devfeed.tech/articles/think-less-locally-local-preference-learning-for-reasoning-compression-117532.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.04814>)

Author: Muyao Yuan, Liyan Xu, Weizhan Zhang, Muyan Jiao, Jiangnan Li, Jinchao Zhang, Jie Zhou

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [post-training](<https://devfeed.tech/topics/post-training.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [data](<https://devfeed.tech/tags/data.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [inference-costs](<https://devfeed.tech/tags/inference-costs.md>), [learning](<https://devfeed.tech/tags/learning.md>), [local](<https://devfeed.tech/tags/local.md>), [mathematics](<https://devfeed.tech/tags/mathematics.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

LoCo targets the decision to keep reasoning or stop, rather than shortening reasoning traces indiscriminately. It finds candidate overthinking states in a model's own traces, freezes each prefix, generates shorter verified continuations from that state, and uses state-conditioned preference pairs with DPO. Training data comes entirely from self-generated mathematical traces. Across five math and three code benchmarks, the authors report completion lengths reduced by 41.7%-45.4% on math and 23.4%-36.3% on code, with accuracy maintained or improved; LoCo achieved the best AUC on all code benchmarks. The reported code results indicate transfer from math-only training to code generation. The supplied text is an abstract and does not provide further experimental details.

### Source excerpt

arXiv:2611.04814v1 Announce Type: new Abstract: Large reasoning models achieve strong performance on challenging tasks, but their long chain-of-thought often incurs substantial inference costs by continuing to deliberate after sufficient information has been obtained. Existing reasoning compression methods primarily operate on reasoning trajectories, paying less attention to the local decision of whether further reasoning is necessary given the current state. We propose LoCo, a local preference learning method for reasoning compression. LoCo detects candidate overthinking states from the model's own traces, freezes the corresponding reasoning prefixes, regenerates shorter verified continuations from the same states, and trains on the resulting state-conditioned preference pairs with DPO. This formulation teaches the model when to continue reasoning and when to conclude, rather than simply encouraging shorter trajectories overall. We construct training data entirely from self-generated mathematical traces and evaluate LoCo on five mathematical and three code generation benchmarks. LoCo reduces completion length by 41.7\%--45.4\% on mathematics and 23.4\%--36.3\% on code, while maintaining or improving accuracy and achieving the best AUC across all code benchmarks. LoCo consistently improves the accuracy-efficiency trade-off on mathematics and transfers effectively to code generation despite math-only training, demonstrating the cross-domain generality of local preference learning for reasoning compression. The code and model will be publicly released at https://github.com/MuyaoYuan/LoCo.

[Next page](<https://devfeed.tech/topics/dpo.md?cursor=WyIyMDI2LTExLTA0VDA1OjAwOjAwKzAwOjAwIiwgImQwNGVjNzY2LThlYjMtNDAxOS1iNmY3LWNkZThhYWJlMDVkOSJd>)