# dpo

Published articles for dpo.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Swiss-Knife: A Framework for Reconfigurable Externalised Multi-Objective Alignment at Decode Time

DevFeed: [Swiss-Knife: A Framework for Reconfigurable Externalised Multi-Objective Alignment at Decode Time](<https://devfeed.tech/articles/swiss-knife-a-framework-for-reconfigurable-externalised-multi-objective-alignment-at-decode-time-143311.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.22226>)

Author: Agnibh Karmakar, Mayur Parvatikar, Shreyash Dhoot, Amit Dhanda, Aman Chadha, Kapil Wanaskar, Vinija Jain, Amitava Das

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [peft](<https://devfeed.tech/topics/peft.md>), [lora](<https://devfeed.tech/topics/lora.md>), [post-training](<https://devfeed.tech/topics/post-training.md>)

Tags: [adversarial](<https://devfeed.tech/tags/adversarial.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [arm](<https://devfeed.tech/tags/arm.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [batch](<https://devfeed.tech/tags/batch.md>), [best](<https://devfeed.tech/tags/best.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [contamination](<https://devfeed.tech/tags/contamination.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decode](<https://devfeed.tech/tags/decode.md>), [design](<https://devfeed.tech/tags/design.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [external](<https://devfeed.tech/tags/external.md>), [f](<https://devfeed.tech/tags/f.md>), [family](<https://devfeed.tech/tags/family.md>), [form](<https://devfeed.tech/tags/form.md>), [framework](<https://devfeed.tech/tags/framework.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

Swiss-Knife treats decode-time alignment as a reconfigurable runtime specification combining hot-swappable scoring models, batch normalization, pairwise aggregation and a selection rule. Its representation theorem describes admissible aggregators as a two-parameter family that includes probit, logistic and pointwise argmax rules. The authors report that pairwise aggregation is stable under adversarial reward contamination, whereas argmax is not; Candidate-Batch Normalization (CBN) makes objective weights invariant to reward-model rescaling. In their DPO-LoRA and uncertainty-aware tournament instantiation, the system achieved harmonic F1 0.797 versus 0.750 for the strongest of six methods, with the lowest refusal rate and highest helpfulness reported. Objective reconfiguration took 0.05 ms without gradient computation. Removing CBN reduced F1 by 0.217; the authors attribute the collapse to a low-variance blade retaining only 9% of its nominal influence. These are claims from the supplied abstract excerpt.

### Source excerpt

arXiv:2609.22226v2 Announce Type: replace-cross Abstract: Decode-time alignment methods steer a frozen language model by scoring candidate continuations with an external reward and selecting the maximiser. We argue that this shared design is a single degenerate point in a much larger space. We introduce Swiss-Knife, a framework for externalised multi-objective alignment in which the alignment specification is a first-class runtime object: hot-swappable scoring blades, a batch normaliser, a pairwise aggregation operator, and a selection rule. Six axioms characterise the admissible aggregation operators, and we prove a representation theorem: every operator satisfying them has the form $R_i = \sum_{j \neq i} g((\mu_i - \mu_j)/s(\sigma_i,\sigma_j))$, a two-parameter family containing probit and logistic comparison rules and pointwise argmax as named coordinates. Within it, pairwise aggregation is Lipschitz-stable under adversarial reward contamination while argmax is not, and Candidate-Batch Normalization (CBN) makes the weight simplex invariant to the rescalings under which reward models are only ever identified. Our reference instantiation pairs DPO-LoRA blades with an uncertainty-aware pairwise tournament. Sweeping the helpfulness/honesty/harmlessness simplex, it attains the best balanced frontier of six decode-time methods (harmonic $F_1$ 0.797 vs. 0.750 for the strongest baseline, $p < 10^{-14}$) with the lowest refusal rate and highest helpfulness of any arm, and reconfigures its objectives in 0.05 ms with no gradient computation. Ablating CBN costs 0.217 $F_1$, and the metadata confirms the predicted mechanism: the lowest-variance blade retains 9% of its nominal 33% influence, and the collapse follows the predicted ordering.

## Margin Dynamics for Large Language Model Alignment

DevFeed: [Margin Dynamics for Large Language Model Alignment](<https://devfeed.tech/articles/margin-dynamics-for-large-language-model-alignment-139904.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21766>)

Author: Xingzi Xu, Mehmet Saygin Seyfioglu, Karim Bouyarmane

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [design](<https://devfeed.tech/tags/design.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [gradient-flow](<https://devfeed.tech/tags/gradient-flow.md>), [ipo](<https://devfeed.tech/tags/ipo.md>), [label-smoothing](<https://devfeed.tech/tags/label-smoothing.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [llm](<https://devfeed.tech/tags/llm.md>), [margin](<https://devfeed.tech/tags/margin.md>), [model](<https://devfeed.tech/tags/model.md>), [model-alignment](<https://devfeed.tech/tags/model-alignment.md>), [models](<https://devfeed.tech/tags/models.md>), [noise](<https://devfeed.tech/tags/noise.md>), [ode](<https://devfeed.tech/tags/ode.md>), [prediction](<https://devfeed.tech/tags/prediction.md>)

### AI overview

The paper models pairwise alignment methods as flows over preferred and dispreferred completion probabilities, reducing each pair to a margin equation with a closed-form solution. It reports that DPO margins grow logarithmically without bound under gradient flow, while IPO margins approach 1/(2τ); label noise changes DPO from unbounded to bounded growth but affects IPO quantitatively. The authors also find that asymmetric weighting can concentrate gradients and affect trainability. For SimPO, any positive label smoothing ε yields a finite margin attractor, whereas standard SimPO (ε=0) has none. Tests across five model families and four benchmarks support the first predictions; in three 8B-9B models, TPO-S with ε=0.05 was competitive with strong baselines and had the highest mean IFEval score among compared methods on each. The authors describe it as a usable operating point, not a new state-of-the-art loss.

### Source excerpt

arXiv:2611.21766v1 Announce Type: cross Abstract: Practitioners choose among LLM alignment methods largely by trial and error. We provide a theoretical foundation: each pairwise method defines a vector field on the probability simplex that governs how the model redistributes mass between preferred and dispreferred completions. Projecting onto a single preferred/dispreferred pair reduces this to a one-dimensional margin ODE with a closed-form solution. These solutions reveal that DPO margins grow without bound (logarithmically as $O(\log t)$ in the gradient-flow limit) while IPO margins converge to a finite $1/(2\tau)$; that label noise induces a qualitative phase transition in DPO (unbounded $\to$ bounded) but affects IPO only quantitatively; and that gradient concentration under asymmetric weighting governs trainability at scale. We confirm all three predictions across five model families spanning four benchmarks. The closed-form solutions also enable principled method design: we prove that adding any label smoothing $\varepsilon > 0$ to SimPO creates a finite margin attractor at $\Delta^* = \beta^{-1}\log((1-\varepsilon)/\varepsilon)$, while $\varepsilon = 0$ (standard SimPO) has no attractor. We test this across three models (8B-9B) and confirm the prediction: the theory-derived configuration (TPO-S, $\varepsilon = 0.05$) is competitive with the strongest baselines, attaining the highest mean IFEval score among the compared methods on all three, validating that the predicted finite attractor is a usable operating point rather than proposing a new state-of-the-art loss.

## Arabic Safety Alignment as Selective Refusal: An Empirical Study of SFT, DPO, and Guard Calibration

DevFeed: [Arabic Safety Alignment as Selective Refusal: An Empirical Study of SFT, DPO, and Guard Calibration](<https://devfeed.tech/articles/arabic-safety-alignment-as-selective-refusal-an-empirical-study-of-sft-dpo-and-guard-calibration-139811.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.29378>)

Author: Mohamad Zbib, Mohamad Bazzi, Ammar Mohanna

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [dpo](<https://devfeed.tech/topics/dpo.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [arabic](<https://devfeed.tech/tags/arabic.md>), [computer-science](<https://devfeed.tech/tags/computer-science.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [guard](<https://devfeed.tech/tags/guard.md>), [guards](<https://devfeed.tech/tags/guards.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [safety](<https://devfeed.tech/tags/safety.md>), [sft](<https://devfeed.tech/tags/sft.md>), [study](<https://devfeed.tech/tags/study.md>)

### AI overview

The study finds that Arabic safety alignment requires choosing a model-specific balance between refusing harmful prompts (H) and avoiding refusals of benign prompts (B). Across five models and 130 runs on AraSafe, refusal-only SFT tended toward blanket refusal, while selected mixed-SFT settings achieved H of 90-93% with B of 14-23%; four settings cleared the H>=90% target in all three runs, and Fanar did so in two. DPO and inference guards shifted B and H differently across models, so they were not consistent upgrades. On Arabizi, selected SFT raised H for all five models, but none reached 90%, indicating partial transfer from Modern Standard Arabic. In a 300-response audit, Qwen3Guard and Aya Expanse 32B scored 88.7% and 91.0% accuracy, with no conclusive paired difference. The abstract recommends retaining interventions only when they meet a defined deployment target.

### Source excerpt

arXiv:2608.29378v2 Announce Type: replace-cross Abstract: Arabic large language models must refuse harmful prompts without over-refusing benign or sensitive prompts, yet a single refusal rate hides this trade-off. We evaluate it using benign refusal B and harmful-prompt refusal H, where H measures refusal rather than harmful compliance. Across five Arabic-capable models and 130 runs on the full human-written AraSafe set, refusal-only supervised fine-tuning (SFT) collapses toward blanket refusal, whereas selected mixed-SFT configurations reach H = 90% to 93% at B = 14% to 23%; four selected configurations exceed the H = 90% target in all three runs, while Fanar does so in two of three. Direct Preference Optimization (DPO) and inference guards change B and H differently across models rather than acting as uniform upgrades. In a blinded 300-response audit, annotator binary-refusal agreement is 89.0% (kappa = 0.78); Qwen3Guard and Aya Expanse 32B reach 88.7% and 91.0% accuracy, respectively, with no conclusive paired difference. Selected SFT raises H on Arabizi for all five models, but none reaches 90%, showing only partial transfer from Modern Standard Arabic. Overall, the results support model-specific operating-point selection: set a deployment target and retain only interventions that improve it.

## GEM-MM: Reducing VLM Hallucination with Entropy-Guided Preference Alignment

DevFeed: [GEM-MM: Reducing VLM Hallucination with Entropy-Guided Preference Alignment](<https://devfeed.tech/articles/gem-mm-reducing-vlm-hallucination-with-entropy-guided-preference-alignment-137524.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20016>)

Author: Yuanshan Chua, Xuejiao Zhao

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [post-training](<https://devfeed.tech/topics/post-training.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [chain-of-thought](<https://devfeed.tech/tags/chain-of-thought.md>), [chair](<https://devfeed.tech/tags/chair.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [policy](<https://devfeed.tech/tags/policy.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [rlhf](<https://devfeed.tech/tags/rlhf.md>), [vlm](<https://devfeed.tech/tags/vlm.md>)

### AI overview

With a 3,000-prompt preference budget, GEM-MM reduces reported vision-language model hallucination rates while outperforming the compared alignment baselines on several preference metrics. The method samples on-policy candidates, emphasizes high-entropy reasoning forks where visual claims are made, penalizes uncertain final answers, and updates the model with group-normalized policy gradients. In experiments using Qwen3-VL-4B-Instruct, AMBER hallucination falls to 20.6 from 23.4-24.0 and CHAIR to 4.6 from 5.0-5.1 at matched caption coverage; the entropy-depth preference score reaches 61.7%, versus 57.3-58.4% for baselines. The abstract also reports top rankings across three HallusionBench granularities and a 62.4% near-chosen rate at a conservative operating point. It notes a collapse mode when exceeding the fair preference budget.

### Source excerpt

arXiv:2611.20016v1 Announce Type: new Abstract: Vision-language models (VLMs) are routinely aligned with offline pairwise RLHF objectives, but multimodal preference data is scarce and treats every response token as equally informative. Under a small preference budget, pair construction becomes nearly irrelevant: standard DPO-style, image-conditional, and self-improvement objectives all perform within 1.1 points on MM-RLHF metrics and 0.5 points on hallucination rates, matching the SFT baseline. We call this entropy-blind preference optimization: the learning signal ignores policy uncertainty, which in chain-of-thought VLMs concentrates on specific fork tokens where visual claims are made. We introduce GEM-MM, extending entropy-guided preference modeling to VLMs. It samples on-policy candidates, rewards high-entropy reasoning forks, penalizes uncertain final answers, and updates the model via group-normalized policy gradients. On a 3000-prompt MM-RLHF budget with Qwen3-VL-4B-Instruct, GEM-MM achieves the best entropy-depth preference score (61.7% vs. 57.3-58.4% baselines) and implicit-reward accuracy (41.4%). Crucially, it reduces AMBER hallucination from 23.4-24.0 to 20.6 and CHAIR from 5.0-5.1 to 4.6 at matched caption coverage. It also ranks best on all three HallusionBench granularities, with a conservative operating point raising the near-chosen rate to 62.4%. An off-the-shelf multimodal reward model ranks GEM-MM first, and two such models independently confirm our main ablations (p < 0.001). We report confidence intervals, paired significance tests, metric-reward functional overlaps, and document a collapse mode when exceeding the fair budget. Code and evaluation harness: https://github.com/SNOWTEAM2023/GEM-MM

## SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topology

DevFeed: [SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topology](<https://devfeed.tech/articles/sarl-label-free-reinforcement-learning-by-rewarding-reasoning-topology-137244.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2603.27977>)

Author: Yifan Wang, Bolian Li, David Cho, Ruqi Zhang, Fanping Sui, Ananth Grama

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [grpo](<https://devfeed.tech/topics/grpo.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [framework](<https://devfeed.tech/tags/framework.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [kl-divergence](<https://devfeed.tech/tags/kl-divergence.md>), [maps](<https://devfeed.tech/tags/maps.md>), [math](<https://devfeed.tech/tags/math.md>), [performance](<https://devfeed.tech/tags/performance.md>), [ppo](<https://devfeed.tech/tags/ppo.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [rlvr](<https://devfeed.tech/tags/rlvr.md>), [structure](<https://devfeed.tech/tags/structure.md>), [supervision](<https://devfeed.tech/tags/supervision.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [teaching](<https://devfeed.tech/tags/teaching.md>), [thinking](<https://devfeed.tech/tags/thinking.md>), [topology](<https://devfeed.tech/tags/topology.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

SARL trains reasoning models without answer labels by building a map of intermediate thinking steps for each response and rewarding the map's topology, aiming for locally coherent and globally efficient reasoning. The abstract reports gains over prior label-free methods on four verifiable math benchmarks (+9.1% with PPO and +11.6% with GRPO on average) and on five WildBench task categories (+34.6% with PPO and +30.4% with GRPO on average). It also reports lower KL divergence and higher policy entropy, interpreted as more stable and exploratory training. These are abstract-level reported results; no implementation details or evaluation caveats are provided here.

### Source excerpt

arXiv:2603.27977v3 Announce Type: replace Abstract: Reinforcement learning is critical to improving large reasoning models, but its success relies heavily on verifiable rewards (RLVR), making it hard to use in open-ended domains where correctness is ambiguous and cannot be verified. Moreover, reasoning trajectories remain largely unconstrained, and optimizing solely toward the final answer can favor early exploitation over generalization. In this work, we ask whether general reasoning ability can be improved by teaching models how to think (the structure of reasoning) rather than what to produce (the outcome of reasoning), and we extend traditional RLVR to open-ended settings. We introduce Structure-Aware Reinforcement Learning (SARL), a label-free framework that constructs per-response reasoning maps from intermediate thinking steps and rewards their reasoning topology. SARL shifts supervision from destination to path, encouraging reasoning trajectories that are both locally coherent and globally efficient. On verifiable math tasks, SARL outperforms prior label-free RL baselines and even exceeds RL methods with ground truth supervision, with average gains of +9.1% under PPO and +11.6% under GRPO across four math benchmarks, with particularly large improvements on AIME25 (+35.5% with PPO and +44.7% with GRPO). On non-verifiable open-ended tasks, SARL achieves average gains of +34.6% under PPO and +30.4% under GRPO on WildBench across five task categories, outperforming prior label-free RL methods and DPO, which relies on additional preference labels. Beyond strong performance, SARL exhibits substantially lower KL divergence and higher policy entropy, indicating more stable and exploratory training dynamics. Code and data are available at \href{https://github.com/cacayaya/SARL}{Code Link}.

## Contrastive Activation-Guided Zeroth-Order Preference Optimisation for Memory-Efficient LLM Alignment

DevFeed: [Contrastive Activation-Guided Zeroth-Order Preference Optimisation for Memory-Efficient LLM Alignment](<https://devfeed.tech/articles/contrastive-activation-guided-zeroth-order-preference-optimisation-for-memory-efficient-llm-alignment-139077.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20635>)

Author: Weiyun Huang, Alessio Galatolo

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [backpropagation](<https://devfeed.tech/tags/backpropagation.md>), [behaviour](<https://devfeed.tech/tags/behaviour.md>), [ca](<https://devfeed.tech/tags/ca.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-memory](<https://devfeed.tech/tags/gpu-memory.md>), [hh](<https://devfeed.tech/tags/hh.md>), [human-feedback](<https://devfeed.tech/tags/human-feedback.md>), [learning](<https://devfeed.tech/tags/learning.md>), [llm](<https://devfeed.tech/tags/llm.md>), [memory](<https://devfeed.tech/tags/memory.md>), [momentum](<https://devfeed.tech/tags/momentum.md>), [optimisation](<https://devfeed.tech/tags/optimisation.md>), [order](<https://devfeed.tech/tags/order.md>)

### AI overview

CA-ZODPO reduces activation-memory demands for preference alignment by restricting zeroth-order perturbations to a low-rank subspace built from activation differences between chosen and rejected responses, and keeping momentum in those low-rank coordinates. On Qwen2.5-1.5B with Anthropic HH-RLHF, it outperforms common zeroth-order baselines and is competitive with first-order DPO in pairwise preference evaluation. The abstract reports about 59% lower peak GPU memory at sequence length 2048, while retaining zeroth-order training's sequence-length scaling behavior.

### Source excerpt

arXiv:2611.20635v1 Announce Type: new Abstract: Direct Preference Optimisation (DPO) simplifies reinforcement learning from human feedback, but still relies on traditional backpropagation and its associated activation-memory cost. Zeroth-order (ZO) optimisation avoids backpropagation, yet isotropic perturbations can produce noisy updates for preference objectives defined by differences between chosen and rejected responses. We introduce Contrastive Activation-guided Zeroth-Order Direct Preference Optimisation (CA-ZODPO), which constructs a low-rank search subspace from the activation discrepancy between the chosen and rejected responses in each preference pair. Perturbations are restricted to this subspace, while momentum is maintained in low-rank coordinates to preserve the memory advantages of ZO training. On Qwen2.5-1.5B with Anthropic HH-RLHF, CA-ZODPO outperforms common ZO baselines and remains competitive with first-order DPO in pairwise preference evaluation. It also reduces peak GPU memory by about 59\% at a sequence length of 2048 while retaining the sequence-length scaling behaviour of zeroth-order training.

## Human or Synthetic? What a Fixed Budget Buys for Post-Training

DevFeed: [Human or Synthetic? What a Fixed Budget Buys for Post-Training](<https://devfeed.tech/articles/human-or-synthetic-what-a-fixed-budget-buys-for-post-training-133156.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16895>)

Author: Shourik Banerjee, Ethan Mendes, Mohit Raghavendra, Wei Xu, Alan Ritter

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [post-training](<https://devfeed.tech/topics/post-training.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [human feedback](<https://devfeed.tech/topics/human-feedback.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [annotation](<https://devfeed.tech/tags/annotation.md>), [budget](<https://devfeed.tech/tags/budget.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [deduplication](<https://devfeed.tech/tags/deduplication.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [human-data](<https://devfeed.tech/tags/human-data.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [performance](<https://devfeed.tech/tags/performance.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [sft](<https://devfeed.tech/tags/sft.md>), [supervised-fine-tuning](<https://devfeed.tech/tags/supervised-fine-tuning.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [synthetic-data](<https://devfeed.tech/tags/synthetic-data.md>)

### AI overview

Under a dollar-matched annotation budget, synthetic preference data was more cost-efficient than human-annotated data at low to moderate spending in the tested post-training experiments. It achieved high win rates with $200-$400; on helpfulness, human data closed the gap only at budgets roughly two orders of magnitude larger, while on summarization it did not close the gap at any tested budget. Results also varied between human preference sources: one widely used source underperformed another, and the difference persisted after domain restriction and deduplication. The abstract reports evaluations across multiple base model families and tasks but does not provide further methodological detail.

### Source excerpt

arXiv:2611.16895v1 Announce Type: cross Abstract: Post-training of large language models typically combines supervised fine-tuning (SFT) with preference-based fine-tuning such as Direct Preference Optimization (DPO). Human-annotated and synthetically generated preference data differ substantially in per-sample cost, but prior work typically holds sample count fixed when comparing the two. In this work, we explicitly study the interplay between cost and performance by comparing the effect of post-training with human and synthetic preference data under a fixed, dollar-matched annotation budget. We train and evaluate multiple base model families across helpfulness and summarization tasks. We find that synthetic data is far more cost-efficient at low-to-moderate budgets, achieving high win rates with as little as \$200--\$400. On helpfulness, human data closes the gap only at budgets roughly two orders of magnitude larger; on summarization, it does not close the gap at any tested budget. We also find that human-annotated data varies in reliability: one widely used human preference source substantially underperforms another, and the gap persists even after standard remedies such as domain restriction and deduplication.

## A Who's Who and a What's What of GDPR

DevFeed: [A Who's Who and a What's What of GDPR](<https://devfeed.tech/articles/a-who-s-who-and-a-what-s-what-of-gdpr-131061.md>)

Original publisher: [Read original article](<https://www.syncfusion.com/blogs/post/gdpr-roles-and-responsibilities>)

Author: Sindhusree Prasanna

Published: 2026-11-12T05:00:00Z

Content type: article

Language: en

Sources: [Syncfusion](<https://devfeed.tech/sources/syncfusion-blogs.md>)

Topics: [online privacy](<https://devfeed.tech/topics/online-privacy.md>), [sensitive data](<https://devfeed.tech/topics/sensitive-data.md>)

Tags: [compliance](<https://devfeed.tech/tags/compliance.md>), [compliance-data-privacy-data-protection-gdpr-security-syncfusion](<https://devfeed.tech/tags/compliance-data-privacy-data-protection-gdpr-security-syncfusion.md>), [controllers](<https://devfeed.tech/tags/controllers.md>), [data-privacy](<https://devfeed.tech/tags/data-privacy.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [data-protection](<https://devfeed.tech/tags/data-protection.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [gdpr](<https://devfeed.tech/tags/gdpr.md>), [gdpr-compliance-basics](<https://devfeed.tech/tags/gdpr-compliance-basics.md>), [gdpr-roles](<https://devfeed.tech/tags/gdpr-roles.md>), [gdpr-roles-explained](<https://devfeed.tech/tags/gdpr-roles-explained.md>), [personal-data](<https://devfeed.tech/tags/personal-data.md>), [personal-data-under-gdpr](<https://devfeed.tech/tags/personal-data-under-gdpr.md>), [processors](<https://devfeed.tech/tags/processors.md>), [security](<https://devfeed.tech/tags/security.md>), [syncfusion](<https://devfeed.tech/tags/syncfusion.md>)

### AI overview

The text explains GDPR's key concepts and assigns distinct responsibilities to the people and organizations involved in personal data processing. Personal data includes information that can identify someone directly or indirectly, such as an IP address or cookie ID; properly anonymized data is outside GDPR's scope, while listed special category data receives additional protection. Controllers decide why and how data is processed, and processors handle it on a controller's instructions under a legal agreement. Data subjects are the individuals the information concerns. Some controllers must appoint a DPO; others may choose one. Organizations without an EU/EEA office that process data from people there are described as needing a written-authorized representative based in a relevant EU/EEA state. The text distinguishes this representative from a DPO and says their contact details should be provided to affected individuals.

### Source excerpt

Learn the fundamentals of GDPR, including data controllers, processors, DPOs, personal data, anonymized data, and compliance responsibilities.

## How Post-Training Influences Verbalized Algorithmic Computation in Language Models

DevFeed: [How Post-Training Influences Verbalized Algorithmic Computation in Language Models](<https://devfeed.tech/articles/how-post-training-influences-verbalized-algorithmic-computation-in-language-models-128731.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13552>)

Author: Niall Dalton

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [post-training](<https://devfeed.tech/topics/post-training.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [rlvr](<https://devfeed.tech/topics/rlvr.md>), [Chain-of-thought](<https://devfeed.tech/topics/chain-of-thought.md>), [dpo](<https://devfeed.tech/topics/dpo.md>)

Tags: [addition](<https://devfeed.tech/tags/addition.md>), [chain-of-thought](<https://devfeed.tech/tags/chain-of-thought.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [rlvr](<https://devfeed.tech/tags/rlvr.md>), [sft](<https://devfeed.tech/tags/sft.md>)

### AI overview

On multi-digit addition with Olmo-3-7B-Instruct checkpoints, RLVR-produced chains of thought led to higher accuracy across receiving checkpoints than chains from SFT or DPO. Probes recovered carry information from internal activations at similar accuracy in SFT and RLVR, but manipulating that information did not reliably control outputs. After RLVR, carry-related statements received higher importance scores than matched random controls. DPO showed similar sensitivity to carry-related statements without comparable accuracy gains. The results suggest post-training can change how useful written reasoning is and how strongly models respond to its content; greater sensitivity alone did not ensure better performance.

### Source excerpt

arXiv:2611.13552v1 Announce Type: cross Abstract: Post-training improves language-model reasoning, but how it changes the use of internal information and verbalized chain of thought (CoT) remains unclear. We study successive SFT, DPO, and RLVR checkpoints of Olmo-3-7B-Instruct on multi-digit addition, using carry values as known intermediate state. Probes recover carry information from internal activations with similar accuracy in SFT and RLVR, although manipulating this information does not reliably control model outputs. In contrast, the usefulness of verbalized reasoning differs: CoTs produced by RLVR yield higher accuracy across receiving checkpoints than those produced by SFT or DPO. We also show that carry-related statements receive higher importance scores after RLVR than matched random controls. DPO shows similar sensitivity to carry-related statements but without comparable accuracy gains. These results suggest that post-training changes the usefulness of written reasoning and sensitivity to its content in meaningful ways, although greater sensitivity does not necessarily coincide with better problem performance.

## Proximal Preference Optimization: A Unified Framework for Stable and Reward-Free Alignment

DevFeed: [Proximal Preference Optimization: A Unified Framework for Stable and Reward-Free Alignment](<https://devfeed.tech/articles/proximal-preference-optimization-a-unified-framework-for-stable-and-reward-free-alignment-125254.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10785>)

Author: Qisai Liu, Zhanhong Jiang, Zahid Hasan, Soumik Sarkar

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [post-training](<https://devfeed.tech/topics/post-training.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [better](<https://devfeed.tech/tags/better.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [effective](<https://devfeed.tech/tags/effective.md>), [framework](<https://devfeed.tech/tags/framework.md>), [free](<https://devfeed.tech/tags/free.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [improvement](<https://devfeed.tech/tags/improvement.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [reward](<https://devfeed.tech/tags/reward.md>), [stable](<https://devfeed.tech/tags/stable.md>)

### AI overview

P²O adds a clipped importance ratio to DPO's preference margin, aiming to enable reward-model-free training with safer dataset reuse and bounded policy divergence. PKTO extends it with KTO-style utility margins, an adaptive baseline that tightens the effective trust region as the policy drifts, and asymmetric weighting for imbalanced preferences. The paper reports KL trust-region bounds, a policy-improvement lower bound, and convergence guarantees for multi-epoch updates. Across multiple LLMs and three preference benchmarks, the authors report that both methods match or exceed DPO and KTO in alignment quality and are more stable under repeated dataset reuse.

### Source excerpt

arXiv:2611.10785v1 Announce Type: new Abstract: We introduce a proximal preference optimization framework that unifies the implicit reward formulation of Direct Preference Optimization (DPO) with the trust-region stability of Proximal Policy Optimization (PPO). Our framework yields two complementary algorithms. P$^2$O (Proximal Preference Optimization) introduces a clipped importance ratio directly into the DPO preference margin, producing a reward-model-free algorithm that supports safe dataset reuse, bounded policy divergence, and monotonic preference improvement guarantees. PKTO (Proximal Kahneman-Tversky Optimization) extends P$^2$O by replacing the DPO margin with KTO's prospect-theoretic utility margins, adding a self-regulating adaptive baseline that tightens the effective trust region as the policy drifts, and an asymmetric weighting that handles imbalanced or asymmetric preference datasets. We provide a unified theoretical analysis, including Kullback Leibler (KL) trust-region bounds, a policy-improvement lower bound, and convergence guarantees under multi-epoch proximal updates. Empirically, P$^2$O and PKTO match or exceed DPO and KTO in alignment quality while maintaining better stability under repeated dataset reuse across multiple LLMs on three preference benchmarks. The code base is available here.

## Don't Blame The Preferences: Examining The Emergence of Sycophancy in LLM Training

DevFeed: [Don't Blame The Preferences: Examining The Emergence of Sycophancy in LLM Training](<https://devfeed.tech/articles/don-t-blame-the-preferences-examining-the-emergence-of-sycophancy-in-llm-training-125873.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10045>)

Author: Chao Peter Yang, Catherine Zehr, Pranav Bijith, Brandon Fain

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [post-training](<https://devfeed.tech/topics/post-training.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [data augmentation](<https://devfeed.tech/topics/data-augmentation.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-sycophancy](<https://devfeed.tech/tags/ai-sycophancy.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [data](<https://devfeed.tech/tags/data.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [interpretability](<https://devfeed.tech/tags/interpretability.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-training](<https://devfeed.tech/tags/llm-training.md>), [llms](<https://devfeed.tech/tags/llms.md>), [mechanistic-interpretability](<https://devfeed.tech/tags/mechanistic-interpretability.md>), [model](<https://devfeed.tech/tags/model.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

Across three model families and three benchmarks, the study finds sycophancy is largely present by the end of supervised fine-tuning (SFT), rather than being mainly introduced by preference alignment. SFT checkpoints score at least 92% as high as preference-aligned models and at least 85% as high as final models; on average they are about as sycophantic as the final models. Preference alignment raises sycophancy in four of nine model-benchmark cases, by an average of 4.2 percentage points in those cases. Mechanistic interpretability analysis finds SFT datasets have roughly the same average activation of sycophancy-related representations as other training data, but contribute substantially more examples; the highest-scoring individual datasets are all from SFT. The authors suggest addressing sycophancy across the training pipeline, particularly through pre-training and SFT data selection and augmentation.

### Source excerpt

arXiv:2611.10045v1 Announce Type: new Abstract: AI sycophancy, the tendency of large language models (LLMs) to flatter users and affirm their beliefs, can undermine accuracy and contribute to misinformation, dependence, and overconfidence. Although prior work has benchmarked sycophancy in released models and proposed mitigations, its origins during training remain poorly understood. Preference alignment methods such as RLHF and DPO are often blamed, but their effects are rarely compared with earlier training stages. We investigate sycophancy across the training pipelines of three model families and three benchmarks. We find that sycophancy is largely established by the end of supervised fine-tuning (SFT), rather than introduced by preference alignment. Across all nine model benchmark combinations, SFT checkpoints attain at least 92% of preference aligned models' sycophancy scores and at least 85% of final models' scores; on average, they are about as sycophantic as final models. Preference alignment increases sycophancy in only four of nine cases, by an average of 4.2 percentage points when it does. To probe the underlying causes, we isolate model representations associated with sycophancy using mechanistic interpretability and measure their activation on training datasets from each stage. SFT datasets show roughly the same average activation as other training data but contribute substantially more examples, and the highest-scoring individual datasets all come from SFT. Our findings suggest that addressing sycophancy requires interventions throughout the training pipeline, particularly in data selection and augmentation for pre-training and SFT.

## Seeing Isn't Enough: Unlocking Visual Reasoning by Bridging the Reasoning-Access Gap with TIGER

DevFeed: [Seeing Isn't Enough: Unlocking Visual Reasoning by Bridging the Reasoning-Access Gap with TIGER](<https://devfeed.tech/articles/seeing-isn-t-enough-unlocking-visual-reasoning-by-bridging-the-reasoning-access-gap-with-tiger-125216.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10552>)

Author: Gregory Kang Ruey Lau (Department of Computer Science, National University of Singapore, Singapore, CNRS@CREATE, Singapore), Minh Huynh Nguyen (Department of Computer Science, National University of S

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [charts](<https://devfeed.tech/tags/charts.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [group-relative-policy-optimization](<https://devfeed.tech/tags/group-relative-policy-optimization.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [image](<https://devfeed.tech/tags/image.md>), [improvements](<https://devfeed.tech/tags/improvements.md>), [input](<https://devfeed.tech/tags/input.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [patching](<https://devfeed.tech/tags/patching.md>), [policy](<https://devfeed.tech/tags/policy.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [rlvr](<https://devfeed.tech/tags/rlvr.md>), [sft](<https://devfeed.tech/tags/sft.md>), [spatial](<https://devfeed.tech/tags/spatial.md>), [spatial-reasoning](<https://devfeed.tech/tags/spatial-reasoning.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>)

### AI overview

The paper argues that multimodal language models can read text in images yet struggle to apply their text-based reasoning to that visual content. TIGER targets this reasoning-access gap by finding synthetic text/image pairs where a model solves the text version but fails on the rendered image, then post-training on the image inputs. The authors report that GRPO, as well as SFT and DPO variants, improves performance on several real-world visual benchmarks despite training only on synthetic rendered-text images without multimodal annotations. TIGER also adds gains when combined with existing RLVR visual-reasoning methods. Activation analyses support the interpretation that training helps connect visual inputs to text reasoning states.

### Source excerpt

arXiv:2611.10552v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) inherit strong textual reasoning capabilities from their language backbones and text pretraining, but often fail to apply the same level of competency to visual inputs. We show that this degradation cannot be attributed to only perception failures, by evaluating models on semantically-equivalent text and rendered-text image inputs: models can accurately read text in the images and recover near-text performance when provided their own transcriptions, but just cannot reason well on the images directly. We characterize this discrepancy as a modality reasoning-access gap, where models fail to route perceived visual content into existing reasoning machinery accessible with text. To bridge this, we propose TIGER (Text-to-Image Gap-targeted training for Enhanced Reasoning), which first mines synthetic modality counterfactuals where a model succeeds on a text problem but fails when provided the corresponding rendered image, and post-trains the model on the image input via Group Relative Policy Optimization (GRPO), along with SFT and DPO variants. Importantly, although trained exclusively on synthetic rendered-text images with no multimodal annotations, TIGER results in model performance improvements across diverse real-world visual benchmarks including geometric diagrams (MathVerse), multi-disciplinary sciences (EMMA, MMMU), complex charts (ChartQAPro), and embodied spatial reasoning (RoboSpatial-Home). TIGER also complements existing state-of-the-art RLVR visual reasoning methods, yielding further performance gains. Analysis involving reasoning-subspace activation and activation patching provide support indicating that TIGER helps to align visual-to-reasoning pathways toward text reasoning states. Our implementation is available at https://github.com/minhngh/Tiger.

## TLRS-DPO: Token-Level Risk Scoring for Better Direct Preference Optimization

DevFeed: [TLRS-DPO: Token-Level Risk Scoring for Better Direct Preference Optimization](<https://devfeed.tech/articles/tlrs-dpo-token-level-risk-scoring-for-better-direct-preference-optimization-125344.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11232>)

Author: Suxin Ji, Hungtao Wan

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [drop-in](<https://devfeed.tech/tags/drop-in.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llms](<https://devfeed.tech/tags/llms.md>), [offline](<https://devfeed.tech/tags/offline.md>), [quality](<https://devfeed.tech/tags/quality.md>), [reference](<https://devfeed.tech/tags/reference.md>), [risk](<https://devfeed.tech/tags/risk.md>), [safety](<https://devfeed.tech/tags/safety.md>), [schedule](<https://devfeed.tech/tags/schedule.md>), [token](<https://devfeed.tech/tags/token.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

TLRS-DPO adds token-level risk control to offline, reference-based DPO, addressing its sequence-level objective's inability to localize risky patterns within otherwise acceptable responses. It derives localized risk scores from policy-reference log-ratios on rejected responses, gradually increases their compensation during training, and selectively penalizes high-risk tokens rather than entire sequences. The authors report improved alignment and safety-relevant behavior with minimal impact on helpfulness versus strong DPO baselines; the method is described as a drop-in modification to standard DPO trainers.

### Source excerpt

arXiv:2611.11232v1 Announce Type: new Abstract: Direct Preference Optimization (DPO) is a simple and widely used alternative to RLHF for aligning large language models (LLMs), but its sequence-level objective cannot precisely locate or suppress the local patterns that make an otherwise acceptable response unsafe or undesirable. We propose TLRS-DPO (Token-Level Risk Scoring for Better Direct Preference Optimization), which augments DPO with fine-grained token-level control while remaining fully offline and reference-based. TLRS-DPO introduces a token-level robust compensation term that converts the policy-reference log-ratio on rejected responses into localized risk scores, a curriculum-style schedule that gradually increases the strength of this compensation during training, and a risk-adaptive screening mechanism that selectively penalizes only high-risk tokens instead of entire sequences. This design bridges sequence-level preference fitting and token-level behavior control, and can be implemented as a drop-in modification to standard DPO trainers. Empirically, TLRS-DPO improves alignment quality and safety-relevant behavior with minimal impact on helpfulness compared to strong DPO baselines.

## Personalized Alignment of Language Models with Human Value Profiles

DevFeed: [Personalized Alignment of Language Models with Human Value Profiles](<https://devfeed.tech/articles/personalized-alignment-of-language-models-with-human-value-profiles-125840.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09919>)

Author: Rashon Poole, Farnaz Jahanbakhsh

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [personalization](<https://devfeed.tech/topics/personalization.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [human feedback](<https://devfeed.tech/topics/human-feedback.md>)

Tags: [activation-steering](<https://devfeed.tech/tags/activation-steering.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-hc](<https://devfeed.tech/tags/cs-hc.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [personalization](<https://devfeed.tech/tags/personalization.md>), [profile](<https://devfeed.tech/tags/profile.md>), [prompting](<https://devfeed.tech/tags/prompting.md>), [survey](<https://devfeed.tech/tags/survey.md>), [value](<https://devfeed.tech/tags/value.md>)

### AI overview

The paper proposes personalizing language model responses around users' stable Schwartz value profiles. For each query, it selects relevant values, then applies targeted prompting, activation steering, or LoRA-DPO. In tests using 1,501 value-tension scenarios and more than 152 value profiles sampled from a nationally representative survey, value-targeted methods outperformed no-personalization, demographic, and full-profile prompting baselines. Steering gains varied across users' positions in the value space. Combining targeted prompting with activation steering or LoRA-DPO produced the strongest personalization while maintaining logical quality within an adequate range.

### Source excerpt

arXiv:2611.09919v1 Announce Type: cross Abstract: Large language models are increasingly used for subjective tasks such as seeking advice, where there is often no single correct answer. Yet dominant alignment methods learn a single behavior policy that applies uniformly across users, limiting their ability to reflect individual value priorities. We propose personalized value alignment, a framework for adapting model responses to users' stable value profiles rather than to personas, demographics, or task-specific preferences used in prior personalization work. Our approach represents users through Schwartz value profiles and selects which of their values are relevant to each query before applying targeted prompting, activation steering, or LoRA-DPO. We create a dataset of 1,501 value-tension scenarios and evaluate over 152 diverse value profiles sampled from a nationally representative survey. Value-targeted methods outperform no-personalization, demographic, and full-profile prompting baselines, indicating that gains come from focused conditioning on query-specific value priorities rather than mere exposure. We further find that steering effects are not uniform across users: alignment gains vary by where a user's priorities fall in the value space. Combining targeted prompting with activation steering or LoRA-DPO yields the strongest personalization while keeping logical quality within adequate range.

## When Scores Conflict with Preferences: Pre-Alignment Diagnosis and Adaptive Correction of Constraint Drift in DPO

DevFeed: [When Scores Conflict with Preferences: Pre-Alignment Diagnosis and Adaptive Correction of Constraint Drift in DPO](<https://devfeed.tech/articles/when-scores-conflict-with-preferences-pre-alignment-diagnosis-and-adaptive-correction-of-constraint-drift-in-dpo-120607.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07375>)

Author: Ping Liu, Yan Yan

Published: 2026-11-06T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [constraint](<https://devfeed.tech/tags/constraint.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diagnosis](<https://devfeed.tech/tags/diagnosis.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>)

### AI overview

In standard DPO, preference pairs whose preferred response scores worse on a pointwise criterion can increase constraint cost, while matching pairs can reduce it; the net effect depends on their balance. The paper introduces a dataset-statistics diagnostic, ρ*eq, to predict whether a dataset is self-correcting or needs intervention. Its Conflict-aware DPO (CoDPO) method uses that diagnostic to set conflict-pair down-weighting and adds an adaptive margin shift to address remaining drift. In the reported experiments, the diagnostic's intervention decisions matched observed drift across all domains. Across four main-comparison domains, CoDPO reduced drift while largely preserving vanilla DPO's preference learning, with a more favorable drift-preference trade-off than hard filtering and the constrained and multi-objective baselines evaluated.

### Source excerpt

arXiv:2611.07375v1 Announce Type: new Abstract: Direct Preference Optimization (DPO) is increasingly used in settings where pairwise preferences coexist with pointwise criteria such as response length, answer correctness, or safety scores. When the two disagree, standard DPO treats pairs whose preferred output scores worse on the pointwise criterion (conflict pairs) in the same way as the remaining (matching) pairs. We characterize a systematic failure mode, constraint drift: for sufficiently small updates, conflict pairs tend to increase constraint cost while matching pairs tend to repair it, and the net change depends on their balance. This balance yields a closed-form pre-alignment diagnostic, $\rho^{*}_{\mathrm{eq}}$, computable from dataset statistics, that indicates whether a dataset is self-correcting or requires intervention. Building on this diagnostic, we propose Conflict-aware DPO (CoDPO), which down-weights conflict pairs with a strength set from $\rho^{*}_{\mathrm{eq}}$ rather than tuned, and adds an adaptive margin shift that addresses residual drift during training. Empirically, drift grows monotonically with conflict exposure on the conflict-dominated domains, and the intervention decision given by $\rho^{*}_{\mathrm{eq}}$ matches the observed drift on every domain, including one that it identifies as self-correcting. On all four domains of the main comparison, CoDPO reduces drift while largely preserving the preference learning of vanilla DPO. Compared with hard filtering and recent constrained and multi-objective baselines, CoDPO also achieves a more favorable drift-preference trade-off.

## The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention

DevFeed: [The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention](<https://devfeed.tech/articles/the-alignment-tax-is-graded-task-selective-capability-loss-under-length-normalized-dpo-with-sft-retention-120535.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.06476>)

Author: Ye Yuan

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [lora](<https://devfeed.tech/topics/lora.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [characterization](<https://devfeed.tech/tags/characterization.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [financial](<https://devfeed.tech/tags/financial.md>), [format](<https://devfeed.tech/tags/format.md>), [json](<https://devfeed.tech/tags/json.md>), [lora](<https://devfeed.tech/tags/lora.md>), [margin](<https://devfeed.tech/tags/margin.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [model](<https://devfeed.tech/tags/model.md>), [objectives](<https://devfeed.tech/tags/objectives.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [retention](<https://devfeed.tech/tags/retention.md>), [sft](<https://devfeed.tech/tags/sft.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [tax](<https://devfeed.tech/tags/tax.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

In a study of Qwen3.5-4B on three structured financial-filing tasks, increasing the implicit reward margin under length-normalized DPO with an SFT retention term produced task-selective capability loss. Across 8 seeds, extraction was preserved within a ±5-point equivalence margin (−0.55 points); JSON performance fell 3.52 points, with equivalence not established; and router performance fell 30 points, mainly in a margin 1-3 damage window that included a transient format-break spike. At least 96% of margin growth came from suppressing rejected completions while chosen reward stayed approximately flat; the study does not assign the behavioral damage to a single gradient channel. Measured reward margins differed by up to 3.4x between nf4 training and fp16 evaluation on the unquantized base. These findings are limited to one 4B model, sparse LoRA, easily separable synthetic preferences, and the tested objective; the authors leave a no-retention control and measured format-alignment metric for future work.

### Source excerpt

arXiv:2611.06476v2 Announce Type: replace Abstract: Setting. This is an empirical characterization study; we propose no new method and run no comparisons against alternative objectives. We fine-tune Qwen3.5-4B with sparse LoRA and length-normalized DPO plus an SFT retention term (the RPO objective) on structured financial-filing tasks, and track downstream capability as the implicit reward margin grows to $\sim 7$ $\beta$-scaled units (nf4 training; $\sim 3$ under fp16 measurement on the unquantized base). Question. Does margin growth degrade capability uniformly across tasks, or selectively? Findings. In a pre-registered study (8 seeds, 6 unseen at freeze) with artifact-aware scoring and TOST equivalence testing, the alignment tax is graded and task-selective: extraction is preserved ($\Delta = -0.55$ pts, equivalent at $\pm5$ pts), json loses little ($\Delta = -3.52$ pts, equivalence fails on a clustered item-level interval), and router degrades severely ($\Delta = -30$ pts). Router's loss concentrates in a margin 1-3 damage window containing a transient format-break spike. The margin is accounted for almost entirely ($\geq 96\%$) by suppression of rejected completions, with chosen reward approximately flat, consistent with the retention term anchoring it; we do not attribute the behavioral damage to any single gradient channel. Gold-logp deflation is task-selective (extraction $-0.15$, json $-1.04$, router $-1.47$ nats/token under fp16), with gold selectively protected relative to incorrect controls on extraction (8/8 seeds). QLoRA reward margins differ up to $3.4\times$ between nf4 training and unquantized fp16 measurement, with an axis-dependent gap. Scope. Length-normalized rewards with SFT retention, sparse LoRA on 8/32 layers, one 4B model, easily-separable synthetic preferences, three structured tasks. A no-retention control and a measured format-alignment metric are left to future work.

## Vision Paper: Why Distributed LLM Post-Training and Decentralized Alignment Are Now Essential

DevFeed: [Vision Paper: Why Distributed LLM Post-Training and Decentralized Alignment Are Now Essential](<https://devfeed.tech/articles/vision-paper-why-distributed-llm-post-training-and-decentralized-alignment-are-now-essential-118389.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05841>)

Author: Zhanhong Jiang, Qisai Liu, Soumik Sarkar

Published: 2026-11-05T05:00:00Z

Content type: opinion

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [post-training](<https://devfeed.tech/topics/post-training.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Federated Learning](<https://devfeed.tech/topics/federated-learning.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [data-sovereignty](<https://devfeed.tech/tags/data-sovereignty.md>), [decentralized](<https://devfeed.tech/tags/decentralized.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [equity](<https://devfeed.tech/tags/equity.md>), [federated](<https://devfeed.tech/tags/federated.md>), [healthcare](<https://devfeed.tech/tags/healthcare.md>), [llm](<https://devfeed.tech/tags/llm.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [research](<https://devfeed.tech/tags/research.md>), [robustness](<https://devfeed.tech/tags/robustness.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [scale](<https://devfeed.tech/tags/scale.md>), [unique](<https://devfeed.tech/tags/unique.md>), [utility](<https://devfeed.tech/tags/utility.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

The paper argues that LLM post-training should support federated and decentralized learning as a core approach, since preference data in fields such as healthcare, law, finance, and multilingual communities may not be suitable for centralization. It grounds the case in privacy and data sovereignty, pluralism and representational equity, and robustness and scalability. It identifies reference drift, heterogeneous preferences, and the privacy-versus-utility trade-off for contrastive losses as technical challenges, and proposes algorithmic directions and a research agenda. These details are described only at abstract level in the supplied text.

### Source excerpt

arXiv:2611.05841v1 Announce Type: new Abstract: This vision paper argues that distributed learning, including federated and decentralized, should become a first-class paradigm in LLM post-training, not merely a privacy add-on. Current methods like RLHF and DPO rest on an invisible assumption: preference data should be centralized. We contend this assumption is scientifically unjustified, practically unsustainable, and ethically problematic. As LLMs deploy into healthcare, law, finance, and multilingual communities, preference data originates from populations that cannot or will not centralize. We develop a three-pillar argument grounded in (1) privacy and data sovereignty, (2) pluralism and representational equity, and (3) robustness and scalability. We formalize key technical challenges unique to this intersection, reference drift, preference heterogeneity, and the DP-utility trade-off for contrastive losses, and propose concrete algorithmic directions, and sketch a research agenda. Our analysis shows distributed post-training is not only necessary but tractable, and ignoring it risks AI that is legally noncompliant, representationally narrow, and brittle at scale.

## Objective-Conditioned Active Preference Learning for Multi-Objective Alignment

DevFeed: [Objective-Conditioned Active Preference Learning for Multi-Objective Alignment](<https://devfeed.tech/articles/objective-conditioned-active-preference-learning-for-multi-objective-alignment-118331.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05506>)

Author: Duc Anh Vu, Thong Nguyen, Hoang Pham, Ponhvoan Srey, Luu Anh Tuan

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [annotation](<https://devfeed.tech/tags/annotation.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [can](<https://devfeed.tech/tags/can.md>), [change](<https://devfeed.tech/tags/change.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [cost](<https://devfeed.tech/tags/cost.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [effectiveness](<https://devfeed.tech/tags/effectiveness.md>), [either](<https://devfeed.tech/tags/either.md>), [existing](<https://devfeed.tech/tags/existing.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [external](<https://devfeed.tech/tags/external.md>), [framework](<https://devfeed.tech/tags/framework.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [learning](<https://devfeed.tech/tags/learning.md>)

### AI overview

OGC-DPO reduces preference-labeling costs for multi-objective LLM alignment by treating each response-pair/objective combination as a query and selecting queries that expand gradient coverage for each objective. It estimates informativeness from the policy's implicit gradient, without an external reward model. In experiments across varying numbers of objectives, it outperformed existing active preference-learning methods at the same annotation cost. The authors also report that informative conflicts between preferences can provide a useful training signal, rather than being uniformly harmful.

### Source excerpt

arXiv:2611.05506v1 Announce Type: cross Abstract: Aligning large language models (LLMs) with several objectives, such as helpfulness, harmlessness, and honesty, typically relies on preference judgment for each objective, growing the annotation cost linearly with the number of objectives. Active preference learning can reduce this cost by selecting the comparisons whose labels are expected to change the policy most. Existing remedies either lack an objective-aware query selection mechanism for the multi-objective setting or regard conflicting preferences as harmful, therefore excluding them from the training data. We introduce OGC-DPO (Objective-conditioned Gradient Coverage DPO), an active learning framework that treats each (response pair, objective) combination as a separate query and measures its informativeness through the policy's own implicit gradient, without relying on an external reward model. OGC-DPO then selects the queries that most expand each objective's gradient coverage. Empirical experiments across varying numbers of objectives demonstrate the effectiveness of OGC-DPO over existing active preference learning methods under the same annotation cost. Further analysis demonstrates that informative conflicts, rather than being uniformly harmful, provide a useful policy training signal.

## Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization

DevFeed: [Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization](<https://devfeed.tech/articles/aligning-language-models-with-general-preferences-an-empirical-study-of-direct-nash-optimization-117064.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05373>)

Author: Austin Tsai, Saurav Nanda, Ganapathy Parthasarathy, Bhuvnesh Kumar, Parivesh Choudhary

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [llama](<https://devfeed.tech/topics/llama.md>), [open-llm-leaderboard](<https://devfeed.tech/topics/open-llm-leaderboard.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [llama-3-1](<https://devfeed.tech/tags/llama-3-1.md>), [margin](<https://devfeed.tech/tags/margin.md>), [mistral](<https://devfeed.tech/tags/mistral.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [weight](<https://devfeed.tech/tags/weight.md>)

### AI overview

DNO-Prct's iterative preference-training procedure improved AlpacaEval 2.0 length-controlled win rates after three iterations: Llama 3.1 8B Instruct rose from 31.7% to 45.3% against GPT-4o, and Mistral 7B Instruct v0.1 from 6.8% to 18.4%. The procedure samples model responses on-policy, has an LLM rate them alongside a teacher's gold response, selects large-margin preference pairs, then applies a DPO-style update. Same-family open-weight annotators retained 62% of Llama's and 82% of Mistral's gain relative to GPT-4o annotation. Ablations removing gold responses, data look-back, or non-teacher-preferred pairs scored 6-10 points lower; changing sample count or margin had smaller reported effects. These are single-run results, with GPT-4o serving as both teacher and judge.

### Source excerpt

arXiv:2611.05373v1 Announce Type: new Abstract: Direct Nash Optimization (DNO) aligns a language model to a general preference function by approximating the Nash equilibrium of a two-player preference game. Its practical variant, DNO-Prct, iterates on-policy sampling, LLM-based rating of the samples together with a teacher's gold response, large-margin preference-pair selection and a DPO-style contrastive update. We re-implement DNO-Prct independently and study it on two open-weight instruct models, Llama 3.1 8B Instruct and Mistral 7B Instruct v0.1, with GPT-4o or a larger open-weight model of the same family as the annotator. On AlpacaEval 2.0 with GPT-4o as judge, three iterations raise the length-controlled (LC) win rate against GPT-4o from 31.7% to 45.3% for Llama and from 6.8% to 18.4% for Mistral, and the tuned 8B model scores above Llama 3.1 70B Instruct on every reported win rate. With same-family open-weight annotators, the runs retain 62% (Llama) and 82% (Mistral) of the LC gain achieved with GPT-4o as annotator, so the rating step does not require a proprietary model. In a separate ablation configuration, variants that remove the gold responses, remove the data look-back, or accept only teacher-preferred pairs score 6 to 10 LC points below the full procedure, whereas varying the number of on-policy samples per prompt (1 to 7) moves the LC win rate by at most 2.1 points, and a margin sweep (1 to 5) run in its own pipeline by at most 2.4 points. We also report the cost of the pipeline and where the practical algorithm departs from the theory it approximates. All results are single runs, and GPT-4o is both the teacher and the evaluation judge; we discuss what this implies for the reported gains.

## TopoDPO: Dense Preference Alignment for Structure-Sensitive Aerial Referring Segmentation

DevFeed: [TopoDPO: Dense Preference Alignment for Structure-Sensitive Aerial Referring Segmentation](<https://devfeed.tech/articles/topodpo-dense-preference-alignment-for-structure-sensitive-aerial-referring-segmentation-115178.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.01551>)

Author: Huaiyuan Peng, Jisheng Dang, Wencan Zhang, Jingwen Zhao, Hong Peng, Wenjun Ma, Bin Hu, Qi Tian, Tat-Seng Chua

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [bootstrap](<https://devfeed.tech/tags/bootstrap.md>), [code](<https://devfeed.tech/tags/code.md>), [component](<https://devfeed.tech/tags/component.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [error](<https://devfeed.tech/tags/error.md>), [framework](<https://devfeed.tech/tags/framework.md>), [image](<https://devfeed.tech/tags/image.md>), [intersection](<https://devfeed.tech/tags/intersection.md>), [local](<https://devfeed.tech/tags/local.md>), [margins](<https://devfeed.tech/tags/margins.md>), [masks](<https://devfeed.tech/tags/masks.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [mining](<https://devfeed.tech/tags/mining.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [objectives](<https://devfeed.tech/tags/objectives.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [structure](<https://devfeed.tech/tags/structure.md>)

### AI overview

TopoDPO addresses the mismatch between spatially varied mask preferences and image-level preference objectives for aerial referring segmentation. It selects preference pairs using connected-component discrepancy and macro-corner recall, then applies preference penalties to local margins before spatial aggregation against a frozen supervised reference. On the 4,590-instance Topo-RS-Hard set, it raises IoU from 50.91% to 51.24% over image-level DPO and lowers Betti-0 error from 1.34 to 0.90. Reported paired bootstrap intervals exclude zero for all five metrics. Analysis across 39,244 validation records finds the clearest structural gains where pre-alignment component-count errors are moderate or severe.

### Source excerpt

arXiv:2611.01551v1 Announce Type: cross Abstract: Preference optimization for multimodal segmentation faces a granularity mismatch: dense masks contain spatially heterogeneous preference signals, whereas image-level objectives assign one shared nonlinear weight to all spatial margins. We introduce TopoDPO, a topology-aware dense preference alignment framework for structure-sensitive aerial referring segmentation. Topo-Prior mines informative preference pairs using connected-component discrepancy and macro-corner recall, while the dense objective applies the preference penalty to each local margin before spatial aggregation against a frozen supervised reference. On Topo-RS-Hard, a 4590-instance structural stress set, TopoDPO improves intersection over union (IoU) from 50.91% to 51.24% over image-level direct preference optimization (DPO) and reduces Betti-0 error from 1.34 to 0.90; paired image-cluster bootstrap intervals against image-level DPO exclude zero for all five reported metrics. Matched supervised controls and a complete factorial analysis separate the effects of topology-aware pair mining, localized correction, and dense reference-relative preference weighting. An unfiltered stratification over all 39,244 validation records further shows the clearest structural gains on samples with moderate or severe pre-alignment component-count errors. The code is publicly available at: https://github.com/phy709/TopoDPO.

## Axion 1.0: Open Agent Foundation Models, Simplified

DevFeed: [Axion 1.0: Open Agent Foundation Models, Simplified](<https://devfeed.tech/articles/axion-1-0-open-agent-foundation-models-simplified-115438.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.03233>)

Author: Yuqi Jia, Yunjie Tian, Jiayu Zheng, Yijun Liang, Jingchen Sun, Mengyu Zhao, Di Fu

Published: 2026-11-03T05:00:00Z

Content type: release

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [rlvr](<https://devfeed.tech/topics/rlvr.md>), [dpo](<https://devfeed.tech/topics/dpo.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [axion](<https://devfeed.tech/tags/axion.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [data](<https://devfeed.tech/tags/data.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [environments](<https://devfeed.tech/tags/environments.md>), [external](<https://devfeed.tech/tags/external.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [foundation-model](<https://devfeed.tech/tags/foundation-model.md>), [foundation-models](<https://devfeed.tech/tags/foundation-models.md>), [model-weights](<https://devfeed.tech/tags/model-weights.md>), [open](<https://devfeed.tech/tags/open.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [release](<https://devfeed.tech/tags/release.md>), [sft](<https://devfeed.tech/tags/sft.md>), [simple](<https://devfeed.tech/tags/simple.md>), [tools](<https://devfeed.tech/tags/tools.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

Axion 1.0 uses a three-stage training recipe: two lightweight supervised fine-tuning stages followed by reinforcement learning, without dedicated mid-training, DPO, or on-policy distillation. The first SFT stage builds self-contained reasoning with challenging problems strengthened using the base model's successful solution strategies. The second adds tool-dependent reasoning through automatically verifiable tasks grounded in real-world data and deterministic computation; reinforcement learning targets longer-horizon reasoning and interaction with tools and environments. The abstract reports strong results across agentic benchmarks and competitiveness with leading commercial models on several. The release includes the recipe, data construction and processing methods, prompts, and training code, intended to support reproduction and extension.

### Source excerpt

arXiv:2611.03233v1 Announce Type: new Abstract: We introduce Axion 1.0, a fully open agent foundation model that combines strong and broad agentic capabilities with a remarkably simple training recipe. Axion 1.0 is trained in only three stages: two lightweight supervised fine-tuning stages followed by reinforcement learning, without dedicated mid-training, direct preference optimization (DPO), or on-policy distillation (OPD). The first SFT stage develops strong self-contained reasoning through model-aware data construction, where we synthesize challenging problems and adversarially strengthen them based on the base model's own successful solution strategies. The second stage extends these capabilities to tool-dependent reasoning through a scalable pipeline for constructing diverse and automatically verifiable tasks grounded in real-world data sources and deterministic computation. Finally, reinforcement learning further strengthens long-horizon reasoning and interaction with external tools and environments. Despite this compact training recipe, Axion 1.0 achieves strong performance across a broad suite of agentic benchmarks and is competitive with leading commercial models on several of them. Equally importantly, Axion 1.0 is open beyond model weights: we release the complete recipe, including data construction, generation, filtering, verification, prompts, and training code, enabling the entire development process to be reproduced and extended. Axion 1.0 demonstrates that strong agent foundation models can be both open and simple, offering a practical and reproducible path toward advancing open agentic intelligence.

## Predicting Human-Gain Curves under Local Proxy Optimization from Repeated Ratings

DevFeed: [Predicting Human-Gain Curves under Local Proxy Optimization from Repeated Ratings](<https://devfeed.tech/articles/predicting-human-gain-curves-under-local-proxy-optimization-from-repeated-ratings-112027.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.41114>)

Author: Moonwon Choi, Kisung Nam, Seunggeun Lee

Published: 2026-11-02T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [human feedback](<https://devfeed.tech/topics/human-feedback.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [lora](<https://devfeed.tech/topics/lora.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [branch](<https://devfeed.tech/tags/branch.md>), [bridge](<https://devfeed.tech/tags/bridge.md>), [but](<https://devfeed.tech/tags/but.md>), [can](<https://devfeed.tech/tags/can.md>), [changes](<https://devfeed.tech/tags/changes.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [correlation](<https://devfeed.tech/tags/correlation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [d](<https://devfeed.tech/tags/d.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [error](<https://devfeed.tech/tags/error.md>), [local](<https://devfeed.tech/tags/local.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [proxy](<https://devfeed.tech/tags/proxy.md>), [under](<https://devfeed.tech/tags/under.md>)

### AI overview

PACE uses repeated human ratings to estimate how human-perceived quality may change as a fixed proxy is optimized gently. It models the local gain at KL radius d as A√d + Bd, with A capturing first-order proxy-human alignment and B its leading second-order correction. The authors report lower held-out gain-prediction error than A-only, zero-gain, and reliability-only baselines across several evaluation datasets and a GPT-4o judge proxy experiment. PACE is competitive with flexible proxy-conditional calibration given the same pre-pool information; a LoRA-DPO experiment offers a modest link to actual optimization. Source coefficients help transfer across datasets, while target calibration improves predictions. The diagnostic is limited to local, one-sided optimization along the positive tilt branch of a fixed item pool.

### Source excerpt

arXiv:2610.41114v1 Announce Type: new Abstract: Optimizing a proxy objective can improve human-perceived quality, but the resulting gain can vary with optimization strength. Average proxy-human correlation alone does not reveal how human-perceived quality changes as optimization strength increases. We propose PACE (Proxy Alignment and Curvature Expansion), a pre-optimization diagnostic that uses repeated human ratings to predict the human-gain curve induced by a fixed proxy. The method models proxy optimization as a KL-regularized exponential tilt. At KL radius $d$, the local gain is approximated by $A\sqrt{d}+Bd$, where $A$ measures first-order proxy-human alignment and $B$ captures the leading second-order correction along the proxy-optimization path. We estimate $A$ and $B$ from repeated human ratings on calibration items and test the resulting gain-curve prediction on held-out evaluation items. Across SummEval, analysis-eligible WMT MQM clusters, an OpenMEVA-MAGS ROC/WP repeated-rating slice, and a GPT-4o judge proxy experiment, PACE consistently reduces held-out gain-prediction error relative to $A$-only, zero-gain, and reliability-only baselines. With the same pre-pool information, it is competitive with flexible proxy-conditional calibration, and a LoRA-DPO experiment provides a modest bridge to actual optimization. Transfer experiments across datasets and evaluation slices show that source coefficients are useful priors, but target calibration further improves prediction. The resulting diagnostic is local and one-sided: it characterizes gentle optimization along the positive tilt branch of a fixed item pool.

## Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning

DevFeed: [Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning](<https://devfeed.tech/articles/reliability-weighted-direct-preference-optimization-penalized-weights-for-robust-preference-learning-111601.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.41588>)

Author: Bin Luo, Param Venkat Vivek Kesireddy, Anil Vallepu

Published: 2026-11-02T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [annotation](<https://devfeed.tech/tags/annotation.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [error](<https://devfeed.tech/tags/error.md>), [estimator](<https://devfeed.tech/tags/estimator.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [learning](<https://devfeed.tech/tags/learning.md>), [linear](<https://devfeed.tech/tags/linear.md>), [margin](<https://devfeed.tech/tags/margin.md>), [margins](<https://devfeed.tech/tags/margins.md>), [minimax](<https://devfeed.tech/tags/minimax.md>), [model](<https://devfeed.tech/tags/model.md>), [noise](<https://devfeed.tech/tags/noise.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [policy](<https://devfeed.tech/tags/policy.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [robust](<https://devfeed.tech/tags/robust.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

RW-DPO jointly learns a preference policy and per-comparison reliability weights, addressing noisy or flipped labels that standard DPO treats as correct. A minimax-concave penalty on the negative log weight leaves comparisons the model agrees with at weight 1 and downweights those it contradicts; closed-form updates add negligible cost over DPO. The paper proves flipped comparisons cannot all be recovered from one annotation each without margins of order log n, and gives consistency and recovery results in a linear-index model under a correctly directed initial fit, including when a fixed fraction of labels is flipped. Simulations report lower average estimation error than DPO and other compared robust methods, with more precise flags than label-correction methods. In language-model fine-tuning with 10-30% flipped labels, reported win rate does not decline as the flip rate rises and exceeds DPO trained on clean labels, matching the strongest compared method.

### Source excerpt

arXiv:2610.41588v1 Announce Type: cross Abstract: Direct Preference Optimization (DPO) treats every reported comparison as correct, yet preference data contain reversed, tied or otherwise noisy comparisons. RW-DPO gives each comparison an unknown reliability weight $w_i\in[0,1]$ that enters the likelihood linearly -- DPO is $w\equiv1$, a reversed comparison $w_i=0$ -- and estimates the weights jointly with the policy under a minimax-concave penalty on $-\log w_i$. The penalty keeps every comparison the model agrees with at exactly $w_i=1$ and acts only on those it contradicts, through closed-form weight updates at negligible cost over DPO. We prove that recovering all flipped comparisons from a single annotation per comparison requires margins of order $\log n$. Furthermore, in a linear-index model, starting from an initial fit pointing in the correct direction, our estimator jointly consistently recovers the preference direction and weights, as well as identifies the flipped comparisons above this margin scale, even when a fixed fraction of comparisons is flipped. In simulations RW-DPO has the lowest average estimation error among DPO, noise-rate corrections and recent robust methods, with far more precise flags than recent label-correction methods. In DPO fine-tuning of a language model with 10-30% flipped labels, its win rate does not fall as the flip rate grows and exceeds that of DPO trained on clean labels, matching the strongest recent method with corrections concentrated on the flipped labels.

## SecPO: Principled Adversarial Training for Prompt Injection Security

DevFeed: [SecPO: Principled Adversarial Training for Prompt Injection Security](<https://devfeed.tech/articles/secpo-principled-adversarial-training-for-prompt-injection-security-111235.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.41998>)

Author: Sizhe Chen, Yibo Peng, Jaewon Chang, Chawin Sitawarin, David Wagner

Published: 2026-11-02T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [prompt injection](<https://devfeed.tech/topics/prompt-injection.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [Adversarial attacks](<https://devfeed.tech/topics/adversarial-attacks.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [dpo](<https://devfeed.tech/topics/dpo.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [adversarial](<https://devfeed.tech/tags/adversarial.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-workflows](<https://devfeed.tech/tags/agentic-workflows.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [asr](<https://devfeed.tech/tags/asr.md>), [attack](<https://devfeed.tech/tags/attack.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [data](<https://devfeed.tech/tags/data.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [email](<https://devfeed.tech/tags/email.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompt-injection](<https://devfeed.tech/tags/prompt-injection.md>), [security](<https://devfeed.tech/tags/security.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

SecPO combines a theoretically motivated training loss with adversarial training to improve prompt injection resistance. The authors argue DPO can be limited by anchoring to an insecure reference model; SecPO instead approximates an ideal injection-invariant, utility-preserving reference. Their training pipeline synthesizes stronger injected inputs (best@240). Against Meta-SecAlign, adversarial training alone lowers attack success rates by 1.5-14.5x across three adaptive attacks, and SecPO lowers them by a further 2.4-4.6x versus DPO. On AgentDojo under AttackerMovesSecond, SecPO plus adversarial training with Qwen3.6-27B has 8.8% ASR@800 versus 78.8% for Meta-SecAlign. The authors report no noticeable benign utility reduction across nine benchmarks spanning tool calling, instruction following, science, and math. This overview is based on the supplied abstract.

### Source excerpt

arXiv:2610.41998v1 Announce Type: new Abstract: Recognized as a top threat to AI agents, prompt injection allows instructions from untrusted data (email, documents, websites) to manipulate the agent's operation. The existing best-defended LLM, Meta-SecAlign, is trained with the DPO loss but still suffers from attack success rates (ASRs) approaching 100% under adaptive prompt injections. One reason is that DPO anchors the model around the undefended reference model, which is insecure. We show an idealized reference model---an injection-invariant, utility-preserving oracle---can nevertheless be approximated, as its output distribution in training is interestingly accessible. This new observation motivates a theoretically principled SecPO loss. Another Meta-SecAlign shortcoming is that it trains on weak simulated attacks. We implement an efficient adversarial training (AT) pipeline that synthesizes strong injected inputs (best@240), and train the model on them. Our AT reduces ASRs by 1.5-14.5$\times$ across three SoTA adaptive attacks, compared with Meta-SecAlign; the SecPO loss reduces them by a further 2.4-4.6$\times$ compared with using DPO loss. In agentic workflows, our SecPO + AT Qwen3.6-27B is much more secure than Meta-SecAlign: 8.8% vs. 78.8% AgentDojo ASR@800 under the AttackerMovesSecond attack. Moreover, our defended model shows no noticeable reduction in benign utility in nine benchmarks covering agentic tool calling, instruction following, science, and math. We release our $\href{//github.com/Sizhe-Chen/SecPO}{\text{code}}$, $\href{//huggingface.co/datasets/Sizhe-Chen/Qwen3.6-27B-Instruct-SecPO-trainset}{\text{training data}}$ (with optimized injections from AT), and $\href{//huggingface.co/Sizhe-Chen/Qwen3.6-27B-Instruct-SecPO}{\text{model}}$.

[Next page](<https://devfeed.tech/tags/dpo.md?cursor=WyIyMDI2LTExLTAyVDA1OjAwOjAwKzAwOjAwIiwgIjQ0ZGZjYzE5LTU4OTItNGVlYy1hMDYzLTcxMGUxZjUzNjQyMiJd>)