# rlhf

Reinforcement learning from human feedback (RLHF) is a method using preference modeling and reinforcement learning to fine-tune language models.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging

DevFeed: [From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging](<https://devfeed.tech/articles/from-weak-signals-to-strong-models-preference-delta-aggregation-with-lora-merging-145341.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.00357>)

Author: Qi Sun, Siyue Zhang, Yulin Chen, Yuxiang Xue, Ru Peng, Chen Zhao

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [lora](<https://devfeed.tech/topics/lora.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [adapter](<https://devfeed.tech/tags/adapter.md>), [adapters](<https://devfeed.tech/tags/adapters.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [data](<https://devfeed.tech/tags/data.md>), [delta](<https://devfeed.tech/tags/delta.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llms](<https://devfeed.tech/tags/llms.md>), [lora](<https://devfeed.tech/tags/lora.md>), [models](<https://devfeed.tech/tags/models.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [pairs](<https://devfeed.tech/tags/pairs.md>), [pda](<https://devfeed.tech/tags/pda.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>)

### AI overview

Preference Delta Aggregation (PDA) turns preference comparisons between weaker models into separate LoRA adapters trained through preference optimization, then merges them to improve a stronger model. Its Geometric Alignment Merging (GAM) aligns the adapters' left and right singular bases with orthogonal Procrustes rotations before averaging their factors. In evaluations on knowledge reasoning and agentic search, PDA-GAM outperformed the strongest individual delta. Three reasoning deltas and four agentic deltas improved Qwen3-8B's average scores by 6.8 and 7.3 points, respectively--2.1 and 4.3 points above the strongest single-delta baselines. The abstract reports similar gains on a second student model and attributes the results to combining complementary capabilities.

### Source excerpt

arXiv:2606.00357v3 Announce Type: replace Abstract: Training strong large language models (LLMs) requires high-quality supervision, which is often scarce. Recent work shows that paired preference data from weak-weaker model pairs (e.g., Qwen3 4B over 1.7B), despite the limited quality of individual responses, can provide an effective supervision signal through relative quality deltas, which we term a "weak" signal. This motivates a key research question: can multiple "weak" signals be constructively aggregated for improving strong models (e.g., Qwen3 8B)? To this end, we propose Preference Delta Aggregation (PDA), the first framework that derives a preference delta from each weak-weaker model pair, instantiates it as a LoRA adapter learned through preference optimization, and aggregates the resulting deltas via LoRA merging. To improve the composition of independently trained adapters, we introduce Geometric Alignment Merging (GAM), which aligns their left and right singular bases through orthogonal Procrustes rotations and averages the aligned factors. Evaluations on knowledge reasoning and agentic search show that PDA-GAM exceeds the strongest individual delta, with further gains along the evaluated sequences of additional deltas. Using three reasoning deltas and four agentic deltas, PDA-GAM improves Qwen3-8B by 6.8 and 7.3 points on average, respectively, exceeding the strongest single-delta baseline by 2.1 and 4.3 points. These configurations achieve the highest average scores among the evaluated merging methods, with similar gains observed on a second student model. Further analysis suggests that these gains come from composing complementary capabilities encoded across distinct preference deltas. Code is available at https://github.com/AlbertQiSun/Preference-Delta-Aggregation.

## Learning through Discernment: when a small language model learns from user feedback on the job, and when it decays

DevFeed: [Learning through Discernment: when a small language model learns from user feedback on the job, and when it decays](<https://devfeed.tech/articles/learning-through-discernment-when-a-small-language-model-learns-from-user-feedback-on-the-job-and-when-it-decays-142695.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23716>)

Author: Sirisilp Kongsilp

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [lora](<https://devfeed.tech/topics/lora.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [On-device AI](<https://devfeed.tech/topics/on-device-ai.md>)

Tags: [adapter](<https://devfeed.tech/tags/adapter.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [job](<https://devfeed.tech/tags/job.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [learning](<https://devfeed.tech/tags/learning.md>), [small-language-model](<https://devfeed.tech/tags/small-language-model.md>), [user](<https://devfeed.tech/tags/user.md>)

### AI overview

In experiments with Qwen3 0.6B and 1.7B plus LoRA adapters, nightly learning from user feedback depended on whether the judge filtered out failures and whether users could see outcomes. A feedback sense that treated polite give-ups as success collapsed in 3 of 5 lives. A version that first checked for a clear user verdict stayed near oracle performance in 4 of 5 lives for two weeks, then faded to self-judging levels by day 60 as clean lessons became scarce. On τ²-bench, it interpreted reactions correctly, but 80% of lessons were failed attempts users had thanked; across 97 lives, the share of failed lessons strongly predicted outcomes (Spearman ρ = −0.81). The abstract concludes that user-guided learning held up only when failures were excluded and users could observe outcomes, and warns that skewed lessons may narrow the model.

### Source excerpt

arXiv:2611.23716v1 Announce Type: cross Abstract: Small language models on devices could keep improving by learning, overnight, from their own daily work. Without a true outcome, however, something must decide which attempts are worth learning from, and both the model's own judgment and the user's reaction can be wrong. We study a small model (Qwen3 0.6B and 1.7B) with a LoRA adapter, fine-tuned each night on the attempts a judge accepts, over 127 lives of up to a year on WebShop and $\tau^2$-bench. We compare the true outcome, the user's stated verdict and the model's own judgment with a separate, frozen "feedback sense" that reads the user's reaction, built in three versions. Judging itself, the model improved and then decayed, over a year and whatever its adapter. A sense that took polite give-ups for success collapsed 3 of 5 lives; one built for discernment, which first asks whether the user gave a clear verdict, kept 4 of 5 near the oracle for two weeks, then faded on clean but scarce lessons, to the level of self-judging by day 60. When the user cannot see the outcome ($\tau^2$-bench), the sense read reactions correctly, yet 80% of its lessons were failures the customer had thanked. Across 97 lives, the share of failed attempts among the lessons predicted the outcome (Spearman $\rho$ = -0.81). In our experiments, learning from users held up only where the judge kept failures out and the user could see the outcome; lopsided lessons may also narrow the model.

## Reading Human Preference from Looped Transformer Internal States

DevFeed: [Reading Human Preference from Looped Transformer Internal States](<https://devfeed.tech/articles/reading-human-preference-from-looped-transformer-internal-states-142516.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2604.09870>)

Author: Jan Kirin

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>)

Tags: [anthropic](<https://devfeed.tech/tags/anthropic.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [controls](<https://devfeed.tech/tags/controls.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [edge](<https://devfeed.tech/tags/edge.md>), [hh](<https://devfeed.tech/tags/hh.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [layers](<https://devfeed.tech/tags/layers.md>), [linear](<https://devfeed.tech/tags/linear.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [network](<https://devfeed.tech/tags/network.md>), [pair](<https://devfeed.tech/tags/pair.md>), [property](<https://devfeed.tech/tags/property.md>), [rlhf](<https://devfeed.tech/tags/rlhf.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

On Ouro-2.6B-Thinking, a frozen model that runs its layers four times per token, probes of hidden states identify the human-preferred reply in Anthropic's HH-RLHF data only modestly better than simple rules: 61.5% for a linear probe, 64.9% for a nonlinear network scoring replies separately, and 64.7% for the version 1 evaluator on the test split. A network scoring both replies together has an estimated 0.7-point advantage over separate scoring (95% CI 0.0-1.4), leaving open whether preference is relational or a property of each reply. Results account for both pair orders, pair-level splits, and-- for probes--conversation endings where replies differ. Earlier version 1 headline scores (95.2% and 84.5%) fail the first two controls; version 2's relational-probe accuracy was four to five points too low because it omitted the third.

### Source excerpt

arXiv:2604.09870v3 Announce Type: replace-cross Abstract: Show a frozen language model two replies to the same request, one of which a person preferred. Can a small probe on the model's hidden states tell which? On Ouro-2.6B-Thinking, a model that runs its layers four times per token, and Anthropic's HH-RLHF data, the answer is yes, but only faintly. A linear probe picks the preferred reply 61.5% of the time, a small nonlinear network that scores each reply separately 64.9%, and the evaluator trained for version 1 reaches 64.7% on the test split: clearly better than simple rules such as picking the longer reply, but weak. Whether preference is stored as a comparison between the two replies or as a property of each is still open. A linear probe cannot tell, and a network that reads both replies together has an estimated edge of only 0.7 points (95% CI 0.0 to 1.4) over one that scores each reply separately. These numbers hold under three controls: scoring every pair in both orders, splitting the data by pair, and, for the probes, keeping the end of each conversation, where the two replies differ. Version 1's 95.2% and 84.5% were obtained without the first two and do not hold; version 2's corrected relational-probe accuracy, obtained without the third, was four to five points too low. This version replaces both.

## EVA-RSI: Recursive Self-Improvement of Video Evaluators for Judging Temporal Quality

DevFeed: [EVA-RSI: Recursive Self-Improvement of Video Evaluators for Judging Temporal Quality](<https://devfeed.tech/articles/eva-rsi-recursive-self-improvement-of-video-evaluators-for-judging-temporal-quality-142718.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23890>)

Author: Yu Yuan, Ruizhou Ding, Shaoheng Fang, Stanley H. Chan, Zhuo Chen

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [AI response quality](<https://devfeed.tech/topics/ai-response-quality.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>)

Tags: [agent-framework](<https://devfeed.tech/tags/agent-framework.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [motion](<https://devfeed.tech/tags/motion.md>), [multi-agent](<https://devfeed.tech/tags/multi-agent.md>), [physics](<https://devfeed.tech/tags/physics.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [recursive-self-improvement](<https://devfeed.tech/tags/recursive-self-improvement.md>), [rl](<https://devfeed.tech/tags/rl.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [video](<https://devfeed.tech/tags/video.md>), [video-generation](<https://devfeed.tech/tags/video-generation.md>)

### AI overview

EVA-RSI improves video motion evaluation through recursive, multi-agent updates despite lacking a direct verifier and balancing several evaluation dimensions. It uses controlled video edits, such as temporal reversal and frame dropping, to create a fixed proxy exam with construction-defined targets, plus a small set of human-labeled real videos to anchor judgments. Dimension-specific LLM agents propose changes; a leader coordinates them, and Nash bargaining selects updates. Humans design the exam, review candidate changes, and revise the curriculum between stages. Starting from Qwen3-VL-8B, the method reports its largest gain in smoothness and transfer to unseen defect types. It also reports the highest observed scores among 21 tested models on public benchmark dynamics subsets, and increased motion when used as an RL reward or in data selection; the reward also earned human preference.

### Source excerpt

arXiv:2611.23890v1 Announce Type: new Abstract: Video evaluators play an important role in video generation, yet they miss motion defects, limiting the quality of generated videos. Recursive self-improvement (RSI) has made strong progress in mathematics and coding, where explicit verifiers provide reliable feedback on a single goal. Video evaluation poses two difficulties for RSI: it has no verifier, and it must judge many dimensions at once, such as physics, smoothness and dynamics, where a gain on one can cost another. We introduce EVA-RSI, a multi-agent framework for exploring RSI under these two constraints. We build a fixed proxy exam with construction-defined targets by applying controlled changes to videos, such as temporal reversal and frame dropping. These edits supply supervision without human annotation. A small set of human-labeled real videos anchors the exam to human judgment. Each dimension has an LLM agent that proposes improvements. A leader agent coordinates the proposals, and a Nash bargaining rule selects updates that balance progress across dimensions. Humans design the exam, review candidate changes and revise the curriculum between stages. Starting from Qwen3-VL-8B, EVA-RSI improves motion evaluation, with the largest gain on smoothness, and the gains carry over to defect types never seen during the run. On the dynamics subsets of public benchmarks, our evaluator achieves the highest observed scores among 21 tested models. As an RL reward, it increases generated motion and earns human preference. A data-selection pipeline incorporating it also increases motion.

## GEM-MM: Reducing VLM Hallucination with Entropy-Guided Preference Alignment

DevFeed: [GEM-MM: Reducing VLM Hallucination with Entropy-Guided Preference Alignment](<https://devfeed.tech/articles/gem-mm-reducing-vlm-hallucination-with-entropy-guided-preference-alignment-137524.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20016>)

Author: Yuanshan Chua, Xuejiao Zhao

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [post-training](<https://devfeed.tech/topics/post-training.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [chain-of-thought](<https://devfeed.tech/tags/chain-of-thought.md>), [chair](<https://devfeed.tech/tags/chair.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [policy](<https://devfeed.tech/tags/policy.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [rlhf](<https://devfeed.tech/tags/rlhf.md>), [vlm](<https://devfeed.tech/tags/vlm.md>)

### AI overview

With a 3,000-prompt preference budget, GEM-MM reduces reported vision-language model hallucination rates while outperforming the compared alignment baselines on several preference metrics. The method samples on-policy candidates, emphasizes high-entropy reasoning forks where visual claims are made, penalizes uncertain final answers, and updates the model with group-normalized policy gradients. In experiments using Qwen3-VL-4B-Instruct, AMBER hallucination falls to 20.6 from 23.4-24.0 and CHAIR to 4.6 from 5.0-5.1 at matched caption coverage; the entropy-depth preference score reaches 61.7%, versus 57.3-58.4% for baselines. The abstract also reports top rankings across three HallusionBench granularities and a 62.4% near-chosen rate at a conservative operating point. It notes a collapse mode when exceeding the fair preference budget.

### Source excerpt

arXiv:2611.20016v1 Announce Type: new Abstract: Vision-language models (VLMs) are routinely aligned with offline pairwise RLHF objectives, but multimodal preference data is scarce and treats every response token as equally informative. Under a small preference budget, pair construction becomes nearly irrelevant: standard DPO-style, image-conditional, and self-improvement objectives all perform within 1.1 points on MM-RLHF metrics and 0.5 points on hallucination rates, matching the SFT baseline. We call this entropy-blind preference optimization: the learning signal ignores policy uncertainty, which in chain-of-thought VLMs concentrates on specific fork tokens where visual claims are made. We introduce GEM-MM, extending entropy-guided preference modeling to VLMs. It samples on-policy candidates, rewards high-entropy reasoning forks, penalizes uncertain final answers, and updates the model via group-normalized policy gradients. On a 3000-prompt MM-RLHF budget with Qwen3-VL-4B-Instruct, GEM-MM achieves the best entropy-depth preference score (61.7% vs. 57.3-58.4% baselines) and implicit-reward accuracy (41.4%). Crucially, it reduces AMBER hallucination from 23.4-24.0 to 20.6 and CHAIR from 5.0-5.1 to 4.6 at matched caption coverage. It also ranks best on all three HallusionBench granularities, with a conservative operating point raising the near-chosen rate to 62.4%. An off-the-shelf multimodal reward model ranks GEM-MM first, and two such models independently confirm our main ablations (p < 0.001). We report confidence intervals, paired significance tests, metric-reward functional overlaps, and document a collapse mode when exceeding the fair budget. Code and evaluation harness: https://github.com/SNOWTEAM2023/GEM-MM

## Deep Learning for Physicists: From Linear Regression to Reasoning Models

DevFeed: [Deep Learning for Physicists: From Linear Regression to Reasoning Models](<https://devfeed.tech/articles/deep-learning-for-physicists-from-linear-regression-to-reasoning-models-139151.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21295>)

Author: Masanori Hanada

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Deep learning](<https://devfeed.tech/topics/deep-learning.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>)

Tags: [adversarial](<https://devfeed.tech/tags/adversarial.md>), [attention](<https://devfeed.tech/tags/attention.md>), [backpropagation](<https://devfeed.tech/tags/backpropagation.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [gradient-descent](<https://devfeed.tech/tags/gradient-descent.md>), [hep-lat](<https://devfeed.tech/tags/hep-lat.md>), [hep-th](<https://devfeed.tech/tags/hep-th.md>), [human-feedback](<https://devfeed.tech/tags/human-feedback.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [layer](<https://devfeed.tech/tags/layer.md>), [linear-regression](<https://devfeed.tech/tags/linear-regression.md>), [machine](<https://devfeed.tech/tags/machine.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [natural-language-processing](<https://devfeed.tech/tags/natural-language-processing.md>), [networks](<https://devfeed.tech/tags/networks.md>), [neural](<https://devfeed.tech/tags/neural.md>), [physics-comp-ph](<https://devfeed.tech/tags/physics-comp-ph.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>)

### AI overview

These lecture notes offer an introductory path through machine learning that uses physical intuition and assumes little prior knowledge. They begin with maximum likelihood, least squares, linear regression, and gradient descent, then cover neural networks, training challenges, regularization, adversarial examples, and kernel methods. The second half moves through NLP--from embeddings, recurrent networks, and attention to Transformers and large language models--covering topics such as instruction tuning, human feedback, hallucination, chain-of-thought reasoning, and test-time scaling. The notes are aimed at UK third- and fourth-year students, claim no originality, and do not cover developments after spring 2025.

### Source excerpt

arXiv:2611.21295v1 Announce Type: new Abstract: These lecture notes are based on an introductory course on neural networks and deep learning that the author taught in the School of Mathematical Sciences at Queen Mary University of London in 2023/24 and 2024/25. The author, however, is a physicist, and it shows: wherever possible, ideas are explained with physical intuition rather than heavy formalism. Readers with a background in physics or mathematics who want a self-contained, introductory route into modern machine learning may therefore find these notes particularly friendly. The notes start from maximum likelihood, least squares, linear regression and (stochastic) gradient descent. They then cover neural networks, backpropagation, the vanishing gradient problem, batch and layer normalization, overfitting and regularization, adversarial examples, and the kernel method. The second half turns to natural language processing: word embeddings, recurrent networks and LSTMs, the attention mechanism and Transformers, and large language models, including scaling laws, the history of GPT, instruction tuning, reinforcement learning from human feedback, hallucination, alignment faking, chain-of-thought reasoning and test-time scaling. The emphasis on large language models reflects requests from the students. The notes are aimed at third- and fourth-year students in the UK system. Because the course was also taken by students from other departments, very little is assumed: even basic material such as matrix multiplication is explained from scratch, so readers familiar with it can safely skip such parts. No originality is claimed; the aim is a coherent, intuitive path through the material. Developments after spring 2025 are not covered, but the basic ideas emphasized here are unlikely to go out of date soon.

## Contrastive Activation-Guided Zeroth-Order Preference Optimisation for Memory-Efficient LLM Alignment

DevFeed: [Contrastive Activation-Guided Zeroth-Order Preference Optimisation for Memory-Efficient LLM Alignment](<https://devfeed.tech/articles/contrastive-activation-guided-zeroth-order-preference-optimisation-for-memory-efficient-llm-alignment-139077.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20635>)

Author: Weiyun Huang, Alessio Galatolo

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [backpropagation](<https://devfeed.tech/tags/backpropagation.md>), [behaviour](<https://devfeed.tech/tags/behaviour.md>), [ca](<https://devfeed.tech/tags/ca.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-memory](<https://devfeed.tech/tags/gpu-memory.md>), [hh](<https://devfeed.tech/tags/hh.md>), [human-feedback](<https://devfeed.tech/tags/human-feedback.md>), [learning](<https://devfeed.tech/tags/learning.md>), [llm](<https://devfeed.tech/tags/llm.md>), [memory](<https://devfeed.tech/tags/memory.md>), [momentum](<https://devfeed.tech/tags/momentum.md>), [optimisation](<https://devfeed.tech/tags/optimisation.md>), [order](<https://devfeed.tech/tags/order.md>)

### AI overview

CA-ZODPO reduces activation-memory demands for preference alignment by restricting zeroth-order perturbations to a low-rank subspace built from activation differences between chosen and rejected responses, and keeping momentum in those low-rank coordinates. On Qwen2.5-1.5B with Anthropic HH-RLHF, it outperforms common zeroth-order baselines and is competitive with first-order DPO in pairwise preference evaluation. The abstract reports about 59% lower peak GPU memory at sequence length 2048, while retaining zeroth-order training's sequence-length scaling behavior.

### Source excerpt

arXiv:2611.20635v1 Announce Type: new Abstract: Direct Preference Optimisation (DPO) simplifies reinforcement learning from human feedback, but still relies on traditional backpropagation and its associated activation-memory cost. Zeroth-order (ZO) optimisation avoids backpropagation, yet isotropic perturbations can produce noisy updates for preference objectives defined by differences between chosen and rejected responses. We introduce Contrastive Activation-guided Zeroth-Order Direct Preference Optimisation (CA-ZODPO), which constructs a low-rank search subspace from the activation discrepancy between the chosen and rejected responses in each preference pair. Perturbations are restricted to this subspace, while momentum is maintained in low-rank coordinates to preserve the memory advantages of ZO training. On Qwen2.5-1.5B with Anthropic HH-RLHF, CA-ZODPO outperforms common ZO baselines and remains competitive with first-order DPO in pairwise preference evaluation. It also reduces peak GPU memory by about 59\% at a sequence length of 2048 while retaining the sequence-length scaling behaviour of zeroth-order training.

## LLM-Enhanced Deep Reinforcement Learning with Human-in-the-Loop Calibration for Real-Time Optimal Power Flow

DevFeed: [LLM-Enhanced Deep Reinforcement Learning with Human-in-the-Loop Calibration for Real-Time Optimal Power Flow](<https://devfeed.tech/articles/llm-enhanced-deep-reinforcement-learning-with-human-in-the-loop-calibration-for-real-time-optimal-power-flow-132080.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15003>)

Author: Jiayang Tu, Bozhen Jiang, Bojun Du, Yunxiao Ling, Ershun Du, Qin Wang

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-sy](<https://devfeed.tech/tags/cs-sy.md>), [deep-reinforcement-learning](<https://devfeed.tech/tags/deep-reinforcement-learning.md>), [eess-sy](<https://devfeed.tech/tags/eess-sy.md>), [human-in-the-loop](<https://devfeed.tech/tags/human-in-the-loop.md>), [llm](<https://devfeed.tech/tags/llm.md>), [power](<https://devfeed.tech/tags/power.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>)

### AI overview

The paper proposes combining language feedback and expert review to improve deep reinforcement learning for real-time optimal power flow (RT-OPF). During offline training, sparse evaluative feedback from a large language model augments the physics-based reward to represent operational preferences and safety requirements. A selective human-in-the-loop mechanism inspects and corrects questionable model judgments before they continue influencing learning. In case studies on IEEE 30-bus and IEEE 118-bus systems, language feedback improved policy safety compared with vanilla TD3; adding human calibration yielded the strongest overall performance and more stable training. The supplied abstract does not provide further implementation details or quantitative results.

### Source excerpt

arXiv:2611.15003v1 Announce Type: cross Abstract: Real-time optimal power flow (RT-OPF) requires fast and reliable decision-making under time-varying operating conditions. However, many practical operating preferences and safety related requirements are difficult to encode using conventional numerical rewards alone, which limits the effectiveness and controllability of deep reinforcement learning (DRL) for RT-OPF. To address this issue, this paper proposes an LLM-enhanced DRL framework with human-in-the-loop (HITL) for RT-OPF. The proposed method augments the physics-based reward with sparse language evaluative feedback during offline training, so that semantic operational requirements can be incorporated into policy learning. To improve reliability in RT-OPF, a selective HITL calibration mechanism is further introduced to inspect and correct questionable LLM judgments before they continuously shape the learning process. Case studies on the IEEE 30-bus and IEEE 118-bus systems show that LLM feedback improves policy safety over vanilla TD3, while further HITL calibration achieves the best overall performance and more stable training behavior. These results demonstrate the effectiveness of combining language guidance and expert calibration for RT-OPF.

## AVC-Reward: Event-Centric Audio-Video Consistency Evaluation

DevFeed: [AVC-Reward: Event-Centric Audio-Video Consistency Evaluation](<https://devfeed.tech/articles/avc-reward-event-centric-audio-video-consistency-evaluation-131993.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.22321>)

Author: Yuruo Wu, Kaihang Pan, Kai Shen, Jiacheng Li, Shuyuan Tu, Li Tuoyu, Juncheng Li, Siliang Tang

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [Generative AI](<https://devfeed.tech/topics/generative-ai.md>), [human feedback](<https://devfeed.tech/topics/human-feedback.md>), [Dataset development](<https://devfeed.tech/topics/dataset-development.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [audio](<https://devfeed.tech/tags/audio.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [event](<https://devfeed.tech/tags/event.md>), [reward](<https://devfeed.tech/tags/reward.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

AVC-Reward evaluates audio-video consistency by separating scenes into audio and visual events, assessing each event and cross-modal relation, then combining judgments across three levels: global narrative harmony, event timing, and fine-grained attribute alignment. Its AVC-Benchmark uses positive and negative samples from multiple types of generation models to assess agreement with human judgments across generation paradigms and relation complexity. The authors report that AVC-Reward aligns with human judgments better than other automatic methods, gives rationales that localize failures, and improves consistency when used as an online reinforcement-learning signal for a joint generation model.

### Source excerpt

arXiv:2610.22321v2 Announce Type: replace-cross Abstract: As joint audio-video generation models become more capable, they can generate audio-visual scenes exhibiting three forms of complexity: multi-event composition, multi-layer audio mixture, and asymmetric cross-modal relations. However, current evaluations largely reuse metrics from earlier simple and unidirectional tasks. They typically evaluate an audio-video clip as a whole and rely on simple predefined relations, making them unsuitable for these increasingly complex scenes. We propose an event-centric evaluation framework that disentangles audio and visual events at the event level, analyzes each event independently, cross-validates relations across modalities, and aggregates the judgments into a coarse-to-fine, three-level consistency metric. The metric assesses global narrative harmony, event-level temporal synchronization, and fine-grained attribute alignment, providing a general formulation for audio-video consistency evaluation across diverse generation settings and cross-modal relations. We instantiate the metric as AVC-Reward, a trainable audio-video reward model. To evaluate the agreement between automatic judgments and human perception across generation paradigms and relation complexities, we construct AVC-Benchmark, which is primarily built from diverse positive and negative samples generated by various types of audio-video generation models. AVC-Reward achieves state-of-the-art alignment with human judgment among automatic methods, provides interpretable rationales that localize failures, and improves the audio-video consistency of a joint generation model when used as an online reinforcement-learning signal. Project page: https://tsuki0512.github.io/AVC-Reward.github.io/

## Theoretical Limits of Language Model Alignment

DevFeed: [Theoretical Limits of Language Model Alignment](<https://devfeed.tech/articles/theoretical-limits-of-language-model-alignment-131227.md>)

Original publisher: [Read original article](<https://machinelearning.apple.com/research/theoretical-limits-alignment>)

Published: 2026-11-12T00:00:00Z

Content type: article

Language: en

Sources: [Apple Machine Learning Research](<https://devfeed.tech/sources/apple-machine-learning-research.md>)

Topics: [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [dpo](<https://devfeed.tech/topics/dpo.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [best-of](<https://devfeed.tech/tags/best-of.md>), [budget](<https://devfeed.tech/tags/budget.md>), [covariance](<https://devfeed.tech/tags/covariance.md>), [error](<https://devfeed.tech/tags/error.md>), [estimator](<https://devfeed.tech/tags/estimator.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [hacking](<https://devfeed.tech/tags/hacking.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kl-divergence](<https://devfeed.tech/tags/kl-divergence.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [limits](<https://devfeed.tech/tags/limits.md>), [model-alignment](<https://devfeed.tech/tags/model-alignment.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>), [theoretical](<https://devfeed.tech/tags/theoretical.md>)

### AI overview

The excerpt derives an information-theoretic upper limit on expected reward gain for language-model alignment under a fixed KL-divergence budget. Its closed-form bound depends on Jeffreys divergence, and a covariance reformulation offers an estimator using base-model samples. In the proxy-reward setting, reward hacking increases with reward error and as the KL penalty decreases; the analysis shows that ensembling rewards mitigates this gap. Experiments on safety and summarization report that best-of-N approaches the theoretical limit, while PPO and GRPO remain substantially below it. The excerpt argues that reaching optimal alignment without high inference costs requires algorithmic improvements.

### Source excerpt

Language model (LM) alignment improves model outputs to reflect human preferences while preserving the capabilities of the base model. The most common alignment approaches are (i) reinforcement learning, which maximizes the expected reward under a KL-divergence constraint, and (ii) best-of-NNN alignment, which selects the highest-reward output among NNN independent samples. Despite their widespread use, the fundamental limits of reward improvement under a KL budget remain poorly understood. We characterize the information-theoretic limits of KL-regularized alignment by deriving the maximum...

## Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL

DevFeed: [Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL](<https://devfeed.tech/articles/diff-instruct-with-diffused-reward-towards-principled-one-step-generator-rl-128113.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.24001>)

Author: Junyi Wu, Weijian Luo, Haoyang Zheng, Ruizhe Zhang, Guang Lin

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [stable-diffusion](<https://devfeed.tech/topics/stable-diffusion.md>), [diffusion-transformers](<https://devfeed.tech/topics/diffusion-transformers.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [denoising](<https://devfeed.tech/tags/denoising.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [drp](<https://devfeed.tech/tags/drp.md>), [drs](<https://devfeed.tech/tags/drs.md>), [image](<https://devfeed.tech/tags/image.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>), [rlhf](<https://devfeed.tech/tags/rlhf.md>), [text-to-image](<https://devfeed.tech/tags/text-to-image.md>)

### AI overview

DIDR addresses a mismatch in prior one-step generator RL, where terminal image-reward optimization combined with noisy-space distribution matching can exploit stochasticity and reduce image fidelity. The proposed data-free, trajectory-level framework uses Integral KL minimization to propagate the reward-tilted clean-image distribution across diffusion noise levels. Its objective has the same minimizer as clean-image RLHF and induces a Diffused Reward Score correction to the reference score. For practical estimation, Diffused Reward Proxy approximates that correction using differentiable short-step denoising. The authors report Pareto dominance over existing one-step SDXL baselines and, on the 6B Z-Image backbone, higher preference alignment than its 50-step teacher while generating in one step.

### Source excerpt

arXiv:2605.24001v3 Announce Type: replace-cross Abstract: Recent advances in one-step text-to-image generation have enabled real-time synthesis with remarkable efficiency and quality. Previous reinforcement learning methods for one-step generators combine image-space reward optimization with diffusion noisy-space distribution matching. This paradigm brings challenges due to a mismatch between terminal reward optimization and the underlying generative dynamics. As a result, optimization tends to exploit stochastic degrees of freedom, often improving reward at the expense of image fidelity. To address this issue, we propose Diff-Instruct with Diffused Reward (DIDR), a data-free trajectory-level alignment framework derived from Integral KL minimization. DIDR propagates the RLHF-optimal reward-tilted clean-image distribution across all noise levels along the diffusion trajectory. We show that this objective admits the same minimizer as clean-image RLHF, while naturally inducing the Diffused Reward Score (DRS), which acts as a reward-driven correction to the reference score function. To make this practical, we further introduce the Diffused Reward Proxy (DRP), an efficient estimator of DRS based on differentiable short-step denoising. Extensive experiments demonstrate that DIDR consistently Pareto-dominates existing one-step SDXL baselines. Moreover, when transferred to a 6B DiT backbone (Z-Image), DIDR surpasses its 50-step teacher in preference alignment while requiring only a single generation step. Code: https://github.com/Junyi-W/DIDR

## Leveraging Psychophysical Attentional Distribution for Gaze-Augmented Reward Modeling

DevFeed: [Leveraging Psychophysical Attentional Distribution for Gaze-Augmented Reward Modeling](<https://devfeed.tech/articles/leveraging-psychophysical-attentional-distribution-for-gaze-augmented-reward-modeling-128316.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13477>)

Author: Chaeho Lee, Junyup Kim, Wansoo Kim, Young Min Jung, Sang Ho Lee

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [rlhf](<https://devfeed.tech/topics/rlhf.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cognitive-psychology](<https://devfeed.tech/tags/cognitive-psychology.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-hc](<https://devfeed.tech/tags/cs-hc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [eye](<https://devfeed.tech/tags/eye.md>), [human-feedback](<https://devfeed.tech/tags/human-feedback.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [learning](<https://devfeed.tech/tags/learning.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>)

### AI overview

The study reports that a reward model trained with a spatially graded attention distribution centered on the reader's gaze predicted human preferences more accurately than a baseline. The approach extends fixation-centered eye-tracking supervision by drawing on cognitive psychology findings about letter recognition. Some experimental results were qualitatively consistent with established findings on reading attention, which the authors say may reflect cognitively meaningful patterns; the excerpt provides no quantitative results or further methodological details.

### Source excerpt

arXiv:2611.13477v1 Announce Type: cross Abstract: Integrating human feedback into language models has gained increasing attention as a means to align model outputs with human preferences, with Reinforcement Learning from Human Feedback (RLHF) serving as a prominent framework for preference-based alignment. Recent studies have incorporated eye-tracking (ET) data as an additional supervisory signal for reward model training, grounding reward learning in human reading behavior. Building on this approach, we extend prior fixation-centric models by leveraging established findings from cognitive psychology on human letter recognition. Specifically, we model visual attention as a spatially graded distribution centered on the current gaze location and incorporate this representation into reward model training. Our results show that the reward model trained with the proposed gaze distribution achieves higher preference prediction accuracy than a baseline model. Moreover, some results from our experiments are qualitatively consistent with established findings on human attention during reading, suggesting that they potentially capture cognitively meaningful aspects of human attention allocation.

## Enforcing Axioms for AI Alignment under Loss-Based Rules

DevFeed: [Enforcing Axioms for AI Alignment under Loss-Based Rules](<https://devfeed.tech/articles/enforcing-axioms-for-ai-alignment-under-loss-based-rules-128300.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13369>)

Author: Alexandros Hollender, Sonja Kraiczy

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-alignment](<https://devfeed.tech/tags/ai-alignment.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [constitutional-ai](<https://devfeed.tech/tags/constitutional-ai.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-gt](<https://devfeed.tech/tags/cs-gt.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [design](<https://devfeed.tech/tags/design.md>), [human-feedback](<https://devfeed.tech/tags/human-feedback.md>), [linear](<https://devfeed.tech/tags/linear.md>), [pareto-optimality](<https://devfeed.tech/tags/pareto-optimality.md>), [polynomials](<https://devfeed.tech/tags/polynomials.md>), [principles](<https://devfeed.tech/tags/principles.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [relu](<https://devfeed.tech/tags/relu.md>), [reward](<https://devfeed.tech/tags/reward.md>), [rlhf](<https://devfeed.tech/tags/rlhf.md>)

### AI overview

The paper finds that making the reward model richer does not by itself ensure Pareto optimality (PO) in loss-based alignment: PO can fail for fixed-degree polynomials and one-hidden-layer ReLU MLPs of any fixed width d >= 2 under suitable finite parameter bounds. It models principles as voters comparing responses along linear directions in a fixed embedding space. Under idealized assumptions of uniform comparison directions and equal comparison lengths, the authors recover PO for linear rewards trained by loss minimization. They suggest comparison-dataset design as a way to pursue PO while retaining loss-based training; the positive result depends on those assumptions.

### Source excerpt

arXiv:2611.13369v1 Announce Type: cross Abstract: Many alignment methods, most notably reinforcement learning from human feedback (RLHF), train reward models by minimizing a loss on pairwise comparisons of model responses. Inspired by principle-guided methods such as Constitutional AI, we treat each principle as a voter that compares responses according to that principle, for example, preferring the more helpful or less toxic response. We represent these voters as linear directions in a fixed embedding space, a simplifying assumption motivated by the Linear Representation Hypothesis. In this linear social choice model, Ge et al. (2024) showed that an optimal linear reward model can violate Pareto optimality (PO): From the principles-as-voters lens, this means a response A can be less helpful and more toxic than B, yet still receive a higher reward. We ask whether richer reward models or changes to comparison data can recover PO within loss-based training. We show that PO can still fail for polynomials of any fixed degree and one-hidden-layer ReLU multilayer perceptrons (MLPs) of any fixed width $ d\ge 2 $ with suitable finite parameter bounds. We then recover PO for loss-based training of linear rewards under suitable assumptions in an idealized setting with uniform comparison directions and equal comparison lengths. These results motivate dataset design as an approach to recovering PO in constitutional-style alignment while retaining loss-based training.

## AERO: Adaptive Ensemble-Disagreement Routing for Oracle Feedback in Sample-Efficient Online RLHF

DevFeed: [AERO: Adaptive Ensemble-Disagreement Routing for Oracle Feedback in Sample-Efficient Online RLHF](<https://devfeed.tech/articles/aero-adaptive-ensemble-disagreement-routing-for-oracle-feedback-in-sample-efficient-online-rlhf-125247.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10750>)

Author: Shrey Singhal, Nannan Huang, Haytham M. Fayek

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [rlhf](<https://devfeed.tech/topics/rlhf.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [online](<https://devfeed.tech/tags/online.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [rlhf](<https://devfeed.tech/tags/rlhf.md>), [routing](<https://devfeed.tech/tags/routing.md>), [sample](<https://devfeed.tech/tags/sample.md>)

### AI overview

AERO routes preference queries according to uncertainty, aiming to spend human-oracle labels where they are most informative. In its fully online RLHF setup, an ensemble Epistemic Reward Model (ERM) serves both as a reward posterior for exploration and as a source of synthetic preference labels. Ensemble vote entropy measures committee disagreement; an adaptive threshold uses that signal to choose between oracle feedback and ERM-generated labels. The abstract reports competitive alignment performance across model families and scales, with fewer oracle queries than online DPO and strong active-exploration baselines, but gives no numerical results or experimental details.

### Source excerpt

arXiv:2611.10750v1 Announce Type: new Abstract: Aligning Large Language Models (LLMs) with human preferences via Reinforcement Learning from Human Feedback (RLHF) has driven substantial improvements across a wide range of tasks, but the cost of acquiring preference labels remains the dominant bottleneck in the alignment pipeline. While many recent alignment methods improve sample efficiency through active exploration over candidate responses, they typically query the oracle uniformly across contexts, even for contexts where the learned reward estimates are already reliable enough to provide synthetic labels. We propose Adaptive Ensemble-disagreement Routing for Oracle feedback (AERO), a fully online RLHF algorithm built on a Dyna-style model-based RL architecture, in which an ensemble-based Epistemic Reward Model (ERM) serves both as a reward posterior for exploration and as a learned preference-feedback model for synthetic labeling. AERO uses ensemble vote entropy, a Query-by-Committee disagreement measure, combined with an adaptive thresholding mechanism to decide which contexts require oracle feedback and which can instead receive ERM-based synthetic preference labels, concentrating expensive labels where they are most informative. Empirical results across different model families and scales show that AERO achieves highly competitive alignment performance while reducing the required oracle-query budget compared to online DPO and strong active-exploration baselines.

## PORT: Preference Optimization via Robust Token-Level Reweighting

DevFeed: [PORT: Preference Optimization via Robust Token-Level Reweighting](<https://devfeed.tech/articles/port-preference-optimization-via-robust-token-level-reweighting-126308.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.12763>)

Author: Ding Zhu, Xiukun Wei, Tian Xie, Zhihui Zhu, Xueru Zhang, Mohammad Mahdi Khalili

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [rlhf](<https://devfeed.tech/topics/rlhf.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [Language models](<https://devfeed.tech/topics/language-models.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [annotation](<https://devfeed.tech/tags/annotation.md>), [annotations](<https://devfeed.tech/tags/annotations.md>), [approach](<https://devfeed.tech/tags/approach.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [bias](<https://devfeed.tech/tags/bias.md>), [but](<https://devfeed.tech/tags/but.md>), [can](<https://devfeed.tech/tags/can.md>), [central](<https://devfeed.tech/tags/central.md>), [code](<https://devfeed.tech/tags/code.md>), [contain](<https://devfeed.tech/tags/contain.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [data](<https://devfeed.tech/tags/data.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [effectiveness](<https://devfeed.tech/tags/effectiveness.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [port](<https://devfeed.tech/tags/port.md>), [robust](<https://devfeed.tech/tags/robust.md>), [token](<https://devfeed.tech/tags/token.md>)

### AI overview

PORT addresses noisy preference annotations by reweighting tokens instead of treating each response as wholly correct or incorrect. It uses the empirical CDF of token log-probabilities to selectively suppress potentially corrupted tokens, serving as a forward-pass-only proxy for gradient-norm penalization without extra backward-pass computation. The authors report theoretical analysis that PORT reduces gradient bias from noisy preference labels by minimizing an upper bound, and experiments across diverse noise settings show improved robustness over sequence-level baselines.

### Source excerpt

arXiv:2611.12763v1 Announce Type: cross Abstract: Preference optimization has become a central approach for aligning large language models with human values, but its effectiveness depends heavily on the quality of preference annotations. In practice, preference data is often noisy due to annotation errors and ambiguity. Existing robust preference optimization methods primarily operate at the sequence level, implicitly treating entire responses as uniformly correct or incorrect. However, rejected responses may still contain informative reasoning steps, while preferred responses can include subtle errors. In this work, we propose \textbf{P}reference \textbf{O}ptimization via \textbf{R}obust \textbf{T}oken-level reweighting \textbf{(PORT)}, a fine-grained framework for robust alignment under noisy preferences. PORT performs token-level reweighting using the empirical cumulative distribution function (CDF) of token log-probabilities, yielding an efficient forward-pass-only proxy to gradient-norm penalization that selectively suppresses corrupted tokens without additional backward-pass computation. We provide theoretical analysis showing that PORT reduces the gradient bias from noisy preference labels by minimizing an upper bound on the bias term. Extensive experiments across diverse noise settings demonstrate that PORT consistently improves robustness and outperforms existing sequence-level baselines. The code is available at https://github.com/osu-srml/PORT

## Causal Signal Credit Assignment for Verilog Code Generation via Counterfactual Hardware Verification

DevFeed: [Causal Signal Credit Assignment for Verilog Code Generation via Counterfactual Hardware Verification](<https://devfeed.tech/articles/causal-signal-credit-assignment-for-verilog-code-generation-via-counterfactual-hardware-verification-126192.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11848>)

Author: Yali Du, Ming Li

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Verilog](<https://devfeed.tech/topics/verilog.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [Automated reasoning](<https://devfeed.tech/topics/automated-reasoning.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [credit-assignment](<https://devfeed.tech/tags/credit-assignment.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [signal](<https://devfeed.tech/tags/signal.md>), [verification](<https://devfeed.tech/tags/verification.md>), [verilog](<https://devfeed.tech/tags/verilog.md>)

### AI overview

Causal Signal Credit Assignment (CSCA) addresses the coarse pass/fail feedback of module-level simulation by assigning continuous credit to RTL statements. It verifies sampled candidates at signal level, identifies output-driving assignments, and applies five interventions while holding the rest of the module and testbench fixed. Changes in signal-level errors and behavioral agreement determine statement credits, which are mapped to response-token weights for strict weighted direct preference optimization. The method is designed to reinforce locally supported statements without uniformly rewarding an otherwise incorrect response. Built with Icarus Verilog and a CodeV-derived corpus, it was evaluated on code-completion and specification-to-RTL tasks in VerilogEval 2.0 and RTLLM v2.0. With Qwen2.5-Coder-7B-Instruct, CSCA led the evaluated 7B-scale models and was reported as comparable to the larger GPT-5.4-mini on these benchmarks.

### Source excerpt

arXiv:2611.11848v1 Announce Type: new Abstract: Large language models have improved Verilog generation, yet functional correctness remains difficult to optimize because simulation commonly provides a coarse module-level pass/fail outcome. Such feedback discards useful evidence from partially correct RTL modules and cannot determine whether a source statement is functionally responsible for an observed output behavior. We present a new method named Causal Signal Credit Assignment (CSCA), which is a counterfactual preference-optimization framework for Verilog generation. CSCA first performs continuous signal-level behavioral verification on sampled candidate modules. It then locates output-driving assignment spans and applies five interventions while holding the remaining module and testbench fixed. The resulting changes in signal-level errors are combined with behavioral agreement to form a continuous statement credit, which is mapped to response-token weights for strict weighted direct preference optimization. This enables CSCA to learn from locally supported statements without uniformly reinforcing an otherwise incorrect response. Built on Icarus Verilog and a CodeV-derived corpus, CSCA is evaluated on VerilogEval~2.0 and RTLLM~v2.0, including both code completion and specification-to-RTL tasks. With a Qwen2.5-Coder-7B-Instruct backbone, CSCA leads the evaluated 7B-scale models and remains comparable to the much larger GPT-5.4-mini on the reported Verilog-generation benchmarks.

## Provable Zeroth-Order Safe Exploration for Reinforcement Learning from Human Feedback

DevFeed: [Provable Zeroth-Order Safe Exploration for Reinforcement Learning from Human Feedback](<https://devfeed.tech/articles/provable-zeroth-order-safe-exploration-for-reinforcement-learning-from-human-feedback-120630.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07561>)

Author: Weiqin Chen, Xinmeng Huang, Dongsheng Ding, Santiago Paternain

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [rlhf](<https://devfeed.tech/topics/rlhf.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [human feedback](<https://devfeed.tech/topics/human-feedback.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [constraint](<https://devfeed.tech/tags/constraint.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [human-feedback](<https://devfeed.tech/tags/human-feedback.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [learning](<https://devfeed.tech/tags/learning.md>), [model](<https://devfeed.tech/tags/model.md>), [model-alignment](<https://devfeed.tech/tags/model-alignment.md>), [policies](<https://devfeed.tech/tags/policies.md>), [policy](<https://devfeed.tech/tags/policy.md>), [queries](<https://devfeed.tech/tags/queries.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>), [rlhf](<https://devfeed.tech/tags/rlhf.md>), [safety](<https://devfeed.tech/tags/safety.md>)

### AI overview

ZO-LBPG targets a stronger Safe-RLHF guarantee than eventual feasibility: every policy used during training, including auxiliary exploration policies, remains feasible. It is reward-model-free, optimizing a log-barrier objective with zeroth-order estimates from human preferences. Conservative constraint margins, coupled rollouts, and an adaptive step size are designed to keep noisy-feedback updates from crossing constraint boundaries. The authors prove that, with high probability, all iterates and exploration policies stay strictly feasible and the method globally converges to an approximately optimal constrained policy. The stated complexity is Õ(ε⁻⁶ + d³ε⁻⁵) trajectories and Õ(ε⁻⁸ + d⁵ε⁻⁷) human queries. Experiments on a synthetic CMDP and a language-model alignment task are reported as corroborating the training-time safety guarantees.

### Source excerpt

arXiv:2611.07561v1 Announce Type: new Abstract: Safe reinforcement learning from human feedback (Safe-RLHF) seeks to improve helpfulness while satisfying harmlessness constraints. Existing primal-dual approaches can converge to feasible policies but do not prevent constraint violations during training. We study a stronger requirement, safe exploration for RLHF, under which every policy executed during training, including auxiliary exploration policies, remains feasible. We propose Zeroth-Order Log-Barrier Policy Gradient (ZO-LBPG), a reward-model-free method that optimizes a log-barrier objective using zeroth-order estimates derived directly from human preferences. To ensure safety under noisy feedback, we combine conservative constraint margin estimates with coupled rollouts and an adaptive step size that prevents updates from crossing constraint boundaries. We prove that, with high probability, all iterates and exploration policies remain strictly feasible, and that ZO-LBPG globally converges to an approximately optimal constrained policy. We establish a sample complexity of $\widetilde{\mathcal O}(\varepsilon^{-6}+d^3\varepsilon^{-5})$ trajectories and $\widetilde{\mathcal O}(\varepsilon^{-8}+d^5\varepsilon^{-7})$ human queries. Experiments on a synthetic CMDP and a language-model alignment task corroborate the training-time safety guarantees.

## Unified Hallucination Fuzzing for Multimodal Large Language Models

DevFeed: [Unified Hallucination Fuzzing for Multimodal Large Language Models](<https://devfeed.tech/articles/unified-hallucination-fuzzing-for-multimodal-large-language-models-120071.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.07525>)

Author: Pengfei Zhou, Jiajun Song, Zhiwei Tang, Yixing Ma, Donghui Si, Yuhang Xu, Huiqi Song, Yiyuan Miao, Yichen Qian, Weihua Chen, Wangbo Zhao, Bohan Zhuang, Jiasheng Tang, Yang You

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Fuzzing/Fuzz testing](<https://devfeed.tech/topics/fuzzing.md>), [Hallucination detection](<https://devfeed.tech/topics/hallucination-detection.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [fuzzing](<https://devfeed.tech/tags/fuzzing.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [hallucinations](<https://devfeed.tech/tags/hallucinations.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [metric](<https://devfeed.tech/tags/metric.md>), [model](<https://devfeed.tech/tags/model.md>), [model-robustness](<https://devfeed.tech/tags/model-robustness.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [mutation](<https://devfeed.tech/tags/mutation.md>)

### AI overview

The paper presents UniHall, a fine-grained MLLM hallucination dataset organized around Object, Instruction, and Knowledge, and SAMF, a self-adaptive fuzzing framework that uses evolutionary mutations and multimodal-oracle metrics to probe changing inputs. The authors report that leading MLLMs perform significantly worse under fuzzing than in conventional evaluations, suggesting a gap between reasoning ability and factual grounding. They also report that reinforcement-learning alignment can worsen sycophancy in instruction-following tasks, revealing a helpfulness-hallucination trade-off. The evidence here is the abstract, so details of methods and experiments are not available.

### Source excerpt

arXiv:2608.07525v2 Announce Type: replace-cross Abstract: Hallucination remains a persistent challenge for Multimodal Large Language Models (MLLMs), severely limiting their reliability in high-stakes applications. Existing evaluations, predominantly based on static benchmarks, suffer from narrow taxonomical coverage and rapid performance saturation, failing to reflect model robustness in evolving real-world scenarios. To bridge this gap, we present a systematic evaluation framework integrating a comprehensive benchmark with self-evolving stress testing. First, we introduce UniHall, a fine-grained dataset grounded in a unified taxonomy spanning Object, Instruction, and Knowledge dimensions. Second, to address benchmark saturation, we propose Self-Adaptive Multimodal Fuzzing (SAMF), a self-adaptive framework that employs evolutionary mutation strategies to explore the boundaries of model hallucinations. Crucially, to ensure reliable assessment of dynamic inputs, SAMF incorporates a structured metric suite driven by an ensemble of multi-modal oracles. Our extensive experiments reveal that state-of-the-art MLLMs exhibit significant performance degradation under fuzzing compared to conventional settings, exposing a dissociation between reasoning capabilities and factual grounding. Furthermore, we identify a helpfulness-hallucination trade-off, where reinforcement learning alignment inadvertently exacerbates sycophancy in instruction-following tasks. The framework, code and benchmark are available at https://github.com/LanceZPF/EvalHall.

## Oracle-Robust Online Alignment for Large Language Models

DevFeed: [Oracle-Robust Online Alignment for Large Language Models](<https://devfeed.tech/articles/oracle-robust-online-alignment-for-large-language-models-120388.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.20457>)

Author: Zimeng Li, Mudit Gaur, Vaneet Aggarwal

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [collection](<https://devfeed.tech/tags/collection.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [online](<https://devfeed.tech/tags/online.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [policies](<https://devfeed.tech/tags/policies.md>), [robust](<https://devfeed.tech/tags/robust.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>)

### AI overview

The paper addresses online LLM alignment when observed preference feedback differs from an unknown ground-truth oracle. It formulates alignment as worst-case optimization over a pointwise oracle uncertainty set. For log-linear policies, the robust objective decomposes exactly into the original loss plus an explicit sensitivity penalty. In the finite-response log-linear setting, the authors develop projected stochastic composite updates for the resulting weakly convex objective and prove a \widetilde{O}(\varepsilon^{-2}) oracle complexity bound for reaching approximate stationarity.

### Source excerpt

arXiv:2602.20457v2 Announce Type: replace Abstract: We study online alignment of large language models under misspecified preference feedback, where the observed preference oracle deviates from an ideal but unknown ground-truth oracle. The online LLM alignment problem is a bi-level reinforcement problem due to the coupling between data collection and policy updates. Recently, the problem has been reduced to a tractable single-level objective in the SAIL (Self-Improving Efficient Online Alignment) framework. In this paper, we introduce a pointwise oracle uncertainty set in this problem and formulate an oracle-robust online alignment objective as a worst-case optimization problem. For log-linear policies, we show that this robust objective admits an exact closed-form decomposition into the original loss function plus an explicit sensitivity penalty. In the finite-response log-linear setting, we develop projected stochastic composite updates for the resulting weakly convex objective and prove $\widetilde{O}(\varepsilon^{-2})$ oracle complexity for reaching approximate stationarity.

## PEARL: Prediction Error-Adaptive Regularized Learning for Selective Forgetting under Partial Preference Drift

DevFeed: [PEARL: Prediction Error-Adaptive Regularized Learning for Selective Forgetting under Partial Preference Drift](<https://devfeed.tech/articles/pearl-prediction-error-adaptive-regularized-learning-for-selective-forgetting-under-partial-preference-drift-118390.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05842>)

Author: Rashmi Gangadharaiah, Balakrishnan Narayanaswamy

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [personalization](<https://devfeed.tech/topics/personalization.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [Inference](<https://devfeed.tech/topics/inference.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [bandit](<https://devfeed.tech/tags/bandit.md>), [bandits](<https://devfeed.tech/tags/bandits.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [feature](<https://devfeed.tech/tags/feature.md>), [learning](<https://devfeed.tech/tags/learning.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llms](<https://devfeed.tech/tags/llms.md>), [mlp](<https://devfeed.tech/tags/mlp.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [recommendation](<https://devfeed.tech/tags/recommendation.md>), [rlhf](<https://devfeed.tech/tags/rlhf.md>), [selection](<https://devfeed.tech/tags/selection.md>), [shift](<https://devfeed.tech/tags/shift.md>), [sliding-window](<https://devfeed.tech/tags/sliding-window.md>), [stable](<https://devfeed.tech/tags/stable.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>), [user](<https://devfeed.tech/tags/user.md>), [users](<https://devfeed.tech/tags/users.md>)

### AI overview

PEARL targets partial preference drift by reversing Gram-matrix confidence updates only along feature directions where predictions are surprisingly wrong, preserving confidence in other directions. The paper states that non-adaptive policies incur linear regret after a shift, no fixed exploration rate works well in both stationary and changing settings, and PEARL's cumulative regret scales with the number of shifted users rather than the full population. On the PRISM benchmark (1,500 users, 21 LLMs), it reports gains of 4.0 points over greedy and 2.5 over the best neural bandit baseline (p<0.001), parity with greedy when preferences stay stable, and 3-10 point gains over sliding-window and discounted methods. Its roughly 100K-parameter MLP runs in milliseconds on CPU, keeps LLM weights frozen, and works with API-based models.

### Source excerpt

arXiv:2611.05842v1 Announce Type: cross Abstract: User preferences shift as needs, context, and tastes evolve. This matters for personalized recommendation, LLM response generation, and RLHF-trained reward models, which can become miscalibrated as preferences drift. Existing non-stationary bandit methods forget uniformly across all dimensions, but real preference shifts are typically partial: some dimensions go stale while others remain valid. Neural contextual bandits accumulate confidence monotonically via the Gram matrix, with no mechanism to direct uncertainty where predictions have gone wrong. We propose PEARL (Prediction Error-Adaptive Regularized Learning), which reverses the Gram update along feature directions where predictions are surprisingly wrong, while leaving confidence intact elsewhere. We prove that (i) any non-adaptive policy suffers linear regret after a preference shift, (ii) no fixed exploration rate achieves low regret in both stationary and non-stationary regimes, and (iii) PEARL's cumulative regret scales with the number of shifted users N_s, not the total population N. On PRISM, a personalized LLM response selection benchmark (1,500 users, 21 LLMs), PEARL beats greedy by +4.0 points and the best neural bandit baseline by +2.5 points (p<0.001), while performing on par with greedy when preferences are stable. It beats sliding-window and discounted approaches by 3 to 10 points, showing that selective forgetting outperforms uniform forgetting. The selection policy is a ~100K-parameter MLP that runs in milliseconds on CPU, leaves LLM weights frozen, and works with any API-based model.

## TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots

DevFeed: [TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots](<https://devfeed.tech/articles/therapygym-evaluating-and-aligning-clinical-fidelity-and-safety-in-therapy-chatbots-118203.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2603.18008>)

Author: Fangrui Huang, Souhad Chbeir, Arpandeep Khatua, Sheng Wang, Sijun Tan, Kenan Ye, Lily Bailey, Merryn Daniel, Ryan Louie, Sanmi Koyejo, Ehsan Adeli

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [trust and safety](<https://devfeed.tech/topics/trust-and-safety.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [also](<https://devfeed.tech/tags/also.md>), [annotation](<https://devfeed.tech/tags/annotation.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [auditing](<https://devfeed.tech/tags/auditing.md>), [automated](<https://devfeed.tech/tags/automated.md>), [based](<https://devfeed.tech/tags/based.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bias](<https://devfeed.tech/tags/bias.md>), [capture](<https://devfeed.tech/tags/capture.md>), [chatbots](<https://devfeed.tech/tags/chatbots.md>), [covering](<https://devfeed.tech/tags/covering.md>), [critical](<https://devfeed.tech/tags/critical.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [development](<https://devfeed.tech/tags/development.md>), [dimensions](<https://devfeed.tech/tags/dimensions.md>), [drive](<https://devfeed.tech/tags/drive.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evidence-based](<https://devfeed.tech/tags/evidence-based.md>), [fail](<https://devfeed.tech/tags/fail.md>), [safety](<https://devfeed.tech/tags/safety.md>)

### AI overview

THERAPYGYM evaluates and trains therapy chatbots on clinical fidelity and safety, dimensions the abstract says generic dialogue and preference measures miss. Its automated Cognitive Therapy Rating Scale pipeline scores CBT adherence across multi-turn sessions, while a multi-label scheme covers therapy-specific risks such as failing to address harm or abuse. THERAPYJUDGEBENCH provides 116 dialogues with 1,270 expert ratings to audit and calibrate LLM judges against licensed clinicians. The framework also uses fidelity and safety rewards for reinforcement learning with configurable patient simulations. The abstract reports average CTRS rising from 0.10 to 0.60 by expert ratings, and from 0.16 to 0.59 by LLM judges; it does not provide further detail on safety outcomes.

### Source excerpt

arXiv:2603.18008v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used for mental-health support; yet prevailing evaluation methods--fluency metrics, preference tests, and generic dialogue benchmarks--fail to capture the clinically critical dimensions of psychotherapy. We introduce THERAPYGYM, a framework that evaluates and improves therapy chatbots along two clinical pillars: fidelity and safety. Fidelity is measured using the Cognitive Therapy Rating Scale (CTRS), implemented as an automated pipeline that scores adherence to CBT techniques over multi-turn sessions. Safety is assessed using a multi-label annotation scheme, covering therapy-specific risks (e.g., failing to address harm or abuse). To mitigate bias and unreliability in LLM-based judges, we further release THERAPYJUDGEBENCH, a validation set of 116 dialogues with 1,270 expert ratings for auditing and calibration against licensed clinicians. THERAPYGYM also serves as a training harness: CTRS and safety-based rewards drive RL with configurable patient simulations spanning diverse symptom profiles. Models trained in THERAPYGYM improve on expert ratings, with average CTRS rising from 0.10 to 0.60 (and 0.16 to 0.59 under LLM judges). Our work enables scalable development of therapy chatbots that are faithful to evidence-based practice and safer in high-stakes use.

## Vision Paper: Why Distributed LLM Post-Training and Decentralized Alignment Are Now Essential

DevFeed: [Vision Paper: Why Distributed LLM Post-Training and Decentralized Alignment Are Now Essential](<https://devfeed.tech/articles/vision-paper-why-distributed-llm-post-training-and-decentralized-alignment-are-now-essential-118389.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05841>)

Author: Zhanhong Jiang, Qisai Liu, Soumik Sarkar

Published: 2026-11-05T05:00:00Z

Content type: opinion

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [post-training](<https://devfeed.tech/topics/post-training.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Federated Learning](<https://devfeed.tech/topics/federated-learning.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [data-sovereignty](<https://devfeed.tech/tags/data-sovereignty.md>), [decentralized](<https://devfeed.tech/tags/decentralized.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [equity](<https://devfeed.tech/tags/equity.md>), [federated](<https://devfeed.tech/tags/federated.md>), [healthcare](<https://devfeed.tech/tags/healthcare.md>), [llm](<https://devfeed.tech/tags/llm.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [research](<https://devfeed.tech/tags/research.md>), [robustness](<https://devfeed.tech/tags/robustness.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [scale](<https://devfeed.tech/tags/scale.md>), [unique](<https://devfeed.tech/tags/unique.md>), [utility](<https://devfeed.tech/tags/utility.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

The paper argues that LLM post-training should support federated and decentralized learning as a core approach, since preference data in fields such as healthcare, law, finance, and multilingual communities may not be suitable for centralization. It grounds the case in privacy and data sovereignty, pluralism and representational equity, and robustness and scalability. It identifies reference drift, heterogeneous preferences, and the privacy-versus-utility trade-off for contrastive losses as technical challenges, and proposes algorithmic directions and a research agenda. These details are described only at abstract level in the supplied text.

### Source excerpt

arXiv:2611.05841v1 Announce Type: new Abstract: This vision paper argues that distributed learning, including federated and decentralized, should become a first-class paradigm in LLM post-training, not merely a privacy add-on. Current methods like RLHF and DPO rest on an invisible assumption: preference data should be centralized. We contend this assumption is scientifically unjustified, practically unsustainable, and ethically problematic. As LLMs deploy into healthcare, law, finance, and multilingual communities, preference data originates from populations that cannot or will not centralize. We develop a three-pillar argument grounded in (1) privacy and data sovereignty, (2) pluralism and representational equity, and (3) robustness and scalability. We formalize key technical challenges unique to this intersection, reference drift, preference heterogeneity, and the DP-utility trade-off for contrastive losses, and propose concrete algorithmic directions, and sketch a research agenda. Our analysis shows distributed post-training is not only necessary but tractable, and ignoring it risks AI that is legally noncompliant, representationally narrow, and brittle at scale.

## Evolving Support Priorities in Empathetic Reinforcement Learning

DevFeed: [Evolving Support Priorities in Empathetic Reinforcement Learning](<https://devfeed.tech/articles/evolving-support-priorities-in-empathetic-reinforcement-learning-118265.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.34249>)

Author: Pengyu Huang, Zhiyuan Han, Wenwen Tong, Hewei Guo, Jiangnan Chen, Sirui Chen, Lewei Lu, Beier Zhu, Xun Yang

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [dimensions](<https://devfeed.tech/tags/dimensions.md>), [empathy](<https://devfeed.tech/tags/empathy.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gemini](<https://devfeed.tech/tags/gemini.md>), [generator](<https://devfeed.tech/tags/generator.md>), [llm](<https://devfeed.tech/tags/llm.md>), [mica](<https://devfeed.tech/tags/mica.md>), [needs](<https://devfeed.tech/tags/needs.md>), [online](<https://devfeed.tech/tags/online.md>), [performance](<https://devfeed.tech/tags/performance.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>), [rl](<https://devfeed.tech/tags/rl.md>), [specifications](<https://devfeed.tech/tags/specifications.md>), [state](<https://devfeed.tech/tags/state.md>), [supervised-fine-tuning](<https://devfeed.tech/tags/supervised-fine-tuning.md>), [supervision](<https://devfeed.tech/tags/supervision.md>), [support](<https://devfeed.tech/tags/support.md>)

### AI overview

CARE addresses the mismatch between changing support needs in dialogue and empathetic reinforcement learning methods that use fixed rewards. At each turn, it generates a rubric that adapts the weights and detailed criteria for cognitive, affective, and proactive empathy. The rubric generator is trained with turn-level rubric supervision and human preferences, using supervised fine-tuning followed by preference-based reinforcement learning, then used as an adaptive reward interface for online empathetic RL. Integrated with RLVER and MICA, CARE reportedly achieves state-of-the-art results on SentientBench, EQBench3, and EMPA under three independent LLM judges. On EMPA, its EPM-Idx exceeds the strongest baseline by at least 13 points across all three judges; under Gemini-2.5-Pro, the reported score rises from 28.11 to 83.54. The abstract also reports that rubric priorities vary with dialogue stage and user emotion.

### Source excerpt

arXiv:2609.34249v3 Announce Type: replace Abstract: We identify a fundamental mismatch in empathetic reinforcement learning: support priorities evolve with the dialogue state, yet existing methods typically optimize predefined reward specifications that remain fixed across turns. To model these evolving support priorities, we organize empathetic support along cognitive, affective, and proactive empathy, and propose Context-Adaptive Rubric Evolution (CARE). At each turn, CARE generates a context-adaptive rubric by adjusting both the weights of these three empathy dimensions and their fine-grained evaluation criteria. The rubric generator is trained with turn-level rubric supervision and human preference data through supervised fine-tuning followed by preference-based reinforcement learning, and then serves as an adaptive reward interface for online empathetic RL. Integrated with both RLVER and MICA, CARE achieves state-of-the-art performance across SentientBench, EQBench3, and EMPA under three independent LLM judges. Notably, on EMPA, CARE improves EPM-Idx over the strongest baseline by at least 13 points under all three judges, including an increase from 28.11 to 83.54 under Gemini-2.5-Pro. Further analyses show that learned rubric priorities systematically vary across dialogue stages and user emotions, demonstrating that CARE adapts what is rewarded as support needs evolve. Code is available at https://asa13opp.github.io/CARE-project-page.

## Objective-Conditioned Active Preference Learning for Multi-Objective Alignment

DevFeed: [Objective-Conditioned Active Preference Learning for Multi-Objective Alignment](<https://devfeed.tech/articles/objective-conditioned-active-preference-learning-for-multi-objective-alignment-118331.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05506>)

Author: Duc Anh Vu, Thong Nguyen, Hoang Pham, Ponhvoan Srey, Luu Anh Tuan

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [dpo](<https://devfeed.tech/topics/dpo.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [annotation](<https://devfeed.tech/tags/annotation.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [can](<https://devfeed.tech/tags/can.md>), [change](<https://devfeed.tech/tags/change.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [cost](<https://devfeed.tech/tags/cost.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [dpo](<https://devfeed.tech/tags/dpo.md>), [effectiveness](<https://devfeed.tech/tags/effectiveness.md>), [either](<https://devfeed.tech/tags/either.md>), [existing](<https://devfeed.tech/tags/existing.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [external](<https://devfeed.tech/tags/external.md>), [framework](<https://devfeed.tech/tags/framework.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [learning](<https://devfeed.tech/tags/learning.md>)

### AI overview

OGC-DPO reduces preference-labeling costs for multi-objective LLM alignment by treating each response-pair/objective combination as a query and selecting queries that expand gradient coverage for each objective. It estimates informativeness from the policy's implicit gradient, without an external reward model. In experiments across varying numbers of objectives, it outperformed existing active preference-learning methods at the same annotation cost. The authors also report that informative conflicts between preferences can provide a useful training signal, rather than being uniformly harmful.

### Source excerpt

arXiv:2611.05506v1 Announce Type: cross Abstract: Aligning large language models (LLMs) with several objectives, such as helpfulness, harmlessness, and honesty, typically relies on preference judgment for each objective, growing the annotation cost linearly with the number of objectives. Active preference learning can reduce this cost by selecting the comparisons whose labels are expected to change the policy most. Existing remedies either lack an objective-aware query selection mechanism for the multi-objective setting or regard conflicting preferences as harmful, therefore excluding them from the training data. We introduce OGC-DPO (Objective-conditioned Gradient Coverage DPO), an active learning framework that treats each (response pair, objective) combination as a separate query and measures its informativeness through the policy's own implicit gradient, without relying on an external reward model. OGC-DPO then selects the queries that most expand each objective's gradient coverage. Empirical experiments across varying numbers of objectives demonstrate the effectiveness of OGC-DPO over existing active preference learning methods under the same annotation cost. Further analysis demonstrates that informative conflicts, rather than being uniformly harmful, provide a useful policy training signal.

[Next page](<https://devfeed.tech/topics/rlhf.md?cursor=WyIyMDI2LTExLTA1VDA1OjAwOjAwKzAwOjAwIiwgIjMwMWZiZWQ2LTVjZGUtNDdlZC1iOTZjLWJkYjc3MDE2NTY5MCJd>)