# Reinforcement learning

Reinforcement learning is a machine learning approach in which agents learn to make decisions by interacting with an environment through trial and error and rewards.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Learning Which Evidence-Seeking Decisions Matter: Calibrated Counterfactual Replay for Language Agents

DevFeed: [Learning Which Evidence-Seeking Decisions Matter: Calibrated Counterfactual Replay for Language Agents](<https://devfeed.tech/articles/learning-which-evidence-seeking-decisions-matter-calibrated-counterfactual-replay-for-language-agents-144902.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.05263>)

Author: Renwei Meng, Chengyi Sun, Hongyi Zhang, Xiaoyi Hou

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [learning](<https://devfeed.tech/tags/learning.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>)

### AI overview

PCCC-RL estimates which evidence-seeking actions matter by restoring a saved tool state, comparing the chosen action with a feasible alternative, and letting the same frozen policy complete both branches; the difference in final score estimates the action's value. To reduce replay costs, it predicts credit, checks a random subset, and corrects prediction errors using recorded selection probabilities. Alternatives depend only on the prefix, preserving a usable gradient baseline. On HotpotQA and MuSiQue, joint exact match--requiring both a correct answer and complete supporting evidence--was 56.71% and 36.82%, respectively, 4.19 and 3.86 points above a baseline with the same replay access. Gains concentrate in complete evidence chains and accumulate over many updates; single-update effects remain uncertain, while citation coverage and distinctions such as "none" versus "not all" remain open.

### Source excerpt

arXiv:2606.05263v2 Announce Type: replace Abstract: Language agents answer questions by searching, reading, and checking claims. A reward on the final answer says little about which of those decisions earned it, since an agent can reach the right string after redundant searches or with half the evidence missing. We treat the question as an experiment. At a saved decision point, PCCC-RL, for prefix-conditioned counterfactual credit, restores the tool state, runs the action the agent chose alongside a feasible alternative, and lets the same frozen policy finish both branches. The difference in final score then samples what the decision was worth. Replaying everything is too expensive, so the method predicts each action's credit up front, checks a random subset, and uses the recorded selection probabilities to correct the errors of that prediction, spending the budget where a further check can still move the training signal. Alternatives are drawn from the prefix alone, which keeps the comparison a usable gradient baseline. On HotpotQA and MuSiQue, joint exact match, requiring a correct answer together with complete supporting evidence, reaches 56.71% and 36.82%, gains of 4.19 and 3.86 points over a baseline with the same replay access. The gain concentrates in complete evidence chains and accumulates over many updates. Single-update effects stay uncertain, and citation coverage and scope distinctions such as "none" versus "not all" remain open.

## When Safety Rationales Hallucinate: Diagnosing and Grounding Multimodal Safety Guards

DevFeed: [When Safety Rationales Hallucinate: Diagnosing and Grounding Multimodal Safety Guards](<https://devfeed.tech/articles/when-safety-rationales-hallucinate-diagnosing-and-grounding-multimodal-safety-guards-145588.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25793>)

Author: Yingshuo Wang, Weixiang Zhao, Guancheng Li, Jiahe Guo, Yu Zhang, Yanyan Zhao

Published: 2026-11-20T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [attribution](<https://devfeed.tech/tags/attribution.md>), [classification](<https://devfeed.tech/tags/classification.md>), [classification-accuracy](<https://devfeed.tech/tags/classification-accuracy.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fine](<https://devfeed.tech/tags/fine.md>), [focus](<https://devfeed.tech/tags/focus.md>), [generation](<https://devfeed.tech/tags/generation.md>), [grounded](<https://devfeed.tech/tags/grounded.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [guards](<https://devfeed.tech/tags/guards.md>), [learning](<https://devfeed.tech/tags/learning.md>), [model](<https://devfeed.tech/tags/model.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [safety](<https://devfeed.tech/tags/safety.md>), [scale](<https://devfeed.tech/tags/scale.md>)

### AI overview

The study finds that multimodal safety guards often produce unreliable rationales: they may state unsupported visual facts or assign incorrect safety meanings to grounded content. This persists across the examined methods, vision-language backbones, and datasets, and does not consistently improve with higher prediction accuracy or model scale. In Think-with-Image analysis, errors cluster around fine-grained attributes and relations; visual tools are often used infrequently or without enough spatial focus. The proposed Tool-Contrastive Grounding Reinforcement Learning (TCG-RL) rewards tool use only when it improves grounding relative to a tool-free reference. The abstract reports substantially fewer rationale hallucinations while maintaining competitive safety prediction performance, highlighting that prediction correctness alone does not establish rationale reliability.

### Source excerpt

arXiv:2611.25793v1 Announce Type: new Abstract: Multimodal safety guards increasingly generate rationales alongside safety predictions, yet whether these rationales are reliable remains largely unexplored. We systematically study *rationale hallucination* from two perspectives: visual rationale hallucination, where rationales contain unsupported visual facts, and safety attribution hallucination, where grounded content is assigned an incorrect safety interpretation. Across rationale-generation methods, multiple vision-language backbones, and safety datasets, we find that rationale hallucination is widespread and does not consistently diminish with higher classification accuracy or increased model scale. Further analysis of the Think-with-Image paradigm shows that hallucination is concentrated in fine-grained attributes and relations, while visual tools are often invoked infrequently or without sufficient spatial focus. Based on these findings, we propose *Tool-Contrastive Grounding Reinforcement Learning* (TCG-RL), which rewards tool use only when it improves rationale grounding over a tool-free reference. TCG-RL substantially reduces rationale hallucination while maintaining competitive safety prediction performance. By revealing the gap between prediction correctness and rationale reliability, our study motivates a shift in multimodal safety guarding from outcome-only evaluation toward safety decisions that are both accurate and verifiably grounded.

## Generalist Meets Specialists: An Evolving Multimodal Agent with Persistent Tool Orchestration for Traffic Scenario Understanding

DevFeed: [Generalist Meets Specialists: An Evolving Multimodal Agent with Persistent Tool Orchestration for Traffic Scenario Understanding](<https://devfeed.tech/articles/generalist-meets-specialists-an-evolving-multimodal-agent-with-persistent-tool-orchestration-for-traffic-scenario-understanding-145597.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25868>)

Author: Linjia Kang, Binhan Jiang, Juan Yang, Zhiqin Song, Xiangsen Wu, Qionglin Ren, Jiang Liu, Zhongjiang He, Hao Sun

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Multi-step AI processes](<https://devfeed.tech/topics/multi-step-ai-processes.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [adoption](<https://devfeed.tech/tags/adoption.md>), [agent](<https://devfeed.tech/tags/agent.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [collaboration](<https://devfeed.tech/tags/collaboration.md>), [completion](<https://devfeed.tech/tags/completion.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cost-savings](<https://devfeed.tech/tags/cost-savings.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [error](<https://devfeed.tech/tags/error.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [execution](<https://devfeed.tech/tags/execution.md>), [fast](<https://devfeed.tech/tags/fast.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [generalist](<https://devfeed.tech/tags/generalist.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [specialists](<https://devfeed.tech/tags/specialists.md>), [tool](<https://devfeed.tech/tags/tool.md>), [traffic](<https://devfeed.tech/tags/traffic.md>)

### AI overview

StarTV-Agent combines policy learning with reusable tool workflows to balance adaptive reasoning with lower-cost handling of recurring traffic-understanding tasks. During training, Role-Sensitive Process Optimization combines process and terminal advantages, while Weakness-Driven Demonstration Learning uses validated teacher examples aimed at the student's weak questions. At inference, Lightweight Adaptive Workflow Reuse selects a suitable workflow and binds task-specific arguments and resource references; tasks without a suitable workflow use a slower ReAct-based path. Label-free feedback from execution completion, workflow adoption, and cost savings guides reuse and memory updates. Across three benchmarks, the abstract reports up to 8.0% relative performance improvement and up to 52.1% lower token consumption.

### Source excerpt

arXiv:2611.25868v1 Announce Type: new Abstract: Traffic scenario understanding requires combining fine-grained visual evidence with spatial and temporal reasoning. Tool-augmented vision-language models (VLMs) can use specialized perception tools, but effective collaboration faces two challenges: terminal rewards provide limited guidance for intermediate decisions and error recovery, while repeated planning adds unnecessary cost across similar tasks. Echoing the fast-slow dichotomy of human cognition, we propose StarTV-Agent, which combines policy learning with experience evolution to support adaptive reasoning when needed and efficient reuse of prior experience when possible. During training, Role-Sensitive Process Optimization (RSPO) combines process advantages with terminal advantages, while Weakness-Driven Demonstration Learning (WDL) provides validated teacher demonstrations targeting the student's weak questions. During inference, Lightweight Adaptive Workflow Reuse (LAWR) enables a fast path by selecting suitable workflows and binding task-specific arguments and resource references, while a slow ReAct-based path handles tasks without suitable workflows. We further design label-free feedback based on execution completion, workflow adoption, and cost savings to guide workflow reuse and memory updates. Across three challenging benchmarks, StarTV-Agent achieves up to 8.0% relative performance improvement, while reducing token consumption by up to 52.1%.

## FlowMap-OPD: On-Policy Distillation for Few-Step Flow-Map Generators

DevFeed: [FlowMap-OPD: On-Policy Distillation for Few-Step Flow-Map Generators](<https://devfeed.tech/articles/flowmap-opd-on-policy-distillation-for-few-step-flow-map-generators-144946.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.37851>)

Author: Zhiqi Li, Cheng Wan, Bo Zhu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [flux](<https://devfeed.tech/topics/flux.md>)

Tags: [consistency](<https://devfeed.tech/tags/consistency.md>), [consistency-models](<https://devfeed.tech/tags/consistency-models.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [flow](<https://devfeed.tech/tags/flow.md>), [flux](<https://devfeed.tech/tags/flux.md>), [generators](<https://devfeed.tech/tags/generators.md>), [policy](<https://devfeed.tech/tags/policy.md>), [sd3-5](<https://devfeed.tech/tags/sd3-5.md>), [supervision](<https://devfeed.tech/tags/supervision.md>), [velocity](<https://devfeed.tech/tags/velocity.md>)

### AI overview

FlowMap-OPD trains few-step flow-map students using states collected by the student's native sampler, while separating that collection from teacher supervision. The study finds that supervising instantaneous velocity with a small consistency penalty transfers teacher gains best; directly supervising the deployed flow map performs worst when teacher and student use different pretrained initializations. The authors summarize the approach as "sample with the flow map, learn the velocity." It also supports teachers that define a velocity, including multi-step teachers, and students initialized from multi-step models. On FLUX and SD3.5, one few-step student consolidated three specialist teachers within 300 updates, matched or exceeded each on its own benchmark, and outperformed multi-reward reinforcement learning and LoRA merging. The supplied text is an abstract, so it does not provide full methodological or evaluation details.

### Source excerpt

arXiv:2609.37851v2 Announce Type: replace-cross Abstract: Few-step flow-map generators, such as MeanFlow and consistency models, sample in a few long-range jumps. On-policy distillation (OPD) trains a student on states drawn from its own sampler, but existing methods compare teacher and student through the local stochastic transitions of a multi-step sampler, which a flow-map student never deploys. We present FlowMap-OPD, the first systematic study of OPD for flow-map students. It separates state collection from supervision: the student collects states with its native few-step sampler, while the teacher supervises the flow map, velocities induced from it, or the instantaneous velocity linked to the map by a separately weighted consistency penalty. We find that instantaneous-velocity supervision with a small consistency weight, our chosen form, transfers the teacher's gains best, whereas supervising the deployed map, although the natural choice, transfers them worst when teacher and student start from different pretrained models. This gives a counter-intuitive rule: \emph{sample with the flow map, learn the velocity}. As a further benefit, this form applies to any teacher that defines a velocity, including multi-step ones, and to students initialized from multi-step models. On FLUX and SD3.5, a single few-step student consolidates three specialist teachers within 300 updates, matches or exceeds each on its own benchmark (e.g., OCR 0.883 versus 0.850 on FLUX), and outperforms both multi-reward reinforcement learning and LoRA merging.

## Grid Cells as Spectral Successor Features: A Reusable Basis for Continual Reinforcement Learning

DevFeed: [Grid Cells as Spectral Successor Features: A Reusable Basis for Continual Reinforcement Learning](<https://devfeed.tech/articles/grid-cells-as-spectral-successor-features-a-reusable-basis-for-continual-reinforcement-learning-145188.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26504>)

Author: \\'Elodie C\\^ot\\'e-Gauthier, Doina Precup, Isabeau Pr\\'emont-Schwarz

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [adapting](<https://devfeed.tech/tags/adapting.md>), [basis](<https://devfeed.tech/tags/basis.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [eigenvectors](<https://devfeed.tech/tags/eigenvectors.md>), [features](<https://devfeed.tech/tags/features.md>), [grid](<https://devfeed.tech/tags/grid.md>), [linear-regression](<https://devfeed.tech/tags/linear-regression.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [policy](<https://devfeed.tech/tags/policy.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>)

### AI overview

The paper builds a reusable basis for continual reinforcement learning from eigenvectors of a fixed reference policy's transition matrix. In this basis, each successor feature is the corresponding base feature scaled by 1/(1−γλᵢ), so returns use the same features as immediate rewards; task adaptation becomes linear regression on reward, without a successor-feature network or replay buffer. The agent chooses actions greedily using the reference policy's action values, described as one step of policy improvement that suffices for a single goal. In a two-goal gridworld using a uniform-random-walk reference policy, the approach maintained near-maximal return through goal changes, while SimpleSF and DQN lost most of their return at each change. These results are limited to the described setting.

### Source excerpt

arXiv:2611.26504v1 Announce Type: new Abstract: Continual reinforcement learning requires agents to adapt when the task changes while reusing what they know about the environment. Successor features separate dynamics from task-specific reward, but their effectiveness depends on the choice of base features. Taking the identification of entorhinal grid cells with eigenvectors of the successor representation as a design principle, we build this basis from eigenvectors of the transition matrix of a fixed reference policy. With such a basis, the successor features at a state-action pair are a linear combination of the base features at that same pair, which is not true in general, and the coefficients are known in closed form: each successor feature is its base feature multiplied by $\theta_i = 1/(1-\gamma\lambda_i)$, where $\lambda_i$ is its eigenvalue. The long-term return is thus predicted from the same features as the immediate reward, no successor-feature network or replay buffer is needed, and adapting to a new task reduces to a linear regression on the reward. The agent acts greedily on the action values of the reference policy, which is one step of policy improvement over it and suffices to reach a single goal. In a two-goal gridworld, with the uniform random walk as reference policy, whose eigenvectors are the low-frequency Laplacian eigenvectors, agents built this way maintain near-maximal return across every change of goal, whereas SimpleSF and DQN lose most of their return at each change.

## Reference-Free Humanoid Skateboarding via Termination-Shaped Reinforcement Learning

DevFeed: [Reference-Free Humanoid Skateboarding via Termination-Shaped Reinforcement Learning](<https://devfeed.tech/articles/reference-free-humanoid-skateboarding-via-termination-shaped-reinforcement-learning-145131.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26054>)

Author: Ganeshvar Soenessardien, Michael Przystupa, Kevin Sebastian Luck

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Humanoid Robots](<https://devfeed.tech/topics/humanoid-robots.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Robot Manipulation](<https://devfeed.tech/topics/robot-manipulation.md>)

Tags: [board](<https://devfeed.tech/tags/board.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [deep-reinforcement-learning](<https://devfeed.tech/tags/deep-reinforcement-learning.md>), [humanoid](<https://devfeed.tech/tags/humanoid.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [model](<https://devfeed.tech/tags/model.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>)

### AI overview

The paper presents model-free deep reinforcement learning for teaching the K1 humanoid to skateboard without demonstrations or analytic steering models requiring system identification. It combines task rewards with termination constraints that block behaviors including losing the planted board foot, putting both feet on the board, excessive twisting, separating from the board, and unstable board motion. As these behaviors become less prevalent during training, the constraints produce an implicit progression from stance control toward sustained robot-board locomotion.

### Source excerpt

arXiv:2611.26054v1 Announce Type: cross Abstract: Humanoid locomotion research has advanced rapidly on walking and running, but \emph {object-mediated} locomotion, in which a robot must coordinate its body with a passive external object, remains difficult. One challenging task is skateboarding, which requires simultaneous interaction with the ground and a passive but movable platform to achieve forward motion. Current methods for tackling skateboarding with humanoid robots either require human demonstrations or utilise analytic steering models that require system identification. In this paper, we introduce a deep reinforcement learning framework that does not rely on prior data such as demonstrations and enables robots such as the K1 humanoid to learn to skateboard using model-free learning. The method combines task rewards with exploit-blocking terminations. We introduce a number of constraints restricting behaviours such as losing the planted board foot, standing with both feet on the board, excessive twisting, separation from the board, and unstable board motion. Their changing prevalence during training produces an implicit progression from stance control towards sustained robot--board locomotion.

## The Stability Gap in Continual Reinforcement Learning

DevFeed: [The Stability Gap in Continual Reinforcement Learning](<https://devfeed.tech/articles/the-stability-gap-in-continual-reinforcement-learning-145157.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26252>)

Author: Sebastian Pusch, Matthia Sabatelli, Gido M. van de Ven

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [crl](<https://devfeed.tech/tags/crl.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [inexpensive](<https://devfeed.tech/tags/inexpensive.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [performance](<https://devfeed.tech/tags/performance.md>), [proof-of-concept](<https://devfeed.tech/tags/proof-of-concept.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>)

### AI overview

The work introduces a simple continual reinforcement learning benchmark that varies observation representations while keeping environment dynamics and rewards fixed. It uses the benchmark to show a stability gap: after new tasks are introduced, performance on prior tasks drops substantially, then partially or fully recovers, even though the agent retains complete access to those past tasks. The authors say the benchmark supports proof-of-concept demonstrations, inexpensive experiments, and reproducible sanity checks; the excerpt does not specify particular agents or benchmark results beyond this finding.

### Source excerpt

arXiv:2611.26252v1 Announce Type: new Abstract: Continual reinforcement learning (CRL) aims to enable agents to adapt to changing tasks while preserving previously acquired capabilities. Evaluation often relies on final aggregate metrics that can hide temporary performance degradation during adaptation. This work introduces a simple CRL benchmark that varies observation representations while preserving environment dynamics and rewards, supporting proof-of-concept demonstrations, inexpensive experimentation, and easily reproducible sanity checks. Using this benchmark, we demonstrate the stability gap in CRL: a substantial drop in prior-task performance after the introduction of new tasks, followed by partial or complete recovery. This pattern occurs despite the agent having complete access to past tasks.

## Calibrated Decision Rewards: Training Language Models with a Judge That Knows How Sure It Is

DevFeed: [Calibrated Decision Rewards: Training Language Models with a Judge That Knows How Sure It Is](<https://devfeed.tech/articles/calibrated-decision-rewards-training-language-models-with-a-judge-that-knows-how-sure-it-is-145472.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25135>)

Author: Rudransh Agnihotri, Manasvi Kapoor

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [grpo](<https://devfeed.tech/topics/grpo.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>)

Tags: [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [decision-model](<https://devfeed.tech/tags/decision-model.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [probability](<https://devfeed.tech/tags/probability.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward-model](<https://devfeed.tech/tags/reward-model.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

RLCDR uses an LLM decision model to assign calibrated probability distributions to typed questions about a response, such as whether its final answer is correct. The probabilities are read in one forward pass, adjusted for option order, and calibrated using about a hundred labels. In GRPO training of Qwen3-1.7B on MATH, this reference-free reward raised MATH500 accuracy from 72.2% to 76.8%, close to the answer-key reward's 76.9%; a random reward gained 0.65 points. On the hardest level, accuracy rose from 44.8% to 51.7%. The authors report lower calibration error than a softmax judge using the same 27B backbone, and better first-error detection than a dedicated 7B process reward model. A limitation appears on problems the base model mostly fails: RLCDR still improved MATH500 by 1.9 points but lagged the answer-key reward as its probabilities became overconfident on that distribution.

### Source excerpt

arXiv:2611.25135v1 Announce Type: new Abstract: Reinforcement learning (RL) can only teach a language model what its reward can recognize. Answer checkers are exact but need a reference answer; learned reward models and LLM judges work everywhere, but their scores are not probabilities. We study RLCDR (Reinforcement Learning from Calibrated Decision Rewards). An LLM converted into a decision model answers a typed question about a response ("is the final answer correct?", "which step is the first error?") with a probability distribution, read in one forward pass, debiased over option orders and calibrated with about a hundred labels. Training Qwen3-1.7B with GRPO on MATH, this reference-free reward raises MATH500 accuracy from 72.2% to 76.8% (+4.65 points, p < 1e-4), matching training on the answer key (76.9%), while a random reward gains only +0.65. The gain grows with difficulty, from 44.8% to 51.7% on the hardest level. On the policy's own changing samples the reward stays calibrated: its expected calibration error is half that of a softmax judge on the same 27B backbone (0.074 vs. 0.141), and it shows no sign of reward hacking. Without step training, the same model finds the first wrong step better than a dedicated 7B process reward model (ProcessBench F1 0.772 vs. 0.722, 20/20 splits). We also map where the approach stops being enough: trained only on problems the base model mostly fails, the reference-free reward still improves the policy (+1.9 MATH500 points, p = 0.03) but trails the answer key, because its probabilities become overconfident on that distribution. We state six hypotheses in advance and test each with a control designed to falsify it.

## SiliconGym: A Versatile Reinforcement Learning Environment for Agentic EDA

DevFeed: [SiliconGym: A Versatile Reinforcement Learning Environment for Agentic EDA](<https://devfeed.tech/articles/silicongym-a-versatile-reinforcement-learning-environment-for-agentic-eda-145087.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25658>)

Author: Hejia Zhang, Sheng Lu, Yitong Ma, Zhongming Yu, Zhijing Wu, Wentao Ni, Chia-Tung Ho, Jishen Zhao

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [backends](<https://devfeed.tech/tags/backends.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [eda](<https://devfeed.tech/tags/eda.md>), [environment](<https://devfeed.tech/tags/environment.md>), [hardware-design](<https://devfeed.tech/tags/hardware-design.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [ppa](<https://devfeed.tech/tags/ppa.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [repair](<https://devfeed.tech/tags/repair.md>), [rtl](<https://devfeed.tech/tags/rtl.md>), [simulation](<https://devfeed.tech/tags/simulation.md>), [soc](<https://devfeed.tech/tags/soc.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [synthesis](<https://devfeed.tech/tags/synthesis.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

SiliconGym is an RL environment for hardware design and verification that spans module-level RTL, testbench, and PPA work through multi-round repository repair and full-hierarchy SoC flows. It combines versioned workspaces, budgeted execution, acceptance-gated scoring, and streaming dispatch that overlaps tool checks with generation; a 256-answer batch finished 21.9% faster than native ChipSeek. With interchangeable open-source and commercial backends, the authors report that functional correctness trained using open-source tools transfers to commercial simulation. Paired netlist evaluation is used to assess whether open-source PPA gains persist under commercial synthesis. These findings are reported in the supplied abstract.

### Source excerpt

arXiv:2611.25658v1 Announce Type: cross Abstract: Reinforcement learning (RL) with electronic design automation (EDA) feedback is advancing large language models for hardware design and verification. Extending these gains to hardware-engineering agents requires environments that span tasks, preserve state across tool calls, and support large hierarchies alongside parallel rollouts. We present SiliconGym, the first versatile, cross-task and cross-toolchain RL environment for hardware design and verification. By decoupling versioned workspaces, budgeted execution, and acceptance-gated scoring, SiliconGym scales from module-level RTL, testbench, and PPA tasks to multi-round repository repair and full-hierarchy SoC flows. Streaming dispatch overlaps tool checks with generation, finishing a 256-answer RL batch 21.9% faster than native ChipSeek. Using interchangeable open-source and commercial backends, we show that open-source-trained functional correctness transfers cleanly to commercial simulation, while paired netlist evaluation reveals when open-source PPA gains survive commercial synthesis.

## On the Complexity of Offline Reinforcement Learning with Q\*-Approximation and Partial Coverage

DevFeed: [On the Complexity of Offline Reinforcement Learning with Q\*-Approximation and Partial Coverage](<https://devfeed.tech/articles/on-the-complexity-of-offline-reinforcement-learning-with-q-approximation-and-partial-coverage-144877.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.12107>)

Author: Haolin Liu, Braham Snyder, Chen-Yu Wei

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [computational complexity](<https://devfeed.tech/topics/computational-complexity.md>)

Tags: [approximation](<https://devfeed.tech/tags/approximation.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [learning](<https://devfeed.tech/tags/learning.md>), [offline](<https://devfeed.tech/tags/offline.md>), [offline-reinforcement-learning](<https://devfeed.tech/tags/offline-reinforcement-learning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

The paper shows that Q*-realizability and Bellman completeness alone do not guarantee sample-efficient offline reinforcement learning under partial coverage, establishing this with an information-theoretic lower bound. It proposes a decision-estimation framework that separates complexity into decision complexity and value estimation error. The authors report an ε⁻² sample-complexity bound for soft Q-learning under partial coverage, improving on a prior ε⁻⁴ bound; remove the need for additional online interaction in a value-gap setting; and identify further learnable settings. They also characterize Bellman completeness under partial coverage, offline learnability for general low-Bellman-rank MDPs, and provide what they describe as the first function-approximation analysis of Conservative Q-Learning (CQL).

### Source excerpt

arXiv:2602.12107v3 Announce Type: replace Abstract: We study offline reinforcement learning under $Q^\star$-approximation and partial coverage, a setting that motivates practical algorithms such as Conservative $Q$-Learning (CQL; Kumar et al., 2020) but has received limited theoretical attention. Our work is inspired by the following open question: "Are $Q^\star$-realizability and Bellman completeness sufficient for sample-efficient offline RL under partial coverage?" We answer in the negative via an information-theoretic lower bound. To identify additional structure that enables sample-efficient offline RL under partial coverage, we introduce a general decision-estimation framework, inspired by model-free decision-estimation coefficients (DEC) for online RL (Foster et al., 2023b; Liu et al., 2025b). Our framework decomposes offline RL complexity into decision complexity and value estimation error. This allows modular study of both sub-problems. Our result not only unifies existing results (Chen and Jiang, 2022; Uehara et al., 2023), but further improves and generalizes them. On the decision complexity side, our improvement includes: the first $\epsilon^{-2}$ sample complexity bound for soft $Q$-learning under partial coverage that improves Uehara et al.'s (2023) $\epsilon^{-4}$ bound, the removal of the need for additional online interaction in the value-gap setting of Chen and Jiang (2022), and new learnable settings beyond the above two cases. On the value estimation side, we provide a new characterization of the role of Bellman completeness under partial coverage, and the first characterization of offline learnability for general low-Bellman-rank MDPs (Jiang et al., 2017; Du et al., 2021; Jin et al., 2021). The latter is a canonical online RL setting that has remained unexplored in offline RL except for special cases. As a side contribution, our techniques give the first analysis of CQL in the function approximation setting.

## Forget Freely, Grep Later: Lossless, Index-Free Memory for Long-Horizon LLM Agents

DevFeed: [Forget Freely, Grep Later: Lossless, Index-Free Memory for Long-Horizon LLM Agents](<https://devfeed.tech/articles/forget-freely-grep-later-lossless-index-free-memory-for-long-horizon-llm-agents-145470.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25123>)

Author: Ziyan Yang, Taiwei Shi, Jieyu Zhao

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [agent-memory](<https://devfeed.tech/tags/agent-memory.md>), [agents](<https://devfeed.tech/tags/agents.md>), [context-management](<https://devfeed.tech/tags/context-management.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [expression](<https://devfeed.tech/tags/expression.md>), [grep](<https://devfeed.tech/tags/grep.md>), [index](<https://devfeed.tech/tags/index.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [log](<https://devfeed.tech/tags/log.md>), [memory](<https://devfeed.tech/tags/memory.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [recall](<https://devfeed.tech/tags/recall.md>), [regular](<https://devfeed.tech/tags/regular.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [research](<https://devfeed.tech/tags/research.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [search](<https://devfeed.tech/tags/search.md>), [windows](<https://devfeed.tech/tags/windows.md>)

### AI overview

GrepMem stores an agent's complete history in an append-only plain-text log and uses regular-expression search for on-demand recall. This avoids the embeddings, retrieval index, memory extraction, and additional models used by structured memory approaches, while sidestepping losses from summarization or masking. In evaluations with Qwen3.5-9B trained by reinforcement learning, GrepMem reached 95-97% accuracy on ContextBench, compared with at most 21% for lossy baselines, and exceeded SUPO's summarization by 8.8 percentage points on BCP-Multi. The authors report effective scaling to longer streams and a position on the accuracy-token-cost Pareto frontier across five tasks. These results are limited to the reported evaluations; the supplied excerpt does not describe implementation details or broader deployment evidence.

### Source excerpt

arXiv:2611.25123v1 Announce Type: cross Abstract: Long-horizon LLM agents often exceed their context windows, forcing them to discard information that may be needed later. Existing context management strategies rely on summarization, masking, or structured memory, requiring decisions about what to preserve or how to retrieve it before future needs are known. However, these methods can introduce lossy compression, potentially discarding details needed for future tasks. To address this limitation, we introduce GrepMem, a simple, lossless, and index-free memory mechanism built on two components: an append-only plain-text log of the agent's complete history and a regular-expression search tool for on-demand recall. Unlike structured memory systems, GrepMem requires no embeddings, retrieval index, memory extraction, or additional models. We train Qwen3.5-9B with reinforcement learning under five context management strategies and evaluate on multi-question research sessions, long-stream recall, and agent-trajectory question answering. GrepMem achieves 95-97% accuracy on ContextBench, versus at most 21% for lossy baselines, and outperforms SUPO's summarization by 8.8 percentage points on BCP-Multi. GrepMem scales effectively to increasingly long input streams while remaining on the accuracy-token-cost Pareto frontier across all five evaluation tasks. Our results show that a plain-text log and grep provide a simple but competitive approach to long-horizon agent memory, without requiring complex structured memory infrastructure.

## RL Can Only Reinforce What It Sees Succeed

DevFeed: [RL Can Only Reinforce What It Sees Succeed](<https://devfeed.tech/articles/rl-can-only-reinforce-what-it-sees-succeed-145138.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26084>)

Author: Ruben Sanchez-Corcuera, Unai Zulaika, Aritz Bilbao-Jayo, Miguel Fernandez-de-Retana

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [grpo](<https://devfeed.tech/topics/grpo.md>)

Tags: [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [puzzle](<https://devfeed.tech/tags/puzzle.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [rl](<https://devfeed.tech/tags/rl.md>), [small-language-models](<https://devfeed.tech/tags/small-language-models.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

In a controlled puzzle world where small language models' training priors and correct answers are known, outcome-reward RL turns models' prior hedging into decisions favored by reward. But when the correct action is rare, learning is inconsistent across runs: an action must succeed in enough groups of attempts for RL to reinforce it. The study reports that the expected number of winning groups predicts which runs learn, with about 35 groups corresponding to even odds. This relationship held for new and repeated problems, group sizes of 4-256 attempts, and models from 1.3M to 85M parameters, across the tested learning rates, optimizers, and GRPO variants; these choices shifted the threshold but not the predictive count.

### Source excerpt

arXiv:2611.26084v1 Announce Type: new Abstract: When a familiar rule stops working, an agent has to weigh what pre-training taught it against what it now observes. We ask how outcome-reward reinforcement learning (RL) changes this balance, in a puzzle world where both sides are known: small language models trained from scratch on games played by a Bayesian player, so that the prior and the right answer are exact. Base models hedge as their corpus did, and RL turns that hedging into decisions, in whichever direction the reward favours. But RL can only reinforce what it sees succeed. When the right action is rare, each run either learns it everywhere or not at all, and which runs take off is predicted before training by one count: the expected number of groups of attempts in which the action wins. About 35 such groups give even odds. The count holds for new and repeated problems and for groups of 4 to 256 attempts, and across learning rates, optimizers, GRPO variants and model sizes from 1.3M to 85M parameters, which move the threshold but not what is counted.

## X Hedging: towards an eXplainable artificial intelligence option Hedging framework

DevFeed: [X Hedging: towards an eXplainable artificial intelligence option Hedging framework](<https://devfeed.tech/articles/x-hedging-towards-an-explainable-artificial-intelligence-option-hedging-framework-145112.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25870>)

Author: Bj{\\o}rn Andr\\'e Aaslund, Johannes Berge, Ayoub Haida, Ying Ni, Rita Pimentel

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Research](<https://devfeed.tech/topics/ai-research.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Deep learning](<https://devfeed.tech/topics/deep-learning.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>)

Tags: [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [data-driven](<https://devfeed.tech/tags/data-driven.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [explainability](<https://devfeed.tech/tags/explainability.md>), [framework](<https://devfeed.tech/tags/framework.md>), [lightgbm](<https://devfeed.tech/tags/lightgbm.md>), [math-oc](<https://devfeed.tech/tags/math-oc.md>), [models](<https://devfeed.tech/tags/models.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [neural-networks](<https://devfeed.tech/tags/neural-networks.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [option](<https://devfeed.tech/tags/option.md>), [performance](<https://devfeed.tech/tags/performance.md>), [policies](<https://devfeed.tech/tags/policies.md>), [random](<https://devfeed.tech/tags/random.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [robustness](<https://devfeed.tech/tags/robustness.md>), [x](<https://devfeed.tech/tags/x.md>)

### AI overview

X Hedging replaces the neural networks used in Deep Hedging with LightGBM tree ensembles to estimate dynamic hedging strategies. It sequentially trains the models and adds stochasticity through randomized training orders. In controlled experiments, it reportedly performs comparably to Deep Hedging while improving computational efficiency and user-centered explainability, with substantially greater robustness when training data are limited. The supplied abstract does not provide experiment details or quantify these gains.

### Source excerpt

arXiv:2611.25870v1 Announce Type: new Abstract: We propose X Hedging, an alternative to the Deep Hedging framework that addresses computational and explainability limitations of neural-network-based hedging strategies. Deep Hedging leverages neural networks to learn flexible hedging policies but relies on opaque models and typically requires large training datasets, which can complicate interpretation and limit its effectiveness in data-constrained settings. X Hedging takes a step toward addressing these limitations by replacing neural networks with tree ensemble methods, specifically LightGBM, to estimate hedging strategies. This design provides a computationally efficient alternative while preserving strong hedging performance and moving toward more user-centered explainability. To integrate LightGBM into a reinforcement-learning-style hedging framework, we introduce an optimization approach that sequentially trains tree-ensemble models while introducing stochasticity through random training orders. In controlled experiments, X Hedging achieves performance comparable to Deep Hedging while offering greater user-centered explainability, improved computational efficiency, and substantially higher robustness when training data are limited. The numerical results demonstrate that tree-ensemble methods provide an effective computational alternative to neural networks for data-driven dynamic hedging.

## Structured Role-Aware Policy Optimization for Multimodal Reasoning

DevFeed: [Structured Role-Aware Policy Optimization for Multimodal Reasoning](<https://devfeed.tech/articles/structured-role-aware-policy-optimization-for-multimodal-reasoning-144892.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.07274>)

Author: Bingqing Jiang, Guoxi Zhang, Jasper Wang, Bingning Wang, Difan Zou

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [grpo](<https://devfeed.tech/topics/grpo.md>), [rlvr](<https://devfeed.tech/topics/rlvr.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>)

Tags: [capabilities](<https://devfeed.tech/tags/capabilities.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [credit-assignment](<https://devfeed.tech/tags/credit-assignment.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dependency](<https://devfeed.tech/tags/dependency.md>), [evidence-based](<https://devfeed.tech/tags/evidence-based.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [grounded](<https://devfeed.tech/tags/grounded.md>), [group-relative-policy-optimization](<https://devfeed.tech/tags/group-relative-policy-optimization.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [policy](<https://devfeed.tech/tags/policy.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reasoning-tokens](<https://devfeed.tech/tags/reasoning-tokens.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>), [rlvr](<https://devfeed.tech/tags/rlvr.md>), [roles](<https://devfeed.tech/tags/roles.md>), [structured](<https://devfeed.tech/tags/structured.md>)

### AI overview

SRPO refines GRPO's sequence-level advantage into token-level credit for perception and reasoning, aiming to improve whether correct multimodal answers rely on relevant visual evidence. It uses on-policy contrasts between original and corrupted images to weight perception tokens by visual dependence, and weights reasoning tokens by consistency with the generated perception. A shared trajectory-level baseline combines these signals into positive weights that adjust update magnitudes while preserving the original reward and advantage signs. The method requires neither external reward models nor separate teachers. The abstract reports improved evidence-grounded reasoning across diverse multimodal benchmarks; it gives no specific benchmark results or quantitative gains.

### Source excerpt

arXiv:2605.07274v2 Announce Type: replace-cross Abstract: Reinforcement learning from verifiable rewards (RLVR), especially with Group Relative Policy Optimization (GRPO), has shown strong potential for improving the reasoning capabilities of large vision-language models (LVLMs). However, in multimodal reasoning, final-answer rewards are typically assigned at the sequence level and do not distinguish the functional roles of different tokens, making it difficult to determine whether a correct answer is supported by task-relevant visual evidence. In this paper, we revisit multimodal RLVR from the perspective of role-aware token-level credit assignment, where structured responses are decomposed into perception tokens for extracting visual evidence and reasoning tokens for deriving answers from that evidence. Based on this perspective, we propose Structured Role-aware Policy Optimization (SRPO), which refines the sequence-level GRPO advantage into role-aware token-level advantages without changing the reward function. Specifically, SRPO assigns role-specific credit by using self-distilled on-policy contrasts: perception tokens are emphasized according to their visual dependency under original versus corrupted visual inputs, while reasoning tokens are emphasized according to their consistency with the generated perception. These role-specific signals are further unified through a shared trajectory-level baseline, yielding positive token weights that adjust relative update magnitudes while preserving the original GRPO reward and advantage signs, without requiring external reward models or separate teachers. Experiments across diverse multimodal reasoning benchmarks show that SRPO improves evidence-grounded reasoning, highlighting the importance of moving beyond uniform sequence-level credit toward role-aware optimization for reliable multimodal reasoning.

## Security of Deep Reinforcement Learning for Autonomous Driving: A Survey

DevFeed: [Security of Deep Reinforcement Learning for Autonomous Driving: A Survey](<https://devfeed.tech/articles/security-of-deep-reinforcement-learning-for-autonomous-driving-a-survey-144846.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2212.06123>)

Author: Ambra Demontis, Srishti Gupta, Maura Pintor, Luca Demetrio, Kathrin Grosse, Hsiao-Ying Lin, Chengfang Fang, Battista Biggio, Fabio Roli

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [ai security](<https://devfeed.tech/topics/ai-security.md>), [autonomous vehicles](<https://devfeed.tech/topics/autonomous-vehicles.md>), [Adversarial attacks](<https://devfeed.tech/topics/adversarial-attacks.md>)

Tags: [autonomous-driving](<https://devfeed.tech/tags/autonomous-driving.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [security](<https://devfeed.tech/tags/security.md>), [survey](<https://devfeed.tech/tags/survey.md>)

### AI overview

The survey examines how attacks can undermine reinforcement learning by compromising policy learning or causing trained agents to make erroneous decisions, a concern for safety-critical uses such as autonomous driving. It reviews 86 recent studies and organizes attacks and defenses by threat model and single- versus multi-agent setting, aiming to help select defenses for particular systems. It also assesses how state-of-the-art attacks and defenses apply to autonomous driving. The supplied text does not describe specific attack or defense methods or their comparative effectiveness.

### Source excerpt

arXiv:2212.06123v5 Announce Type: replace Abstract: Reinforcement learning (RL) enables agents to learn optimal behaviors through interaction with their environment and has been increasingly deployed in safety-critical applications, including autonomous driving. Despite its promise, RL is susceptible to attacks designed either to compromise policy learning or to induce erroneous decisions by trained agents. Although the literature on RL security has grown rapidly and several surveys exist, existing categorizations often fall short in guiding the selection of appropriate defenses for specific systems. In this work, we present a comprehensive survey of 86 recent studies on RL security, addressing these limitations by systematically categorizing attacks and defenses according to defined threat models and single- versus multi-agent settings. Furthermore, we examine the relevance and applicability of state-of-the-art attacks and defense mechanisms within the context of autonomous driving, providing insights to inform the design of robust RL systems.

## Extensions of Robbins-Siegmund Theorem with Applications in Reinforcement Learning

DevFeed: [Extensions of Robbins-Siegmund Theorem with Applications in Reinforcement Learning](<https://devfeed.tech/articles/extensions-of-robbins-siegmund-theorem-with-applications-in-reinforcement-learning-144860.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2509.26442>)

Author: Xinyu Liu, Zixuan Xie, Shangtong Zhang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>)

Tags: [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [function-approximation](<https://devfeed.tech/tags/function-approximation.md>), [linear](<https://devfeed.tech/tags/linear.md>), [math-oc](<https://devfeed.tech/tags/math-oc.md>), [probability](<https://devfeed.tech/tags/probability.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [stochastic-processes](<https://devfeed.tech/tags/stochastic-processes.md>)

### AI overview

The paper extends the Robbins-Siegmund theorem to cases where the zero-order term is square-summable rather than summable, adding an assumption on process increments to establish almost sure convergence to a bounded set. It also states almost sure convergence rates, high-probability concentration bounds, and L^p convergence rates. Applied to stochastic approximation and reinforcement learning, the results include claimed first rates and bounds of these kinds for Q-learning with linear function approximation. The supplied text is an abstract, so it does not specify the increment assumption or technical conditions.

### Source excerpt

arXiv:2509.26442v3 Announce Type: replace Abstract: The Robbins-Siegmund theorem establishes the convergence of stochastic processes that are almost supermartingales and is one of the most commonly used approaches for analyzing stochastic iterative algorithms in stochastic approximation and reinforcement learning (RL). However, its original form has a significant limitation as it requires the zero-order term to be summable. In many important RL applications, this summable condition, however, cannot be met. This limitation motivates us to extend the Robbins-Siegmund theorem for almost supermartingales where the zero-order term is not summable, but only square-summable. In particular, we introduce a novel and mild assumption on the increments of the stochastic processes. This together with the square-summable condition enables an almost sure convergence to a bounded set. Additionally, we further provide almost sure convergence rates, high probability concentration bounds, and $L^p$ convergence rates. We then apply the new results to stochastic approximation and RL. Notably, we obtain the first almost sure convergence rate, the first high probability concentration bound, and the first $L^p$ convergence rate for $Q$-learning with linear function approximation.

## Improving Imitation Policies for Multi-Phase Tasks through Residual Learning Allocation

DevFeed: [Improving Imitation Policies for Multi-Phase Tasks through Residual Learning Allocation](<https://devfeed.tech/articles/improving-imitation-policies-for-multi-phase-tasks-through-residual-learning-allocation-145042.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25320>)

Author: Clinton Enwerem, Saleh Kemal, John S. Baras, Calin Belta

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [assembly](<https://devfeed.tech/tags/assembly.md>), [budget](<https://devfeed.tech/tags/budget.md>), [completion](<https://devfeed.tech/tags/completion.md>), [components](<https://devfeed.tech/tags/components.md>), [controller](<https://devfeed.tech/tags/controller.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [cs-sy](<https://devfeed.tech/tags/cs-sy.md>), [eess-sy](<https://devfeed.tech/tags/eess-sy.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [failure](<https://devfeed.tech/tags/failure.md>), [imitation](<https://devfeed.tech/tags/imitation.md>), [learning](<https://devfeed.tech/tags/learning.md>), [phases](<https://devfeed.tech/tags/phases.md>), [policies](<https://devfeed.tech/tags/policies.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [release](<https://devfeed.tech/tags/release.md>), [residual](<https://devfeed.tech/tags/residual.md>), [robotic](<https://devfeed.tech/tags/robotic.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

The paper allocates residual reinforcement learning by estimated completion gain rather than phase failure counts. Completion gain measures the change in full-task completion probability when a state-feedback controller replaces selected base-policy action components in one phase; paired episodes estimate it by factoring in both the chance of reaching that phase and the conditional improvement from there. Positive estimates guide which components to correct and where to spend training episodes. In point-mass docking, the channel phase had the largest gain despite most failures occurring during final docking; gain-based allocation outperformed failure-count allocation, and in a longer-approach variant beat uniform allocation at half the interaction budget. In simulated bimanual assembly, gains were positive only for insertion and release, with few release failures; training in release improved held-out completion by 16.9% in a second configuration.

### Source excerpt

arXiv:2611.25320v1 Announce Type: new Abstract: Residual reinforcement learning corrects an imitation-trained base policy using a limited budget of environment interaction. In a task with several phases, this budget must be divided among the phases, and the phase in which the base policy fails is not necessarily the phase in which a correction most improves task completion. We formalize this distinction through the completion gain: the change in full-task completion probability when, during a single phase, a state-feedback controller replaces selected components of the base policy's action. We decompose the completion gain into the probability of reaching the phase and the conditional improvement in completion from the start of the phase, and we estimate it from paired episodes. The positive estimates determine which action components the residual policy corrects and how training episodes are distributed across phases. In stochastic point-mass docking experiments, most failures occurred in the final docking phase, whereas the largest completion gain occurred in the preceding channel phase; gain-based allocation improved completion over allocation proportional to failure counts and, in a variant with a longer approach, over uniform allocation at half of the budget. In simulated bimanual robotic assembly, the completion-gain estimates were positive only for insertion and release, although few failures occurred during release; in a second assembly configuration, a residual policy trained in the release phase improved completion from held-out states at the start of release by 16.9%.

## Almost Sure Convergence Rates and Concentration of Stochastic Approximation and Reinforcement Learning with Markovian Noise

DevFeed: [Almost Sure Convergence Rates and Concentration of Stochastic Approximation and Reinforcement Learning with Markovian Noise](<https://devfeed.tech/articles/almost-sure-convergence-rates-and-concentration-of-stochastic-approximation-and-reinforcement-learning-with-markovian-noise-144850.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2411.13711>)

Author: Xiaochi Qian, Zixuan Xie, Xinyu Liu, Shangtong Zhang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [approximation](<https://devfeed.tech/tags/approximation.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [learning](<https://devfeed.tech/tags/learning.md>), [math-oc](<https://devfeed.tech/tags/math-oc.md>), [noise](<https://devfeed.tech/tags/noise.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

The paper reports almost sure convergence rates and maximal concentration bounds with sub-Weibull tails for general contractive stochastic approximation algorithms under Markovian noise, and derives convergence rates in L^p. Its key method is to discretize the mean ODE using intervals of diminishing length. Applications include an almost sure convergence rate for Q-learning without count-based learning rates and a maximal concentration bound for off-policy temporal difference learning, both with Markovian samples. These are claims from the supplied abstract; no proof details or quantitative rate formulas are included.

### Source excerpt

arXiv:2411.13711v2 Announce Type: replace Abstract: This paper establishes, to our knowledge, the first almost sure convergence rate and the first maximal concentration bound with sub-Weibull tails for general contractive stochastic approximation algorithms with Markovian noise. As a corollary, we also obtain convergence rates in $L^p$. Key to our successes is a novel discretization of the mean ODE of stochastic approximation algorithms using intervals with diminishing (instead of constant) length. Applications yield the first almost sure convergence rate for $Q$-learning without count-based learning rates and the first maximal concentration bound for off-policy temporal difference learning, both with Markovian samples.

## SheetAlign: Grounded Interaction and Stage-Isolated Learning for Spreadsheet Agents

DevFeed: [SheetAlign: Grounded Interaction and Stage-Isolated Learning for Spreadsheet Agents](<https://devfeed.tech/articles/sheetalign-grounded-interaction-and-stage-isolated-learning-for-spreadsheet-agents-145786.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26069>)

Author: Yuxiang Wang, Junhao Gan, Jianzhong Qi

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [grounded](<https://devfeed.tech/tags/grounded.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [learning](<https://devfeed.tech/tags/learning.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [spreadsheet](<https://devfeed.tech/tags/spreadsheet.md>), [weight](<https://devfeed.tech/tags/weight.md>)

### AI overview

SheetAlign reports improved spreadsheet-task accuracy by grounding agent actions in workbook observations and available APIs, then training program generation and review in separate stages. Its workflow addresses where to operate, what to change, and how to implement transformations; edit-progress rewards guide learning, and inference selects programs based on agreement among resulting workbook states. On SpreadsheetBench, the grounded interface raises Qwen3-4B's average accuracy from 13.5% to 29.2% without training. With learning and execution consistency, SheetAlign reports 32.8% average accuracy on SpreadsheetBench and 30.1% on Domain-Spreadsheet, outperforming comparable baselines. The abstract frames local open-weight models as a way to address sensitive-data and deployment-cost concerns; it does not provide implementation details or evidence beyond these reported results.

### Source excerpt

arXiv:2611.26069v1 Announce Type: cross Abstract: Spreadsheets are widely used for data analysis and supporting decision-making. Automating spreadsheet analysis while protecting sensitive data and controlling deployment costs motivates locally deployed open-weight large language models (LLMs). Meanwhile, complex spreadsheet tasks require both task reasoning and workbook understanding, while predefined operation interfaces impose additional tool-selection and invocation demands. We introduce SheetAlign, a spreadsheet manipulation framework with stage-isolated policy learning, organized around where to operate, what to change, and how to implement transformations. Task-guided inspection and API grounding anchor code generation in workbook observations and available operations, while stage-isolated learning optimizes program generation and review using edit-progress rewards. At inference, execution consistency selects programs through agreement among workbook states. On SpreadsheetBench, our grounded task interface improves Qwen3-4B's accuracy from 13.5% to 29.2% on average without training. With learning and execution consistency, SheetAlign achieves 32.8% accuracy on average on SpreadsheetBench and 30.1% on Domain-Spreadsheet, outperforming comparable baselines. The code will be made publicly available upon publication.

## Collision-Free Is Not Enough: Passage-Conditioned Navigation for Wheeled-Legged Robots

DevFeed: [Collision-Free Is Not Enough: Passage-Conditioned Navigation for Wheeled-Legged Robots](<https://devfeed.tech/articles/collision-free-is-not-enough-passage-conditioned-navigation-for-wheeled-legged-robots-145587.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25788>)

Author: Wai Ka Sun, Waiyiu Pong, Bowen Wang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Robot Navigation](<https://devfeed.tech/topics/robot-navigation.md>), [navigation](<https://devfeed.tech/topics/navigation.md>), [few-shot](<https://devfeed.tech/topics/few-shot.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>)

Tags: [competition](<https://devfeed.tech/tags/competition.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [legged-robots](<https://devfeed.tech/tags/legged-robots.md>), [navigation](<https://devfeed.tech/tags/navigation.md>), [robots](<https://devfeed.tech/tags/robots.md>)

### AI overview

Passage-conditioned navigation records a successful taught crossing for each stair or ledge and, after every local replan, uses an entry gate to make the robot turn or back up and retry when its approach differs by more than 10 cm or 10°; it requires neither a robot model nor additional training. With path, gait schedule, and sensing unchanged, the method increased disturbed ledge crossings within those limits from 24.1% to 88.6%. In contact simulation, robots pushed off course before stairs completed 180 climbs without falling, compared with 9 falls when the stored direction was ignored; after a 1.2 m displacement, they rejoined the route and ended an average 0.14 m from it, versus 1.10 m without rejoining. A real wheeled-legged robot completed six competition runs across 271 m of multi-obstacle terrain and 7.0 m of elevation, finishing fully autonomously.

### Source excerpt

arXiv:2611.25788v1 Announce Type: cross Abstract: A collision-free detour can still bring a legged robot to the next stair or ledge a few centimetres or degrees off, and a fixed locomotion skill, factory or learned, can stall or fall. Existing methods prevent this with a robot model or extensive trial and error, but on a new site with unknown skills neither is available, so they cannot be used. We find that a single 27-minute teaching session already supplies what these methods have to derive or learn: for every stair and ledge, one crossing the skill has completed. Our method, passage-conditioned navigation, makes the route remember how each transition was entered, not only where the path went. It stores each taught crossing as an event and, without training or a robot model, enforces it after every local replan: before each ledge, an entry gate makes the robot turn or back up and retry if it is more than 10 cm or 10{\deg} off the taught crossing. Under localization noise, with path, gait schedule, and sensing unchanged, this raises the share of disturbed ledge crossings that stay within these limits from 24.1% to 88.6%. In contact simulation with our learned policies, a robot pushed off course before stairs never falls in 180 climbs, against 9 falls when the stored direction is ignored; pushed 1.2 m away, it rejoins the route, ending on average 0.14 m from it (1.10 m without rejoining), and finishes as often as an undisturbed robot. This suits any robot with fixed skills that repeats a route over terrain transitions, as in inspection and patrol, where one wrong entry can end the task. We deployed it in competition on a real wheeled-legged robot: over 271 m of multi-obstacle terrain spanning 7.0 m of elevation, it completed all six runs, won third place at the 2026 Global Open-source AI Challenge, which drew 2,999 submissions, and was the only finalist in its task to finish fully autonomously.

## ZeroVLN: Learning to Navigate without Supervised Fine-Tuning of the VLM Brain

DevFeed: [ZeroVLN: Learning to Navigate without Supervised Fine-Tuning of the VLM Brain](<https://devfeed.tech/articles/zerovln-learning-to-navigate-without-supervised-fine-tuning-of-the-vlm-brain-145668.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26414>)

Author: Zixuan Huang, Yang Zhou, Guli Zhang, Sunzhu Li, Hongyan Xie, Zehao Chen, Fuzhen Zhuang, Kaixuan Wang, Xiaozhi Chen, Yikun Ban, Deqing Wang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Robot Navigation](<https://devfeed.tech/topics/robot-navigation.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [model](<https://devfeed.tech/tags/model.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [navigation](<https://devfeed.tech/tags/navigation.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [sft](<https://devfeed.tech/tags/sft.md>), [tool-calling](<https://devfeed.tech/tags/tool-calling.md>), [vision-language-model](<https://devfeed.tech/tags/vision-language-model.md>)

### AI overview

ZeroVLN targets a tradeoff in vision-language navigation: fine-tuning the VLM brain on navigation data can change its weights and weaken general visual capabilities. Instead, the framework leaves the brain without navigation-domain supervised fine-tuning and gives it a semantic tool-calling interface to send natural-language movement requests to an action expert. The expert is first adapted to execute those requests; closed-loop reinforcement learning then jointly improves the brain's movement decisions and the expert's continuous control. On standard VLN-CE benchmarks, using the same navigation episodes, the authors report substantial gains over representative SFT-based hierarchical methods, with only minor changes to the brain's weights and largely preserved general capabilities.

### Source excerpt

arXiv:2611.26414v1 Announce Type: new Abstract: Representative vision-language navigation (VLN) methods typically couple a vision-language model (VLM) brain with a learnable action head and fine-tune the VLM brain on navigation-domain data to predict geometric goals. However, such adaptation can substantially alter base VLM weights and degrade general visual capabilities. We introduce ZeroVLN, a slow-fast navigation framework that achieves effective navigation with zero navigation-domain SFT of the VLM brain. Rather than teaching the brain to predict geometric goals, we equip it with a semantic tool-calling interface for conveying movement decisions to an action expert in natural language. We first adapt the action expert to carry out these movement requests, then jointly improve the brain's movement decisions and the action expert's continuous execution through closed-loop reinforcement learning (RL). Trained on the same navigation episodes, ZeroVLN substantially outperforms representative SFT-based hierarchical VLN methods on standard VLN-CE (continuous environments) benchmarks, while inducing only minor changes to the brain's weights and largely preserving its general capabilities. To our knowledge, ZeroVLN is the first slow-fast VLN framework with zero navigation-domain SFT of the VLM brain, and the first to jointly optimize its slow and fast policies through closed-loop agentic RL in VLN-CE.

## Cross-Question Reliable Reinforcement Learning

DevFeed: [Cross-Question Reliable Reinforcement Learning](<https://devfeed.tech/articles/cross-question-reliable-reinforcement-learning-145036.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25292>)

Author: Hector G. Rodriguez, Marcus Rohrbach

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Uncertainty quantification LLMs](<https://devfeed.tech/topics/uncertainty-quantification-llms.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [error](<https://devfeed.tech/tags/error.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [learning](<https://devfeed.tech/tags/learning.md>), [math](<https://devfeed.tech/tags/math.md>), [math-500](<https://devfeed.tech/tags/math-500.md>), [qa](<https://devfeed.tech/tags/qa.md>), [questions](<https://devfeed.tech/tags/questions.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>), [risk](<https://devfeed.tech/tags/risk.md>), [solved](<https://devfeed.tech/tags/solved.md>), [utility](<https://devfeed.tech/tags/utility.md>), [wikipedia](<https://devfeed.tech/tags/wikipedia.md>)

### AI overview

CARE (Cross-question Abstention Reward Estimation) trains LLMs to rank confidence across a set of question-answer pairs. For several risk tolerances, the set's confidences determine an abstention threshold, and rewards favor confidence scores on the correct side of that threshold. This contrasts with pointwise rewards that push each answer toward confidence 1 if correct and 0 if incorrect, which the authors say loses useful distinctions for selective answering. In math benchmarks MATH-500, GSM8K, and Big-Math-Digits, CARE raises coverage at 5% risk (C@5) from 0.8% to 42.4% versus existing pointwise rewards. Training on HotpotQA also raises C@5 from 12.3% to 23.6% on out-of-distribution general-knowledge, reasoning, and math benchmarks, including CommonsenseQA, GPQA, SimpleQA, and TriviaQA. These findings are reported in the supplied abstract.

### Source excerpt

arXiv:2611.25292v1 Announce Type: new Abstract: AI systems operating near the limits of their capabilities can maximize utility while minimizing the risk of error by abstaining or deferring questions they are not confident about to a human or a stronger model. Recently, large language models (LLMs) have been post-trained to estimate confidence in their own answers. To achieve this, methods reinforce confidence scores based on the correctness signal to each question-answer pair individually, with rewards that encourage assigning confidence 1 to correct answers and 0 to incorrect answers. However, estimating confidence scores is useful in that it enables selecting only the most confident answers, to maximize the share of solved questions while accommodating the user's risk tolerance. Assigning confidence 1 to correct answers and 0 to incorrect ones is theoretically optimal, but unattainable in practice and detrimental for selective prediction with a non-zero risk tolerance: confidences pushed to 0 and 1 carry no information about how answers compare to one another. We therefore introduce CARE (Cross-question Abstention Reward Estimation), which computes rewards over a set of question-answer pairs: At each of several risk tolerances, the confidences across the set induce an abstention threshold, and each confidence is rewarded for lying on the correct side of it. Since the threshold itself comes from the set, every verbalized confidence is reinforced relative to other questions' confidence, not against its own binary correctness label. On math, CARE increases the coverage at 5% risk (C@5) across MATH-500, GSM8K, and Big-Math-Digits from 0.8% to 42.4%, compared to existing pointwise correctness-matching rewards. When trained on multi-hop Wikipedia QA from HotpotQA, CARE improves C@5 in out-of-distribution general-knowledge, reasoning, and math benchmarks (including CommonsenseQA, GPQA, SimpleQA, and TriviaQA) from 12.3% to 23.6%.

## Controlling the Humanoid Robot H1-2 With a Simulated Biologically Plausible Network

DevFeed: [Controlling the Humanoid Robot H1-2 With a Simulated Biologically Plausible Network](<https://devfeed.tech/articles/controlling-the-humanoid-robot-h1-2-with-a-simulated-biologically-plausible-network-145630.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26074>)

Author: Fatima Atwi (LAAS-GEPETTO, Nantes Univ - ECN), Constant Roux (LAAS-GEPETTO), Christian Bergaud (LAAS-MEMS), Olivier Stasse (LAAS-GEPETTO)

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Humanoid Robots](<https://devfeed.tech/topics/humanoid-robots.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>)

Tags: [backpropagation](<https://devfeed.tech/tags/backpropagation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [culture](<https://devfeed.tech/tags/culture.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [e-prop](<https://devfeed.tech/tags/e-prop.md>), [humanoid](<https://devfeed.tech/tags/humanoid.md>), [humanoid-robot](<https://devfeed.tech/tags/humanoid-robot.md>), [learning](<https://devfeed.tech/tags/learning.md>), [matching](<https://devfeed.tech/tags/matching.md>), [network](<https://devfeed.tech/tags/network.md>), [performance](<https://devfeed.tech/tags/performance.md>), [plasticity](<https://devfeed.tech/tags/plasticity.md>), [recurrent](<https://devfeed.tech/tags/recurrent.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [robust](<https://devfeed.tech/tags/robust.md>), [spiking-neural-networks](<https://devfeed.tech/tags/spiking-neural-networks.md>), [stdp](<https://devfeed.tech/tags/stdp.md>), [survival](<https://devfeed.tech/tags/survival.md>), [trends](<https://devfeed.tech/tags/trends.md>), [unitree](<https://devfeed.tech/tags/unitree.md>)

### AI overview

The study compares three learning schemes for a biologically inspired spiking network controlling the Unitree H1-2 humanoid. All use distillation of a reinforcement-learning expert policy with dataset aggregation. The fixed-reservoir approach freezes recurrent weights and trains only a linear readout; reward-modulated STDP with eligibility propagation trains synapses; backpropagation through time serves as a non-biological baseline. E-prop achieved a 100% survival rate, matching backpropagation through time, while the fixed reservoir and reward-modulated STDP reached about 43.7% and 11%, respectively. The authors also report brief bipedal stepping on the full-size robot. These findings, as presented in the abstract, show a performance gap between e-prop and the two approaches described as closest to biological cultures.

### Source excerpt

arXiv:2611.26074v1 Announce Type: new Abstract: Artificial Spiking Neural Networks have gained remarkable attention in the last few years due to their proven ability in simulating biological neural cultures and producing similar activity and trends. Nevertheless, whether they can be trained for humanoid locomotion while retaining biological plausibility and robust performance at once remains an open question. We study this problem by proposing a biologically inspired spiking network, trained by different learning schemes and tested on the H1-2 Unitree Humanoid. These schemes differ in how recurrent weights are trained. Ordered from most to least biologically faithful, they are: a fixed reservoir mimicking an in-vitro neural culture, where recurrent weights are kept frozen and only a linear readout is trained, the same network but with its synapses trained by reward-modulated spike timing dependent plasticity and eligibility propagation, and finally backpropagation through time as a non biological baseline for comparison. All schemes relied on distillation of a reinforcement learning expert policy with dataset aggregation. We show that e-prop reached 100% survival rate matching backpropagation through time, whereas both the fixed reservoir and the reward-modulated STDP, though the closest to biological cultures, plateaued at a much lower performance (~43.7% and ~11% survival). We explore in further detail which trends a real culture could and could not produce until now. Finally we demonstrate brief bipedal stepping on a full size humanoid H1-2.

## CHORUS: Coverage-Guided Agentic RL and Policy Consolidation for Testbench Stimulus Generation

DevFeed: [CHORUS: Coverage-Guided Agentic RL and Policy Consolidation for Testbench Stimulus Generation](<https://devfeed.tech/articles/chorus-coverage-guided-agentic-rl-and-policy-consolidation-for-testbench-stimulus-generation-144918.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.10090>)

Author: Hejia Zhang, Sheng Lu, Zhijing Wu, Yitong Ma, Zhongming Yu, Wentao Ni, Chia-Tung Ho, Jishen Zhao

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [generation](<https://devfeed.tech/tags/generation.md>), [policy](<https://devfeed.tech/tags/policy.md>), [rl](<https://devfeed.tech/tags/rl.md>), [stimulus](<https://devfeed.tech/tags/stimulus.md>), [testbench](<https://devfeed.tech/tags/testbench.md>)

### AI overview

CHORUS uses coverage-guided reinforcement learning to improve LLM-generated testbench stimulus: the model drafts testbenches, then repairs its lowest-scoring draft using simulator feedback. On CVDP-ECov, this raised LLM4Cov agentic Pass@1 from 69.2% to 85.5%. Three separately trained checkpoints performed similarly overall but succeeded on different designs; averaging their weights produced a dense 4B model at 86.7%. Selecting a teacher per prompt by simulator reward and distilling it into a 4B model reached 88.0%, reported as 13.5 percentage points above DeepSeek-R1 (671B).

### Source excerpt

arXiv:2608.10090v2 Announce Type: replace-cross Abstract: Testbenches written by large language models can compile and run while leaving entire datapaths idle. We present CHORUS, the first framework to combine coverage-guided agentic reinforcement learning (RL) with policy consolidation for testbench stimulus generation. We train LLM4Cov's final supervised checkpoint to draft testbenches and repair its lowest-scoring draft using simulator feedback, raising CVDP-ECov agentic Pass@1 from 69.2% to 85.5%. Applying the same training to three checkpoints yields similar overall pass rates, yet the policies succeed on different designs. Averaging their weights combines these strengths in a single dense 4B model that reaches 86.7% agentic Pass@1. When simulator rewards select the teacher for each training prompt, the distilled 4B model reaches 88.0%, 13.5 percentage points above DeepSeek-R1 (671B).

[Next page](<https://devfeed.tech/topics/reinforcement-learning.md?cursor=WyIyMDI2LTExLTIwVDA1OjAwOjAwKzAwOjAwIiwgIjQzMWU3MGQxLWRhYjUtNGI2NC1iOWE2LTc3OGZiN2VhYTI2OSJd>)