# group relative policy optimization Published articles for group relative policy optimization. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Are Dense Rewards Worth Their Tokens in GRPO? DevFeed: [Are Dense Rewards Worth Their Tokens in GRPO?]() Original publisher: [Read original article]() Author: Vikash Singh, Weicong Chen, Debargha Ganguly, Yanyan Zhang, Mohsen Hariri, Sreehari Sankar, Wang Yang, Chaoda Song, Alexander Nemecek, Osama Zafar, Yuting Shao, Kai Ye, Biyao Zhang, Xinpeng Li, Shoure Published: 2026-11-11T05:00:00Z Content type: news Language: en Sources: [arXiv: Artificial Intelligence]() Topics: [grpo](), [Reinforcement learning](), [llm-reasoning]() Tags: [1-7](), [algorithm](), [budget](), [compute](), [controls](), [costs](), [cs-ai](), [cs-cl](), [gpu](), [gradient](), [group-relative-policy-optimization](), [grpo](), [intermediate](), [language-models](), [learning](), [matching](), [models](), [monte-carlo](), [optimization](), [points](), [policy](), [tokens]() ### AI overview Matched-compute evaluation changes the apparent ranking of GRPO training interventions: dense rewards that can look better per update may perform worse once their token and GPU-hour costs are counted. In 1,668 runs across six models, Monte-Carlo rewards used continuations accounting for 91% of generated tokens in this setting; on Qwen2.5-1.5B, this method finished 14.3 percentage points below standard GRPO at equal tokens and 7.8 points below at equal training time. None of four dense-reward variants beat GRPO with equal training time. A per-token gradient test ranked rewards similarly to training but overstated their performance by one to two points. Selecting prompts with predicted success near 1/2 improved accuracy by 1.7 points on Qwen3-0.6B and 0.8 on Qwen2.5-1.5B; gains replicated on fresh seeds and unseen problems, though standard GRPO with twice the initial learning rate came within half a point. The authors argue interventions should be validated under matched compute. ### Source excerpt arXiv:2611.13560v1 Announce Type: cross Abstract: RLVR trains language models to reason by rewarding correct final answers. A widely used algorithm, group-relative policy optimization (GRPO), gives every token of a solution the same credit, so dense process rewards have been proposed to credit intermediate steps, for example with a Monte-Carlo reward that scores each step by sampling continuations from it. Comparing such a reward with GRPO per training update, or vetting it with a gradient test on a single batch, leaves out what it costs: in our setting, the continuations alone take 91% of all generated tokens. We introduce a matched-compute protocol that gives every training intervention the same budget, measured in generated tokens and training GPU-hours, and tests it in preregistered studies with controls for side effects. Across 1,668 runs on six models, matching compute reverses the ranking. On Qwen2.5-1.5B, the Monte-Carlo reward leads standard GRPO per update, yet ends 14.3 percentage points lower at equal tokens and 7.8 points lower at equal training time, and none of four dense-reward variants outperforms GRPO even with as much training time. A per-token gradient test orders these rewards as training does but overestimates each by one to two points, as much as the gains it is meant to detect. In contrast, training on the prompts whose predicted success is nearest 1/2 improves accuracy at matched compute, by 1.7 points on Qwen3-0.6B and 0.8 on Qwen2.5-1.5B. These gains replicate on fresh seeds and unseen problems, and a simple rule, that the candidate prompts are predicted to fail more often than not, predicted where they appear in five of six preregistered tests. Both gains come early in training, and standard GRPO with twice the initial learning rate comes within half a point of them. We argue that training interventions should be validated at matched compute, which overturns some apparent gains and confirms others. ## Seeing Isn't Enough: Unlocking Visual Reasoning by Bridging the Reasoning-Access Gap with TIGER DevFeed: [Seeing Isn't Enough: Unlocking Visual Reasoning by Bridging the Reasoning-Access Gap with TIGER]() Original publisher: [Read original article]() Author: Gregory Kang Ruey Lau (Department of Computer Science, National University of Singapore, Singapore, CNRS@CREATE, Singapore), Minh Huynh Nguyen (Department of Computer Science, National University of S Published: 2026-11-10T05:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence](), [arXiv: Machine Learning]() Topics: [llm-reasoning](), [Artificial Intelligence](), [grpo](), [LLM evaluation / benchmarking]() Tags: [benchmarks](), [charts](), [cs-ai](), [cs-cv](), [cs-lg](), [dpo](), [group-relative-policy-optimization](), [grpo](), [image](), [improvements](), [input](), [language-models](), [large-language-models](), [multimodal](), [optimization](), [patching](), [policy](), [reasoning](), [rlvr](), [sft](), [spatial](), [spatial-reasoning](), [synthetic]() ### AI overview The paper argues that multimodal language models can read text in images yet struggle to apply their text-based reasoning to that visual content. TIGER targets this reasoning-access gap by finding synthetic text/image pairs where a model solves the text version but fails on the rendered image, then post-training on the image inputs. The authors report that GRPO, as well as SFT and DPO variants, improves performance on several real-world visual benchmarks despite training only on synthetic rendered-text images without multimodal annotations. TIGER also adds gains when combined with existing RLVR visual-reasoning methods. Activation analyses support the interpretation that training helps connect visual inputs to text reasoning states. ### Source excerpt arXiv:2611.10552v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) inherit strong textual reasoning capabilities from their language backbones and text pretraining, but often fail to apply the same level of competency to visual inputs. We show that this degradation cannot be attributed to only perception failures, by evaluating models on semantically-equivalent text and rendered-text image inputs: models can accurately read text in the images and recover near-text performance when provided their own transcriptions, but just cannot reason well on the images directly. We characterize this discrepancy as a modality reasoning-access gap, where models fail to route perceived visual content into existing reasoning machinery accessible with text. To bridge this, we propose TIGER (Text-to-Image Gap-targeted training for Enhanced Reasoning), which first mines synthetic modality counterfactuals where a model succeeds on a text problem but fails when provided the corresponding rendered image, and post-trains the model on the image input via Group Relative Policy Optimization (GRPO), along with SFT and DPO variants. Importantly, although trained exclusively on synthetic rendered-text images with no multimodal annotations, TIGER results in model performance improvements across diverse real-world visual benchmarks including geometric diagrams (MathVerse), multi-disciplinary sciences (EMMA, MMMU), complex charts (ChartQAPro), and embodied spatial reasoning (RoboSpatial-Home). TIGER also complements existing state-of-the-art RLVR visual reasoning methods, yielding further performance gains. Analysis involving reasoning-subspace activation and activation patching provide support indicating that TIGER helps to align visual-to-reasoning pathways toward text reasoning states. Our implementation is available at https://github.com/minhngh/Tiger. ## More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models DevFeed: [More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models]() Original publisher: [Read original article]() Author: Xurui Song, Shuo Huai, JingJing Jiang, Jiayi Kong, Jun Luo Published: 2026-11-10T05:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence]() Topics: [vlm](), [llm-reasoning](), [Chain-of-thought](), [grpo](), [Robot Navigation]() Tags: [attention](), [cs-ai](), [cs-cl](), [cs-ro](), [data](), [dataset](), [diagnostic](), [driving](), [fine-tuning](), [group-relative-policy-optimization](), [planning](), [reasoning](), [robustness](), [training](), [vision-language-model]() ### AI overview The study finds that trained vision-language driving agents' trajectory planning depends much more on ego and navigation priors than on their generated chain-of-thought (CoT). The authors create DriveMind, a nuPlan-derived driving question-answering corpus with plan-aligned CoT and separated priors and reasoning signals, then train representative agents using supervised fine-tuning and Group Relative Policy Optimization. In nuPlan evaluations, removing priors sharply lowers planning scores, while removing CoT has only minor effects; attention analysis also indicates planning focuses mainly on priors. The authors propose that training-produced reasoning may be an ancillary byproduct rather than a causal mediator, and offer a training-free probe that tests prior reliance through planning robustness to minor input perturbations. ### Source excerpt arXiv:2510.04532v3 Announce Type: replace Abstract: Vision-Language Model (VLM) driving agents promise explainable end-to-end autonomy by first producing natural-language reasoning and then predicting trajectory planning. However, whether planning is causally driven by this reasoning remains a critical but unverified assumption. To investigate this, we build DriveMind, a large-scale driving Visual Question Answering corpus with plan-aligned Chain-of-Thought (CoT), automatically generated from nuPlan. Our data generation process converts sensors and annotations into structured inputs and, crucially, separates priors from to-be-reasoned signals, enabling clean information ablations. Using DriveMind, we train representative VLM agents with Supervised Fine-Tuning and Group Relative Policy Optimization and evaluate them with nuPlan's metrics. Our results, unfortunately, indicate a consistent causal disconnect in reasoning-planning: removing ego/navigation priors causes large drops in planning scores, whereas removing CoT produces only minor changes. Attention analysis further shows that planning primarily focuses on priors rather than the CoT. Based on this evidence, we propose the Reasoning-Planning Decoupling Hypothesis, positing that the training-yielded reasoning is an ancillary byproduct rather than a causal mediator. To enable efficient diagnosis, we introduce a novel, training-free probe that measures an agent's reliance on priors by evaluating its planning robustness against minor input perturbations. In summary, we provide the community with a new dataset and a diagnostic tool to evaluate the causal fidelity of future models. ## Safe Reinforcement Learning for Multi-Turn Clinical Agents: When Dense Rewards Beat Outcome Rewards DevFeed: [Safe Reinforcement Learning for Multi-Turn Clinical Agents: When Dense Rewards Beat Outcome Rewards]() Original publisher: [Read original article]() Author: Brendan P. Murphy, Agustina Saenz Published: 2026-11-10T05:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence](), [arXiv: Machine Learning]() Topics: [rlvr](), [Reinforcement learning](), [grpo](), [llm-reasoning](), [Large Language Model](), [Machine learning]() Tags: [adaptive](), [agents](), [cs-ai](), [cs-cl](), [cs-lg](), [diagnostic](), [diagnostics](), [evaluation](), [experiments](), [exploitation](), [exploration](), [failures](), [gathering](), [group-relative-policy-optimization](), [information](), [language-model](), [optimization](), [policy](), [process](), [query](), [question](), [reinforcement-learning](), [sequential-decision]() ### AI overview Across three controlled clinical tasks--medication refill triage, acute diagnostic work-up, and dosing titration--outcome-only reinforcement learning often raised aggregate reward while encouraging premature commitment, one-question collapse, or failures on rare high-risk cases. In diagnostic work, dense credit for uncertainty-reducing questions improved held-out value and moderate-case support across three seeds, without the high-risk seed collapse seen with outcome-only training. Separating exploration from exploitation increased high-risk support at its best checkpoint but led to excessive information gathering in low-acuity cases. Process-credit strength depended on the task; after careful supervised warmup, reinforcement learning trimmed questions and degraded high-risk performance, and it did not reliably fix rare gaps inherited from biased training data. The findings indicate that reliable multi-turn training and evaluation must account for the decision procedure, not only verified final outcomes. ### Source excerpt arXiv:2611.12728v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards is increasingly used to improve language model reasoning, yet its effects on sequential decision procedures remain poorly understood. We study this problem in clinical multi-turn agents that must decide what information to request, when to stop gathering evidence, and which action to take. We introduce three controlled environments: medication refill triage, acute diagnostic work-up, and dosing titration, each with hidden clinical states, query costs, exact adaptive oracles, and deterministic outcome verifiers. Across these tasks, outcome-only group relative policy optimization often improves aggregate reward while producing undesirable procedures, including premature commitment, one-question collapse, and failures on rare high-risk cases. In diagnostics, assigning dense credit to questions that reduce uncertainty improves held-out value and moderate-case support across three training seeds relative to outcome-only reinforcement learning, without the high-risk seed collapse observed under outcome-only training. Separating exploration from exploitation further increases high-risk support at its best checkpoint, but at the cost of excessive information gathering on low-acuity cases. Additional experiments show that the optimal strength of process credit is task-dependent; that after a supervised warmup that already gathers carefully, reinforcement learning raises aggregate reward by trimming questions while degrading high-risk performance; and that reinforcement learning does not reliably repair rare behavior gaps inherited from biased supervised data. Verifiable final outcomes are therefore insufficient for training reliable multi-turn agents: reward design must account for the procedure used to reach an answer, and evaluation must distinguish adaptive information gathering from reward-exploiting shortcuts. ## PhyCAD: Physics-Aware Text-to-CAD Generation via Hierarchical Reinforcement Learning DevFeed: [PhyCAD: Physics-Aware Text-to-CAD Generation via Hierarchical Reinforcement Learning]() Original publisher: [Read original article]() Author: Yuxin Cui, Weijie Zheng, Zhengyang Li, Han Wang, Wenjie Feng, Qiyu Kang Published: 2026-11-09T05:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence]() Topics: [grpo](), [Reinforcement learning](), [Large Language Model](), [CadQuery](), [3D modeling](), [rlvr](), [Artificial Intelligence]() Tags: [aggregation](), [automated](), [cad](), [cadquery](), [cae](), [computer-aided-design](), [constraint-satisfaction](), [cs-ai](), [dataset](), [engineering](), [feedback](), [framework](), [generation](), [group-relative-policy-optimization](), [language-models](), [learning](), [optimization](), [physics](), [reinforcement-learning](), [text](), [verification]() ### AI overview PhyCAD trains language models to generate executable CadQuery programs that satisfy geometric and physical constraints, addressing the fact that geometric correctness alone does not ensure physical feasibility. Its dataset links parametric models with geometric specifications, loading conditions, and physical constraints; an automated CAD-CAE verifier combines program execution, geometric checks, and finite element analysis to provide training feedback. To keep unmet requirements from being offset by gains on other criteria, its Hierarchical Balanced Advantage for Group Relative Policy Optimization ranks candidates by verification status, then provides objective-wise peer feedback within each level. Experiments across multiple language models report improved joint geometric and physical constraint satisfaction. The supplied text is an abstract and does not provide quantitative results or implementation details. ### Source excerpt arXiv:2611.09401v1 Announce Type: new Abstract: Language-driven computer-aided design (CAD) generation enables the creation of editable parametric models from text, yet geometric correctness alone does not ensure physical feasibility. We present PhyCAD, a framework for physics-aware Text-to-CAD generation that trains language models to produce executable CadQuery programs under geometric and physical constraints. To support this setting, we construct a Text-to-CAD dataset that links parametric CAD models with geometric specifications, loading conditions, and physical constraints. We further develop an automated CAD--CAE verifier that combines program execution, geometric checks, and finite element analysis to provide feedback for model training. A key learning challenge is to prioritize joint satisfaction of geometric and physical constraints while retaining informative feedback on partial progress. A flat aggregation of heterogeneous rewards can obscure this priority, allowing improvements in individual criteria to compensate for unmet design requirements. We therefore introduce a Hierarchical Balanced Advantage for Group Relative Policy Optimization, grouping candidates into levels according to their verification status and combining between-level progress with objective-wise peer ranking within each level. This formulation preserves cross-level priority in the training signal while providing fine-grained feedback for satisfying constraints and improving feasible designs. Experiments across multiple language models show that PhyCAD consistently improves joint geometric and physical constraint satisfaction. These results demonstrate the value of verifiable engineering feedback for advancing language-driven CAD generation beyond geometric reconstruction toward physics-aware design. ## BNPO: Beta Normalization Policy Optimization DevFeed: [BNPO: Beta Normalization Policy Optimization]() Original publisher: [Read original article]() Author: Changyi Xiao, Mengdi Zhang, Yixin Cao Published: 2026-11-09T05:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence](), [arXiv: Machine Learning]() Topics: [Reinforcement learning](), [llm-reasoning](), [Machine learning](), [grpo]() Tags: [beta](), [cs-ai](), [cs-lg](), [distribution](), [gradient](), [group-relative-policy-optimization](), [grpo](), [language-models](), [large-language-models](), [learning](), [machine-learning](), [optimization](), [policy](), [reasoning](), [reinforcement-learning](), [reward](), [stable](), [static](), [tasks](), [theoretical](), [training](), [variance]() ### AI overview BNPO adaptively normalizes binary rewards with a Beta distribution whose parameters update as the policy changes. The authors say this produces more accurate, lower-variance gradient estimates and steadier training than ignoring normalization or using a static scheme. Their theoretical analysis describes BNPO as reducing variance and generalizing REINFORCE and GRPO under binary rewards; an advantage decomposition extends it to more complex reward settings. Experiments reportedly show state-of-the-art performance among policy optimization methods on reasoning tasks. This overview is based on the supplied abstract. ### Source excerpt arXiv:2506.02864v2 Announce Type: replace Abstract: Reinforcement learning with rule-based, binary-valued reward functions has been shown to significantly enhance the reasoning capabilities of large language models. These models typically rely on REINFORCE-based policy optimization methods, such as REINFORCE with baseline and group relative policy optimization (GRPO). However, a key limitation is that existing methods either ignore reward normalization or use static normalization schemes, which fail to adapt to the evolving policy during training. This mismatch can lead to unstable gradient estimates and hinder training stability. To address this issue, we propose Beta Normalization Policy Optimization (BNPO), a novel method that adaptively normalizes rewards using a Beta distribution with dynamically updated parameters. By aligning normalization with the changing policy distribution, BNPO enables more accurate, lower-variance gradient estimation and promotes stable training dynamics. We provide theoretical analysis showing that BNPO reduces variance and generalizes both REINFORCE and GRPO under binary-valued rewards. In addition, we introduce an advantage decomposition mechanism to extend BNPO to more complex reward settings. Experimental results demonstrate that BNPO achieves state-of-the-art performance among policy optimization methods on reasoning tasks. ## TSFM Meets LLM: Context as Covariates DevFeed: [TSFM Meets LLM: Context as Covariates]() Original publisher: [Read original article]() Author: Hemanth Ram, Sai Shankar Narasimhan, Shubhankar Agarwal, Sandeep Chinchali Published: 2026-11-05T05:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence]() Topics: [Large Language Model](), [AI Foundation Models](), [Fine-tuning](), [grpo](), [llm-reasoning](), [LLM evaluation / benchmarking](), [qwen]() Tags: [benchmarks](), [context](), [cs-ai](), [cs-lg](), [datasets](), [economics](), [finance](), [fine-tuning](), [forecast](), [forecasting](), [foundation](), [foundation-models](), [framework](), [gpt](), [group-relative-policy-optimization](), [grpo](), [history](), [llm](), [llms](), [models](), [multimodal](), [news]() ### AI overview Context as Covariates (CoCo) combines textual reasoning with time-series forecasting by having an LLM convert context into numerical forecast and confidence covariates that guide a TSFM. The authors also describe generating synthetic paired data by annotating real-world time series with LLMs. Training proceeds in two stages: align the LLM with Group Relative Policy Optimization using a reward tied to forecasting gains from its covariates, then fine-tune the TSFM on those covariates. The abstract reports generalization to real benchmarks in finance, economics, weather, traffic, and security. With Qwen3-4B-Instruct-2507 and Chronos2, CoCo achieves up to 12% MAE improvement over the cited state-of-the-art TSFMs and LLM forecasters, including GPT-4o. ### Source excerpt arXiv:2611.05515v1 Announce Type: cross Abstract: Textual information is ubiquitous in real-world time series datasets, appearing as news, reports, and annotations that hold key predictive signals not deducible from numerical history alone. Large Language Models (LLMs) interpret text but lack numerical precision, while Time Series Foundation Models (TSFMs) are quantitatively accurate but cannot reason about the effects of text. Moreover, the lack of large-scale paired multimodal datasets limits progress in bridging this gap. In this work, we address this challenge by providing a scalable approach for generating realistic multimodal datasets by annotating real-world time series with LLMs. Building on this, we propose Context as Covariates (CoCo), a forecasting framework in which an LLM distills its textual reasoning into numerically grounded forecast and confidence covariates that guide a TSFM backbone, thereby exploiting the strengths of both LLMs and TSFMs. We align the two models in stages, first fine-tuning the LLM via Group Relative Policy Optimization (GRPO) with a reward tied to covariate-induced improvement in forecasting performance, and then fine-tuning the TSFM on covariates generated by the aligned LLM. Our experiments show that models trained on our synthetic multimodal corpus generalize to real benchmarks spanning finance, economics, weather, traffic, and security. Notably, CoCo with Qwen3-4B-Instruct-2507 as the LLM and Chronos2 as the TSFM achieves up to 12% MAE improvement over state-of-the-art TSFMs and LLM forecasters, such as GPT-4o. Code, checkpoints, and the evaluation datasets are publicly available. ## Oumi's Frontier AI Engineer now builds AI agents and retrains the models behind them DevFeed: [Oumi's Frontier AI Engineer now builds AI agents and retrains the models behind them]() Original publisher: [Read original article]() Author: Duncan Riley Published: 2026-11-04T14:00:32Z Content type: news Language: en Sources: [SiliconANGLE]() Topics: [AI Development](), [post-training](), [Reinforcement learning](), [AI Infrastructure](), [Artificial Intelligence]() Tags: [agent](), [agents](), [ai](), [ai-agents](), [ai-engineer](), [artificial-intelligence](), [bring-your-own-harness](), [build](), [build-ai-agents](), [builds](), [customers](), [direct-preference-optimization](), [enterprise](), [enterprise-artificial-intelligence](), [faie](), [frontier](), [frontier-ai](), [frontier-ai-engineer](), [gemini](), [google](), [group-relative-policy-optimization](), [harness-engineering](), [manos-koukoumidis](), [model-training](), [models](), [news](), [obvious-management-services](), [obvious-ventures](), [oumi](), [oumi-pbc](), [platform](), [prompt](), [prompt-engineering](), [reinforcement-learning](), [supervised-fine-tuning](), [the-latest](), [venrock](), [venrock-management]() ### AI overview Oumi's Frontier AI Engineer (FAiE) now helps teams build agents from plain-language objectives, define their tools and environments, evaluate performance, and train the models behind them. Failures in production can become evaluation sets and training data for later runs. With Bring Your Own Harness, companies can keep existing orchestration, tools, and memory while connecting agents to Oumi for evaluation and training. The platform adds direct preference optimization alongside supervised fine-tuning, and supports multistage runs that combine methods. Group relative policy optimization, which evaluates models by task outcomes, is available in private preview. Oumi says this approach can address model weaknesses that prompt and harness changes alone may not resolve; the article does not provide measured performance or cost comparisons. ### Source excerpt Enterprise artificial intelligence platform company Oumi PBC today extended its Frontier AI Engineer agent so it can build AI agents for customers and retrain the models those agents run on. The release goes after a problem that tends to surface once agents leave the demo stage. A failing agent might have a bad prompt, a [...] The post Oumi's Frontier AI Engineer now builds AI agents and retrains the models behind them appeared first on SiliconANGLE. ## BrainwashedLLM: Jailbreaking via Automated Adversarial Chain of Thoughts Discovery DevFeed: [BrainwashedLLM: Jailbreaking via Automated Adversarial Chain of Thoughts Discovery]() Original publisher: [Read original article]() Author: Masoud Poorghaffar Aghdam, G\\"oktu\\u{g} Serdar Y{\\i}ld{\\i}r{\\i}m, Sinem Sav Published: 2026-11-04T05:00:00Z Content type: article Language: en Sources: [arXiv: Cryptography and Security]() Topics: [LLM security](), [Adversarial attacks](), [llm-reasoning](), [grpo](), [Chain-of-thought](), [Artificial Intelligence]() Tags: [adversarial](), [alignment](), [attack-surface](), [automated](), [chain-of-thought](), [construction](), [context](), [cs-cr](), [discovery](), [exposed](), [feedback](), [group-relative-policy-optimization](), [grpo](), [imperative](), [input](), [intermediate](), [jailbreaks](), [large](), [learning](), [malicious](), [model]() ### AI overview BrainwashedLLM uses GRPO with a multi-objective, rule-based reward to train a small white-box model to produce adversarial reasoning traces; the reward evaluates only that model's own trace. Injected as input prefixes, the traces transferred zero-shot to four larger black-box reasoning models and steered them toward harmful outputs without target queries, gradient access, prompt optimization, or target-output feedback during attack construction. The reported ASR@5 ranged from 0.74 to 0.98, within 16 points of query-based baselines. The authors argue that exposed reasoning traces create a safety vulnerability and suggest isolating the reasoning buffer as a defense. ### Source excerpt arXiv:2611.04717v1 Announce Type: new Abstract: Large Reasoning Models (LRMs) achieve strong performance on complex tasks by generating explicit Chain-of-Thought (CoT) reasoning, but the safety implications of this intermediate reasoning surface remain underexplored. We demonstrate that the reasoning trace itself is a powerful attack vector. Safety alignment in large models is substantially less robust to malicious reasoning embedded in the context than to direct imperative prompts. We introduce BrainwashedLLM, a weak-to-strong adversarial framework that automates the discovery of reasoning-based jailbreaks. First, we use Group Relative Policy Optimization (GRPO), a Reinforcement Learning (RL) algorithm, with a multi-objective rule-based reward to fine-tune a small, white-box source model to generate adversarial reasoning traces. Crucially, the reward is computed entirely over the source's own reasoning trace. Second, we show that these traces transfer in a zero-shot manner. When traces are injected as input prefixes, they steer larger black-box LRMs to continue the adversarial reasoning and produce harmful outputs, without requiring prompt optimization, gradient access, or any form of output feedback from the target model. Across four state-of-the-art LRMs that span distinct architectures, BrainwashedLLM achieves ASR@5 between 0.74 and 0.98 across targets, within 16 points of query-based baselines, without querying the target during attack construction. To our knowledge, this is the first framework to use RL to directly optimize the CoT of LRMs as an attack surface without any target feedback, and the first to demonstrate zero-shot weak-to-strong transfer of reasoning traces. Our results suggest that exposed CoT is a vulnerability in LRMs and motivate architectural defenses that isolate the reasoning buffer. Warning: This paper contains potentially offensive and harmful text. ## JHiMA-WebRL: A Joint Hierarchical Multi-Agent Reinforcement Learning Framework for Long-Horizon Web Tasks DevFeed: [JHiMA-WebRL: A Joint Hierarchical Multi-Agent Reinforcement Learning Framework for Long-Horizon Web Tasks]() Original publisher: [Read original article]() Author: Shanka Subhra Mondal, Jiayun Wang Published: 2026-11-04T05:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence]() Topics: [Multi Agent Systems](), [Reinforcement learning](), [hierarchical agent systems](), [grpo](), [computer-use](), [Multi-step AI processes](), [AI Agent](), [Fine-tuning](), [llm-reasoning]() Tags: [actions](), [actor](), [agent](), [agents](), [cs-ai](), [environment](), [execution](), [framework](), [group-relative-policy-optimization](), [grpo](), [gui](), [horizon](), [interaction](), [interactive](), [interfaces](), [learning](), [model](), [multi-agent](), [reinforcement-learning](), [tasks](), [web]() ### AI overview JHiMA-WebRL jointly trains a task decomposer and a subtask-conditioned actor to handle long-horizon web tasks. A teacher model first provides structured subtask supervision from multimodal GUI trajectories; both components are fine-tuned on these annotations, then jointly optimized in WebArena using an extension of GRPO with hierarchical rollout grouping and role-specific advantages. On WebArena-Lite-v2, the 7B-plus-7B JHiMA-WebRL-14B reports 37.2% average success, versus 24% for a ScaleCUA-32B agent fine-tuned with RL on the same training tasks and evaluated under the same protocol. Ablations report that updating the decomposer during RL outperforms updating the actor alone. The abstract presents joint role factorization as a potentially competitive alternative to scaling monolithic agents; it does not establish results beyond the stated benchmark and protocol. ### Source excerpt arXiv:2611.05263v1 Announce Type: new Abstract: Multimodal web agents have shown promising capabilities for interacting with websites through graphical user interfaces (GUIs), yet realistic web tasks often require long-horizon, multi-step planning that remains challenging for monolithic policies. Such agents must jointly infer high-level plans, navigate through intermediate states, track progress over extended interaction histories, and execute visually grounded actions. Although prior work has explored modular multi-agent systems with a separate planner, the planner is typically not updated jointly with the actor. We investigate whether explicitly separating and jointly training planning and execution provides a stronger inductive bias for long-horizon web navigation. We introduce JHiMA-WebRL, a joint hierarchical multi-agent reinforcement learning framework for web agents that couples a task decomposer with an actor. To train the decomposer, we use a strong teacher model to construct structured subtask supervision from multimodal GUI trajectories, and use these annotations to fine-tune both the decomposer and the subtask-conditioned actor. We then extend Group Relative Policy Optimization (GRPO) to jointly optimize the decomposer and actor in the interactive WebArena environment through hierarchical rollout grouping and role-specific advantages. On WebArena-Lite-v2, JHiMA-WebRL-14B, consisting of a 7B actor and a 7B decomposer, achieves an average success rate of 37.2%, compared with 24% for a ScaleCUA-32B agent that we fine-tune with RL on the same training tasks and evaluate under the same protocol. Ablations show that jointly updating the decomposer during RL outperforms updating the actor alone. These results suggest that jointly learned role factorization with smaller models can be a competitive alternative to scaling monolithic agents for long-horizon web tasks. ## One Offset, Two Directions: Baseline Adjustment via Subgroup Offsets DevFeed: [One Offset, Two Directions: Baseline Adjustment via Subgroup Offsets]() Original publisher: [Read original article]() Author: Jiaying Zhou, Jiachen Liu, Qingchao Chen Published: 2026-11-03T05:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence]() Topics: [grpo]() Tags: [adjustment](), [alignment](), [coverage](), [cs-ai](), [distributions](), [global](), [group-relative-policy-optimization](), [lambda](), [math](), [optimization](), [prompt](), [quality](), [qwen3](), [reasoning](), [reward](), [strategies](), [strategy]() ### AI overview The paper adjusts GRPO's shared within-prompt reward baseline using each strategy subgroup's mean deviation from the global mean. A signed coefficient λ sets the direction: Alignment (λ > 0) adds the subgroup offset, emphasizing a rollout's standing within its subgroup; Contrast (λ < 0) subtracts it, emphasizing the subgroup's relative quality. The change affects the baseline alone, leaving the reward signal and optimization objective unchanged. On Qwen3-8B-Base and Olmo3-7B-Base across four competition-level math benchmarks, both directions reportedly produced modest, consistent multi-sample coverage gains over a λ = 0 anchor with the same strategy-conditioned sampling; Contrast was stronger. The evidence provided is the abstract. ### Source excerpt arXiv:2611.00782v1 Announce Type: new Abstract: Group Relative Policy Optimization (GRPO) estimates advantages against the within-prompt mean reward, a baseline shared across the group. In practice, rollouts within a prompt are not always homogeneous, differing in their solution strategies and reasoning trajectories, along which they may group into subgroups. We further find that these heterogeneous subgroups may exhibit distinct reward distributions. The global baseline, however, collapses them onto a single credit axis, where the credit reflects neither a rollout's standing within its subgroup nor the subgroup's standing relative to the rest. We therefore study how to adjust the baseline when the rollouts of a prompt fall into heterogeneous subgroups. We define the \emph{subgroup offset} as the deviation of a subgroup mean from the global mean, and shift the baseline along it by a single signed coefficient $\lambda$. The sign of $\lambda$ yields two opposing directions: \emph{Alignment} ($\lambda{>}0$) adds the offset to the baseline, drawing credit toward a rollout's relative position within its subgroup; \emph{Contrast} ($\lambda{<}0$) subtracts it, drawing a rollout's credit toward the relative quality of the subgroup it belongs to. The adjustment acts on the baseline alone, leaving the reward signal and the optimization objective unchanged. In this work, we induce subgroups from strategies, grouping rollouts that share a strategy into the same subgroup. We evaluate both directions on Qwen3-8B-Base and Olmo3-7B-Base across four competition-level math benchmarks, against a $\lambda{=}0$ anchor that shares the same strategy-conditioned sampling and differs only in the baseline. Against this anchor each direction yields modest but consistent gains in multi-sample coverage, with Contrast the stronger of the two. ## UI-UG 2: Advancing Mobile UI Understanding and Generation with Fine-Grained Supervision and Reinforcement Learning DevFeed: [UI-UG 2: Advancing Mobile UI Understanding and Generation with Fine-Grained Supervision and Reinforcement Learning]() Original publisher: [Read original article]() Author: Hao Yang, Shuoyang Liu, Xiaoyu Lin, Zhou Fang, Maji Huang, Zhaosong Huang, Ruichao Mao, Teng Guo, Hai Rao Published: 2026-11-03T05:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence]() Topics: [grpo](), [Reinforcement learning](), [Large Language Model](), [Transformers]() Tags: [cs-ai](), [cs-cv](), [cs-hc](), [domain-specific-language](), [generation](), [group-relative-policy-optimization](), [gui](), [large-language-model](), [llm](), [mobile](), [ocr](), [playwright](), [reinforcement-learning](), [supervision](), [taxonomy](), [ui]() ### AI overview UI-UG 2 is a 4B-parameter multimodal model for mobile interface understanding and structured generation. Its understanding data combines four named sources under an 80-category, two-level taxonomy; generation uses more than 5,000 requirement, screenshot, and JSON tuples expressed through a domain-specific language. For generation, it replaces the offline DPO-style optimization described for UI-UG with online GRPO: sampled outputs are rendered through a Playwright-based server pipeline and assessed using LLM-as-a-Judge and contrastive preference ranking. The abstract reports the strongest referring and dense-grounding results among evaluated models on its 80-category benchmark and the best composite generation score among compared models at 4B scale. It also reports that fine-grained perception supervision improved generation under a fixed training budget, suggesting transfer between the tasks. ### Source excerpt arXiv:2611.00926v1 Announce Type: cross Abstract: Mobile UI understanding and generation require both fine-grained component perception and reliable structured synthesis. Inspired by the unified direction introduced by UI-UG, we present UI-UG 2, a compact 4B-parameter multimodal large language model (MLLM) that advances mobile UI Understanding and Generation with broader task coverage, a front-end-aligned taxonomy, and online render-aware reinforcement learning. UI-UG 2 constructs a multi-source understanding corpus from RICO, VINS, Android in the Wild, and Android Control, annotated with an 80-category two-level UI taxonomy aligned with front-end component semantics, and pairs it with over 5,000 requirement-screenshot-JSON tuples for progressive structured UI generation through a domain-specific language (DSL). For understanding, UI-UG 2 introduces decomposed rewards for category prediction, OCR fidelity, color recognition, dense grounding, and GUI grounding. Unlike the offline Direct Preference Optimization (DPO)-style generation optimization in UI-UG, UI-UG 2 optimizes generation through end-to-end online Group Relative Policy Optimization (GRPO): sampled DSL outputs are rendered with a Playwright-based server-side pipeline and evaluated using LLM-as-a-Judge and contrastive preference ranking. Experiments show that UI-UG 2 achieves the strongest referring and dense-grounding results among the evaluated models on our 80-category benchmark, and achieves the best composite generation score among compared models at a compact scale (4B). More importantly, we observe a consistent understanding-to-generation transfer: adding fine-grained UI perception supervision improves generation quality under a fixed training budget, suggesting that component-level semantics provide a shared representational foundation for both tasks. ## Dataset Mismatch Matters in Group Relative Policy Optimization for Reinforcement Learning from Verifiable Rewards DevFeed: [Dataset Mismatch Matters in Group Relative Policy Optimization for Reinforcement Learning from Verifiable Rewards]() Original publisher: [Read original article]() Author: Zhen-Yu Zhang, Jiandong Zhang, Huaxiu Yao, Gang Niu, Masashi Sugiyama Published: 2026-11-03T05:00:00Z Content type: article Language: en Sources: [arXiv: Machine Learning]() Topics: [grpo](), [rlvr](), [Reinforcement learning](), [llm-reasoning](), [Machine learning]() Tags: [cs-lg](), [dataset](), [distribution](), [domain](), [efficiency](), [feedback](), [group-relative-policy-optimization](), [math](), [math-reasoning](), [model](), [optimization](), [performance](), [privacy](), [reinforcement-learning](), [rlvr](), [sampling](), [source](), [target](), [test](), [training](), [training-dataset]() ### AI overview Learning-to-reweight rollout (L2RR) adapts GRPO to source-target dataset mismatch by using limited aggregated performance feedback from a small held-out target set, without sharing target prompts or rollout traces. It learns a discrete sampling distribution over source prompts through a bi-level objective that maximizes held-out target accuracy while penalizing rollouts with little incremental learning value. The abstract reports improved target-domain performance and rollout efficiency across six math reasoning benchmarks and three model scales. Results are limited to those reported experiments. ### Source excerpt arXiv:2611.00586v1 Announce Type: new Abstract: Recently, reinforcement learning from verifiable rewards (RLVR) is practically very important, where the leading approach is group relative policy optimization (GRPO) that typically faces a trade-off between rollout efficiency and downstream performance. To improve efficiency while preserving performance, existing selective rollout methods mainly exploit only source-side training dynamics to optimize the discrete distribution over source prompts. However, in practice, the source training dataset is rarely identically distributed to the target test dataset. When such a mismatch exists, due to legal or privacy constraints, it is usually not acceptable to share target prompts themselves with the RLVR trainer. Nevertheless, it is still possible for the target data owner to provide limited aggregated feedback from a small held-out target set without revealing target prompts or rollout traces. In this paper, we propose learning-to-reweight rollout (L2RR) that explicitly incorporates target-side performance feedback into GRPO. Specifically, L2RR learns a discrete sampling distribution over source prompts by optimizing a bi-level objective that maximizes accuracy on a held-out target dataset, while simultaneously penalizing rollouts that provide little incremental learning information. Empirical studies on six math reasoning benchmarks and three model scales show that L2RR improves target-domain performance and increases rollout efficiency. ## Think with Skills: Pseudocode Planning Before Execution for Advanced Reasoners DevFeed: [Think with Skills: Pseudocode Planning Before Execution for Advanced Reasoners]() Original publisher: [Read original article]() Author: Boshen Zhou, Boxuan Yue, Xiang Li, Shanchao Yang Published: 2026-11-03T05:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence]() Topics: [llm-reasoning](), [grpo](), [Fine-tuning]() Tags: [agent](), [agents](), [complex](), [cs-ai](), [dependencies](), [execution](), [fine-tuning](), [group-relative-policy-optimization](), [intermediate](), [knowledge](), [language-model](), [operations](), [optimization](), [order](), [pairs](), [performance](), [planning](), [policy](), [procedural](), [pseudocode](), [skills]() ### AI overview Think with Skills improves task success by training agents to convert relevant skill descriptions into task specific pseudocode plans before acting. Plans make operations, conditions, intermediate results, and dependencies explicit, helping address implicit requirements and coordination across multiple skills. Training pairs tasks and skills with plans and execution trajectories, uses supervised fine tuning to initialize the behavior, then applies Group Relative Policy Optimization to optimize end to end task performance. On SkillsBench, reported task success increased from 35.0% to 62.0%. The supplied text is an abstract, so it does not detail evaluation setup or limitations. ### Source excerpt arXiv:2611.02830v1 Announce Type: new Abstract: Language model agents often struggle with complex tasks even when equipped with expert-written skills. Such skills encode declarative knowledge, whereas successful execution requires procedural knowledge tailored to the task at hand. Natural language skills may leave conditions and dependencies implicit, and tasks involving multiple skills further require agents to coordinate intermediate results and execution order. We introduce Think with Skills, a method that trains reasoning models to turn relevant skill descriptions into task-specific pseudocode plans before execution. The pseudocode explicitly represents operations, conditions, intermediate results, and dependencies, providing a structured bridge from skill descriptions to agent actions. We construct training data that pairs tasks and skills with pseudocode plans and execution trajectories, use supervised fine tuning to initialize this behavior, and further optimize end-to-end task performance with Group Relative Policy Optimization. On SkillsBench, Think with Skills raises task success from 35.0% to 62.0%. These results show that proceduralizing expert knowledge before execution substantially improves skill use.