# grpo

Group Relative Policy Optimization (GRPO) is an algorithm for training large language models with reinforcement learning that uses relative rewards from multiple sampled responses instead of a value-function approximation.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining

DevFeed: [ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining](<https://devfeed.tech/articles/roam-grpo-enforcing-reasoning-diversity-in-grpo-via-routed-off-mode-approach-mining-145610.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25926>)

Author: Jeff Duan, Matt Thomson

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [grpo](<https://devfeed.tech/topics/grpo.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [rlvr](<https://devfeed.tech/topics/rlvr.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [creativity](<https://devfeed.tech/tags/creativity.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [diversity](<https://devfeed.tech/tags/diversity.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [math](<https://devfeed.tech/tags/math.md>), [mining](<https://devfeed.tech/tags/mining.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [problems](<https://devfeed.tech/tags/problems.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [rlvr](<https://devfeed.tech/tags/rlvr.md>), [sft](<https://devfeed.tech/tags/sft.md>), [strategies](<https://devfeed.tech/tags/strategies.md>), [supervised-fine-tuning](<https://devfeed.tech/tags/supervised-fine-tuning.md>), [well](<https://devfeed.tech/tags/well.md>)

### AI overview

ROAM-GRPO aims to counter reasoning-mode collapse in GRPO by routing well-solved problems to an "off-mode" track. The model generates distinct solution strategies, an LLM judge checks their logical validity and distinctness, and accepted strategies are self-distilled into the policy. In the reported experiment, training Qwen3.5-2b on 1,000 MATH level-5 problems increased peak pass@16 by 4.7% and correct embedding cosine distance by approximately 1.5-2x on a difficult evaluation set. The supplied text does not detail the evaluation setup or broader limitations.

### Source excerpt

arXiv:2611.25926v1 Announce Type: new Abstract: Supervised fine-tuning (SFT) paired with Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant recipe for post-training LLMs on complex reasoning tasks, but has been shown to collapse the policy into a low-diversity mode. Repeated LLM outputs are nearly identical, and pass@k accuracy converges or degrades as a result. We propose ROAM-GRPO (Routed Off-mode Approach Mining for GRPO), an augmentation to the GRPO algorithm which incentivizes exploration of distinct reasoning strategies and aims to increase output creativity during evaluation. ROAM-GRPO detects well-solved problems during training and transfers them to an "off-mode" track. The model is prompted to produce distinct solution strategies to such problems, which are verified for logical validity and distinctness by an LLM judge and subsequently self-distilled into the policy. Training Qwen3.5-2b on a 1,000-problem subset of the MATH lvl5 dataset, ROAM-GRPO implementation increases peak pass@16 by 4.7% and correct embedding cosine distance by approximately 1.5-2x on a difficult evaluation set.

## When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoning

DevFeed: [When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoning](<https://devfeed.tech/articles/when-and-how-much-to-imagine-adaptive-test-time-scaling-with-world-models-for-visual-spatial-reasoning-145315.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.08236>)

Author: Shoubin Yu, Yue Zhang, Zun Wang, Jaehong Yoon, Huaxiu Yao, Mingyu Ding, Mohit Bansal

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [World models](<https://devfeed.tech/topics/world-models.md>), [grpo](<https://devfeed.tech/topics/grpo.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [imagination](<https://devfeed.tech/tags/imagination.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [spatial-reasoning](<https://devfeed.tech/tags/spatial-reasoning.md>), [world-models](<https://devfeed.tech/tags/world-models.md>)

### AI overview

Visual imagination can help spatial reasoning when a scene must be assessed from an unseen viewpoint, but unnecessary or excessive imagined evidence can waste computation or reduce accuracy. The paper introduces AVIC, which checks whether current visual evidence is sufficient before selectively invoking and scaling a world model, and AVIC-R, which learns that policy using GRPO with answer-correctness rewards and imagination-cost penalties, without labels for when or how much to imagine. On SAT, MMSI, and R2R, selective control reportedly matches or outperforms fixed imagination strategies with fewer world-model calls and language tokens. AVIC-R also reportedly surpasses GPT-4o and GPT-4.1 while using the world model less often. The supplied abstract does not provide quantitative results or benchmark-specific details.

### Source excerpt

arXiv:2602.08236v3 Announce Type: replace-cross Abstract: Despite rapid progress in MLLMs, visual spatial reasoning remains unreliable when correct answers depend on how a scene would appear under unseen or alternative viewpoints. Recent work addresses this by augmenting reasoning with world models for visual imagination, but questions such as when imagination is actually necessary, how much of it is beneficial, and when it becomes harmful, remain poorly understood. In practice, indiscriminate imagination can increase computation and even degrade performance by introducing misleading evidence. In this work, we present an in-depth analysis of test-time visual imagination as a controllable resource for spatial reasoning. We first study when static visual evidence is sufficient, when imagination improves reasoning, and how excessive or unnecessary imagination affects accuracy and efficiency. To support this analysis, we then introduce AVIC, an adaptive test-time framework with world models that explicitly reasons about the sufficiency of current visual evidence before selectively invoking and scaling visual imagination. Finally, to further learn this gating and planning behavior without any annotation of when and how much to imagine, we introduce AVIC-R, which trains the policy via GRPO from QA-correctness rewards and penalties by imagination cost. Across spatial reasoning benchmarks (SAT, MMSI) and an embodied navigation benchmark (R2R), our results reveal clear scenarios where imagination is critical, marginal, or detrimental, and show that selective control can match or outperform fixed imagination strategies with substantially fewer world-model calls and language tokens. Our AVIC-R surpasses strong proprietary baselines including GPT-4o and GPT-4.1 while invoking the world model less often. Overall, our findings highlight the importance of analyzing and controlling test-time imagination for efficient and reliable spatial reasoning.

## Calibrated Decision Rewards: Training Language Models with a Judge That Knows How Sure It Is

DevFeed: [Calibrated Decision Rewards: Training Language Models with a Judge That Knows How Sure It Is](<https://devfeed.tech/articles/calibrated-decision-rewards-training-language-models-with-a-judge-that-knows-how-sure-it-is-145472.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25135>)

Author: Rudransh Agnihotri, Manasvi Kapoor

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [grpo](<https://devfeed.tech/topics/grpo.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>)

Tags: [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [decision-model](<https://devfeed.tech/tags/decision-model.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [probability](<https://devfeed.tech/tags/probability.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward-model](<https://devfeed.tech/tags/reward-model.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

RLCDR uses an LLM decision model to assign calibrated probability distributions to typed questions about a response, such as whether its final answer is correct. The probabilities are read in one forward pass, adjusted for option order, and calibrated using about a hundred labels. In GRPO training of Qwen3-1.7B on MATH, this reference-free reward raised MATH500 accuracy from 72.2% to 76.8%, close to the answer-key reward's 76.9%; a random reward gained 0.65 points. On the hardest level, accuracy rose from 44.8% to 51.7%. The authors report lower calibration error than a softmax judge using the same 27B backbone, and better first-error detection than a dedicated 7B process reward model. A limitation appears on problems the base model mostly fails: RLCDR still improved MATH500 by 1.9 points but lagged the answer-key reward as its probabilities became overconfident on that distribution.

### Source excerpt

arXiv:2611.25135v1 Announce Type: new Abstract: Reinforcement learning (RL) can only teach a language model what its reward can recognize. Answer checkers are exact but need a reference answer; learned reward models and LLM judges work everywhere, but their scores are not probabilities. We study RLCDR (Reinforcement Learning from Calibrated Decision Rewards). An LLM converted into a decision model answers a typed question about a response ("is the final answer correct?", "which step is the first error?") with a probability distribution, read in one forward pass, debiased over option orders and calibrated with about a hundred labels. Training Qwen3-1.7B with GRPO on MATH, this reference-free reward raises MATH500 accuracy from 72.2% to 76.8% (+4.65 points, p < 1e-4), matching training on the answer key (76.9%), while a random reward gains only +0.65. The gain grows with difficulty, from 44.8% to 51.7% on the hardest level. On the policy's own changing samples the reward stays calibrated: its expected calibration error is half that of a softmax judge on the same 27B backbone (0.074 vs. 0.141), and it shows no sign of reward hacking. Without step training, the same model finds the first wrong step better than a dedicated 7B process reward model (ProcessBench F1 0.772 vs. 0.722, 20/20 splits). We also map where the approach stops being enough: trained only on problems the base model mostly fails, the reference-free reward still improves the policy (+1.9 MATH500 points, p = 0.03) but trails the answer key, because its probabilities become overconfident on that distribution. We state six hypotheses in advance and test each with a control designed to falsify it.

## OceanCoder: A Verification-Guided Large Language Model for Automated Simulation Modelling in Ocean Engineering

DevFeed: [OceanCoder: A Verification-Guided Large Language Model for Automated Simulation Modelling in Ocean Engineering](<https://devfeed.tech/articles/oceancoder-a-verification-guided-large-language-model-for-automated-simulation-modelling-in-ocean-engineering-145165.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26333>)

Author: Can Xue, Changhong He, Ziyi Sun, Anliang Ge, Shimin Yu, Bingchen Liang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Climate / Weather / Ocean Modeling](<https://devfeed.tech/topics/climate-weather-ocean-modeling.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [modelling](<https://devfeed.tech/tags/modelling.md>), [programming](<https://devfeed.tech/tags/programming.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [simulation](<https://devfeed.tech/tags/simulation.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

OceanCoder uses engineering verification as a training signal for automated ocean simulation modelling: successful code execution alone does not show that a model or its results meet engineering requirements. Built on Qwen3-8B, it is trained on 12,000 verified examples across 12 task categories, then guided by graded rewards from checks of inputs and constraints, completion of required analyses, and plausibility of computed outputs. The paper reports Pass@1 rising from 0% for the base model to 46.77% after supervised fine-tuning and 51.43% after group relative policy optimisation, outperforming the compared base models and closed-source system. Its benchmark includes out-of-distribution tasks and real-world engineering cases; the supplied text does not specify the comparison system or broader deployment limitations.

### Source excerpt

arXiv:2611.26333v1 Announce Type: new Abstract: Numerical simulation is important in ocean engineering, but model construction requires expert knowledge. Large language models support automated modelling, but code syntax and successful execution alone do not establish whether generated models and results meet engineering requirements. We present OceanCoder, a Qwen3-8B-based large language model for automated simulation modelling in ocean engineering. To address specialised programming requirements and scarce training data, we develop an extensible data construction pipeline producing 12,000 verified supervised fine-tuning (SFT) examples across 12 engineering task categories. A three-layer verification framework checks model inputs against engineering parameters and task constraints, verifies completion of required simulation analyses, and assesses the engineering plausibility of computed outputs. Its outcomes provide graded rewards for group relative policy optimisation (GRPO), guiding the model towards engineering task completion. We also construct the first benchmark for automated modelling in ocean engineering, covering out-of-distribution tasks and real-world engineering cases. Experiments and ablation studies demonstrate the effectiveness of the data pipeline, verification framework and reward design. OceanCoder outperforms both base models and the closed-source comparison, with Pass@1 increasing from 0% for the base model to 46.77% after SFT and 51.43% after GRPO. These results support engineering verification as a training signal for automated simulation modelling.

## RL Can Only Reinforce What It Sees Succeed

DevFeed: [RL Can Only Reinforce What It Sees Succeed](<https://devfeed.tech/articles/rl-can-only-reinforce-what-it-sees-succeed-145138.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26084>)

Author: Ruben Sanchez-Corcuera, Unai Zulaika, Aritz Bilbao-Jayo, Miguel Fernandez-de-Retana

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [grpo](<https://devfeed.tech/topics/grpo.md>)

Tags: [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [puzzle](<https://devfeed.tech/tags/puzzle.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [rl](<https://devfeed.tech/tags/rl.md>), [small-language-models](<https://devfeed.tech/tags/small-language-models.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

In a controlled puzzle world where small language models' training priors and correct answers are known, outcome-reward RL turns models' prior hedging into decisions favored by reward. But when the correct action is rare, learning is inconsistent across runs: an action must succeed in enough groups of attempts for RL to reinforce it. The study reports that the expected number of winning groups predicts which runs learn, with about 35 groups corresponding to even odds. This relationship held for new and repeated problems, group sizes of 4-256 attempts, and models from 1.3M to 85M parameters, across the tested learning rates, optimizers, and GRPO variants; these choices shifted the threshold but not the predictive count.

### Source excerpt

arXiv:2611.26084v1 Announce Type: new Abstract: When a familiar rule stops working, an agent has to weigh what pre-training taught it against what it now observes. We ask how outcome-reward reinforcement learning (RL) changes this balance, in a puzzle world where both sides are known: small language models trained from scratch on games played by a Bayesian player, so that the prior and the right answer are exact. Base models hedge as their corpus did, and RL turns that hedging into decisions, in whichever direction the reward favours. But RL can only reinforce what it sees succeed. When the right action is rare, each run either learns it everywhere or not at all, and which runs take off is predicted before training by one count: the expected number of groups of attempts in which the action wins. About 35 such groups give even odds. The count holds for new and repeated problems and for groups of 4 to 256 attempts, and across learning rates, optimizers, GRPO variants and model sizes from 1.3M to 85M parameters, which move the threshold but not what is counted.

## Structured Role-Aware Policy Optimization for Multimodal Reasoning

DevFeed: [Structured Role-Aware Policy Optimization for Multimodal Reasoning](<https://devfeed.tech/articles/structured-role-aware-policy-optimization-for-multimodal-reasoning-144892.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.07274>)

Author: Bingqing Jiang, Guoxi Zhang, Jasper Wang, Bingning Wang, Difan Zou

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [grpo](<https://devfeed.tech/topics/grpo.md>), [rlvr](<https://devfeed.tech/topics/rlvr.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>)

Tags: [capabilities](<https://devfeed.tech/tags/capabilities.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [credit-assignment](<https://devfeed.tech/tags/credit-assignment.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dependency](<https://devfeed.tech/tags/dependency.md>), [evidence-based](<https://devfeed.tech/tags/evidence-based.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [grounded](<https://devfeed.tech/tags/grounded.md>), [group-relative-policy-optimization](<https://devfeed.tech/tags/group-relative-policy-optimization.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [policy](<https://devfeed.tech/tags/policy.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reasoning-tokens](<https://devfeed.tech/tags/reasoning-tokens.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>), [rlvr](<https://devfeed.tech/tags/rlvr.md>), [roles](<https://devfeed.tech/tags/roles.md>), [structured](<https://devfeed.tech/tags/structured.md>)

### AI overview

SRPO refines GRPO's sequence-level advantage into token-level credit for perception and reasoning, aiming to improve whether correct multimodal answers rely on relevant visual evidence. It uses on-policy contrasts between original and corrupted images to weight perception tokens by visual dependence, and weights reasoning tokens by consistency with the generated perception. A shared trajectory-level baseline combines these signals into positive weights that adjust update magnitudes while preserving the original reward and advantage signs. The method requires neither external reward models nor separate teachers. The abstract reports improved evidence-grounded reasoning across diverse multimodal benchmarks; it gives no specific benchmark results or quantitative gains.

### Source excerpt

arXiv:2605.07274v2 Announce Type: replace-cross Abstract: Reinforcement learning from verifiable rewards (RLVR), especially with Group Relative Policy Optimization (GRPO), has shown strong potential for improving the reasoning capabilities of large vision-language models (LVLMs). However, in multimodal reasoning, final-answer rewards are typically assigned at the sequence level and do not distinguish the functional roles of different tokens, making it difficult to determine whether a correct answer is supported by task-relevant visual evidence. In this paper, we revisit multimodal RLVR from the perspective of role-aware token-level credit assignment, where structured responses are decomposed into perception tokens for extracting visual evidence and reasoning tokens for deriving answers from that evidence. Based on this perspective, we propose Structured Role-aware Policy Optimization (SRPO), which refines the sequence-level GRPO advantage into role-aware token-level advantages without changing the reward function. Specifically, SRPO assigns role-specific credit by using self-distilled on-policy contrasts: perception tokens are emphasized according to their visual dependency under original versus corrupted visual inputs, while reasoning tokens are emphasized according to their consistency with the generated perception. These role-specific signals are further unified through a shared trajectory-level baseline, yielding positive token weights that adjust relative update magnitudes while preserving the original GRPO reward and advantage signs, without requiring external reward models or separate teachers. Experiments across diverse multimodal reasoning benchmarks show that SRPO improves evidence-grounded reasoning, highlighting the importance of moving beyond uniform sequence-level credit toward role-aware optimization for reliable multimodal reasoning.

## EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KL

DevFeed: [EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KL](<https://devfeed.tech/articles/ema-policy-gradient-taming-reinforcement-learning-for-llms-with-ema-anchor-and-top-k-kl-144874.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.04417>)

Author: Lunjun Zhang, Jimmy Ba

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [anchor](<https://devfeed.tech/tags/anchor.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deep-q-learning](<https://devfeed.tech/tags/deep-q-learning.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [llms](<https://devfeed.tech/tags/llms.md>), [math-reasoning](<https://devfeed.tech/tags/math-reasoning.md>), [policy](<https://devfeed.tech/tags/policy.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [rl](<https://devfeed.tech/tags/rl.md>), [top-k](<https://devfeed.tech/tags/top-k.md>)

### AI overview

EMA-PG combines two changes to LLM policy gradients: replacing a fixed RL anchor policy with an exponential moving average (EMA), and using a Top-k KL estimator to interpolate between exact and sampled KL. The authors report deriving stability conditions for the EMA anchor and show that the estimator gives unbiased KL values and gradients at any k. With GRPO, R1-distilled Qwen-1.5B scores 53.9% on OlympiadBench versus 50.8% for GRPO. On seven search-based Q&A datasets using Qwen-3B, EMA-PG improves GRPO by an average of 35.6%; HotpotQA rises from 29.7% to 45.8%, and 2WikiMultiHopQA from 27.4% to 40.5%. These are abstract-reported results; the supplied text does not include experimental details.

### Source excerpt

arXiv:2602.04417v2 Announce Type: replace Abstract: Reinforcement Learning (RL) has enabled Large Language Models (LLMs) to acquire increasingly complex reasoning and agentic behaviors. In this work, we propose two simple techniques to improve policy gradient algorithms for LLMs. First, we replace the fixed anchor policy during RL with an Exponential Moving Average (EMA), similar to a target network in deep Q-learning. Second, we introduce Top-$k$ KL estimator, which allows for flexible interpolation between exact KL and sampled KL. We derive the stability conditions for using EMA anchor; moreover, we show that our Top-$k$ KL estimator yields both unbiased KL values and unbiased gradients at any $k$, while bringing the benefits of exact KL. When combined with GRPO, the two techniques (EMA-PG) lead to a significant performance boost. On math reasoning, it allows R1-distilled Qwen-1.5B to reach 53.9% on OlympiadBench compared to 50.8% by GRPO. On agentic RL domains, with Qwen-3B base, EMA-PG improves GRPO by an average of 35.6% across 7 datasets of Q&A with search engines, including 29.7% $\rightarrow$ 45.8% on HotpotQA, 27.4% $\rightarrow$ 40.5% on 2WikiMultiHopQA. Overall, we show that EMA-PG is a simple, principled, and powerful approach to scaling RL for LLMs. Code: https://github.com/LunjunZhang/ema-pg

## Reinforcement-Aligned Probabilistic Weather Forecasting with Discrete Tokens

DevFeed: [Reinforcement-Aligned Probabilistic Weather Forecasting with Discrete Tokens](<https://devfeed.tech/articles/reinforcement-aligned-probabilistic-weather-forecasting-with-discrete-tokens-145071.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25570>)

Author: Jinhan Li, Zhisheng Chen, Yuxuan Li, Yuan Gao, Fan Xu, Hao Wu, Jinlong Du, Kun Wang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Climate / Weather / Ocean Modeling](<https://devfeed.tech/topics/climate-weather-ocean-modeling.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [caching](<https://devfeed.tech/tags/caching.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [group-relative-policy-optimization](<https://devfeed.tech/tags/group-relative-policy-optimization.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [ocean](<https://devfeed.tech/tags/ocean.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [probabilistic](<https://devfeed.tech/tags/probabilistic.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [scale](<https://devfeed.tech/tags/scale.md>), [simulations](<https://devfeed.tech/tags/simulations.md>), [skill](<https://devfeed.tech/tags/skill.md>), [temperature](<https://devfeed.tech/tags/temperature.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [weather](<https://devfeed.tech/tags/weather.md>), [weather-forecasting](<https://devfeed.tech/tags/weather-forecasting.md>)

### AI overview

WeatherCraft uses coarse-to-fine discrete weather tokens and a Transformer that predicts tokens in parallel within each scale to generate probabilistic medium-range forecasts. Exact token log-probabilities let Group Relative Policy Optimization optimize downstream rewards without differentiating through evaluators or coupled models. On ERA5, its eight-member ensemble mean reduced day-15 normalized RMSE by 17.6% versus the strongest evaluated deterministic baseline; with KV caching, inference was about 18x faster than GenCast. The abstract reports improved ensemble skill from direct fair-CRPS adaptation and better forecasts of persistent land-temperature and coupled ocean-temperature extremes from reward-based post-training. These are abstract-level results; details of evaluation and broader performance are not provided.

### Source excerpt

arXiv:2611.25570v1 Announce Type: new Abstract: Global medium-range weather forecasting must not only generate ensembles efficiently to characterize uncertainty in future states, but also serve downstream applications such as extreme-event warnings and cross-sphere coupled simulations. We introduce WeatherCraft, a framework that unifies probabilistic generation and reinforcement learning alignment through discrete weather tokens, bringing the GPT-style pretraining--post-training paradigm to global medium-range weather forecasting. A multiscale tokenizer represents atmospheric states as a coarse-to-fine token hierarchy, while a Transformer models causal dependencies across time and scale and predicts tokens within each scale in parallel. Direct sampling generates diverse ensembles, and exact token log-probabilities enable Group Relative Policy Optimization (GRPO) to maximize downstream task rewards without differentiating through evaluators or coupled models. On the ERA5 benchmark, WeatherCraft's eight-member ensemble mean reduces day-15 normalized RMSE by 17.6% relative to the strongest evaluated deterministic baseline and, with KV caching, achieves an approximately $18\times$ inference speedup over GenCast. Direct fair-CRPS adaptation improves ensemble probabilistic skill, while reward-based post-training improves forecasts of persistent land temperature extremes and coupled ocean heat extremes. This work establishes discrete autoregressive modeling as a general foundation for scalable, task-alignable Earth system foundation models.

## GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives

DevFeed: [GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives](<https://devfeed.tech/articles/groundingpi-a-grounding-foundation-model-towards-physical-intelligence-with-visual-primitives-144951.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.39601>)

Author: Qize Yu, Lianrui Fan, Boyu Chen, Jiaqi Liang, Xini Ding, Yue Chen, Zetian Song, Yuran Wang, Yi Zou, Kaixuan Wang, Tianxing Chen, Wenxuan Song, Bohan Zhou, Mingleyang Li, Siqiao Huang, Yuqi Ye, Caigao

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [Physical AI](<https://devfeed.tech/topics/physical-ai.md>), [Robot Manipulation](<https://devfeed.tech/topics/robot-manipulation.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [autonomous-driving](<https://devfeed.tech/tags/autonomous-driving.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [foundation-model](<https://devfeed.tech/tags/foundation-model.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [ocr](<https://devfeed.tech/tags/ocr.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>), [robotic-manipulation](<https://devfeed.tech/tags/robotic-manipulation.md>)

### AI overview

GroundingPI is a 4B model that represents points and boxes as quantized coordinates in a shared vocabulary, trained with multimodal and spatial pretraining, supervised fine-tuning, and GRPO reinforcement learning. The abstract reports an average of 73.68% across 34 grounding benchmarks, ahead of the larger GPT-6 Astra at 71.54%. As a visual backbone, it outperformed evaluated alternatives in all four out-of-distribution settings across RoboTwin 2.0 and RoboCasa-GR1, with gains up to 24.8%; on RoboCasa-GR1, using 50% of demonstrations beat baselines using 75%. Its mean open-loop L2 error on nuScenes was 0.296 m. The authors report that greater pretraining scale improved grounding and manipulation, dense grounding benefited both, and OCR may aid perceptual learning.

### Source excerpt

arXiv:2609.39601v3 Announce Type: replace-cross Abstract: Precise grounding matters. It specifies which object is the target and where that object is, even in clutter and for tiny objects, and it has to be fast enough for closed-loop control. Yet vision-language-action (VLA) and world-action models (WAMs) take perception from general-purpose vision-language and video-generation backbones, which still fail in these settings. We introduce GroundingPI, a 4B grounding foundation model that generates points and boxes as quantized coordinates in a shared vocabulary. Training combines multimodal and spatial pretraining, supervised fine-tuning, and reinforcement learning with GRPO, using supervision from public datasets and dedicated data engines. Against 44 baselines across 34 grounding benchmarks spanning 11 perceptual capabilities, GroundingPI establishes a new state of the art, averaging 73.68%, above the larger GPT-6 Astra (71.54%). As a downstream visual backbone, GroundingPI improves performance on robotic manipulation and autonomous driving. Across RoboTwin 2.0 and RoboCasa-GR1, it outperforms all evaluated backbones in all four out-of-distribution settings, with relative gains of up to 24.8% over the strongest baseline. On RoboCasa-GR1, GroundingPI trained with 50% of the demonstrations outperforms those baselines trained with 75%. Its mean open-loop L2 error on nuScenes is 0.296 m. We systematically analyze GroundingPI's pretraining in scale and data composition. Grounding quality and downstream manipulation improve with pretraining scale, with generalization tracking perceptual gains most closely. Analyzing the data recipe across these 11 perceptual capabilities shows dense grounding's substantial benefits for both, and OCR's potential as a catalyst for perceptual learning. These results support grounding as a perceptual foundation, and dedicated perceptual pretraining as a promising direction for foundation models of physical intelligence.

## Actor-Predictor Architecture for Reinforcement Learning with Formal Specification Objectives

DevFeed: [Actor-Predictor Architecture for Reinforcement Learning with Formal Specification Objectives](<https://devfeed.tech/articles/actor-predictor-architecture-for-reinforcement-learning-with-formal-specification-objectives-145652.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26313>)

Author: Josef Brozovich, Qining Zhang, Lei Ying

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [concept-learning](<https://devfeed.tech/topics/concept-learning.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Isaac](<https://devfeed.tech/topics/isaac.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [actor](<https://devfeed.tech/tags/actor.md>), [approach](<https://devfeed.tech/tags/approach.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [features](<https://devfeed.tech/tags/features.md>), [formal](<https://devfeed.tech/tags/formal.md>), [framework](<https://devfeed.tech/tags/framework.md>), [grid](<https://devfeed.tech/tags/grid.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [isaac-lab](<https://devfeed.tech/tags/isaac-lab.md>), [learning](<https://devfeed.tech/tags/learning.md>), [logic](<https://devfeed.tech/tags/logic.md>), [optimized](<https://devfeed.tech/tags/optimized.md>), [progress](<https://devfeed.tech/tags/progress.md>), [regular](<https://devfeed.tech/tags/regular.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>), [robotic](<https://devfeed.tech/tags/robotic.md>), [specification](<https://devfeed.tech/tags/specification.md>)

### AI overview

Actor-Predictor learns logic progress directly from partial trajectories, avoiding explicit automaton construction for reinforcement learning with formal specifications. A predictor estimates whether a partial trajectory will later satisfy the specification; its learned features become auxiliary actor inputs, and the actor is optimized with GRPO and RWR. The authors show that the representation is equivalent to an automaton state when predictive accuracy is sufficient. In grid-world and Isaac Lab robotic tasks, they report better performance than automaton-based baselines on LTL-constrained tasks, and performance on non-regular specifications such as $(a^n b^n)$, which finite-automaton approaches cannot handle.

### Source excerpt

arXiv:2611.26313v1 Announce Type: new Abstract: One fundamental challenge in reinforcement learning (RL) is enforcing temporal safety constraints, such as satisfying formal specifications. Existing methods often transform logic formulas into automata for state augmentation and reward shaping. However, this approach assumes that automaton construction is both feasible and tractable, which limits its applicability to small specifications and linear temporal logic (LTL). This paper proposes Actor-Predictor, a framework that learns a surrogate representation of logic progress without explicitly constructing an automaton. The predictor continuously estimates whether a partial trajectory will satisfy the formal specification in a later step, and its learned features are provided as auxiliary inputs to the actor, which is then optimized using GRPO and RWR. We show theoretically that, under sufficient predictive accuracy, the learned representation is equivalent to the automaton state. Experiments on grid world and robotic tasks in Isaac Lab demonstrate that Actor-Predictor outperforms automaton-based baselines on LTL-constrained tasks. It also performs well on non-regular formal specifications, such as $(a^n b^n)$, for which finite-automaton-based approaches are not applicable.

## Multi-Turn Agentic Reinforcement Learning with Tunix and Gemma 4 on Google TPUs

DevFeed: [Multi-Turn Agentic Reinforcement Learning with Tunix and Gemma 4 on Google TPUs](<https://devfeed.tech/articles/multi-turn-agentic-reinforcement-learning-with-tunix-and-gemma-4-on-google-tpus-144221.md>)

Original publisher: [Read original article](<https://developers.googleblog.com/multi-turn-agentic-reinforcement-learning-with-tunix-and-gemma-4-on-google-tpus/>)

Author: Wei Wei

Published: 2026-11-19T19:49:56.121412Z

Content type: tutorial

Language: en

Sources: [Google Developers Blog](<https://devfeed.tech/sources/google-developers-blog.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [gemma](<https://devfeed.tech/topics/gemma.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Multi-step AI processes](<https://devfeed.tech/topics/multi-step-ai-processes.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-tpu](<https://devfeed.tech/tags/cloud-tpu.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [gemma-4](<https://devfeed.tech/tags/gemma-4.md>), [google](<https://devfeed.tech/tags/google.md>), [group-relative-policy-optimization](<https://devfeed.tech/tags/group-relative-policy-optimization.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [learning](<https://devfeed.tech/tags/learning.md>), [llm](<https://devfeed.tech/tags/llm.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [tool](<https://devfeed.tech/tags/tool.md>), [tpus](<https://devfeed.tech/tags/tpus.md>), [training](<https://devfeed.tech/tags/training.md>), [using](<https://devfeed.tech/tags/using.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

The post describes training Gemma 4 E2B as a multi-turn Plane Strike agent with Tunix agentic GRPO on one Cloud TPU v6e-8. The game requires repeated board observations and `strike` tool calls; a custom environment, Gemma 4 chat parser, and conversation agent handle state, native tool responses, and action validation. Asynchronous vLLM rollouts with prefix caching support variable-length turns, while exact token continuity avoids retokenization drift. Starting from a 9.4% held-out win rate and 22.44 average strikes, the model reached 95% wins and about 16 strikes after 100 training steps; tool-call rate approached 100% and invalid actions approached zero. All strikes currently share a trajectory-level reward, which the post identifies as a limitation that finer per-turn credit assignment could address.

### Source excerpt

This post walks developers through training a multi-turn, tool-using LLM agent using Tunix and Gemma 4 on Cloud TPUs. By leveraging agentic Group Relative Policy Optimization (GRPO) and asynchronous vLLM rollouts, the approach significantly improves complex reasoning and tool compliance in interactive environments.

## Evidence-Based Credit Assignment for Search-Augmented Language Models

DevFeed: [Evidence-Based Credit Assignment for Search-Augmented Language Models](<https://devfeed.tech/articles/evidence-based-credit-assignment-for-search-augmented-language-models-142705.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23824>)

Author: Wen Wang, Jindong Wang, Jiahua Bao, Tu Yongsiqi, Yihao Liu, Junwei He, Wenkui Fan

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [credit-assignment](<https://devfeed.tech/tags/credit-assignment.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evidence-based](<https://devfeed.tech/tags/evidence-based.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [search](<https://devfeed.tech/tags/search.md>)

### AI overview

Evidence-Based Credit Assignment (ECA) adds a training-only signal to terminal answer rewards for search-augmented language models. A frozen snapshot of the policy estimates how retrieved, deduplicated documents change the likelihood of a reference answer using a prompt containing only the question and evidence, helping distinguish useful searches even when sampled answers all fail. The paper compares ECA-Step, which accumulates step-clipped potential increments, with ECA-Horizon, which preserves the horizon-wide potential gain; both use cross-fold group normalization, novelty gating, and search-local token assignment. Across seven Search QA benchmarks, ECA-Horizon raises strict macro exact match over outcome-only GRPO by 1.03 points on Qwen2.5-3B-Instruct and 4.80 points on Qwen2.5-7B-Instruct, and reportedly outperforms RLOO and GiGPO. The abstract also reports zero-shot transfer to MultiHop-RAG; it provides no further evaluation details.

### Source excerpt

arXiv:2611.23824v1 Announce Type: new Abstract: Reinforcement learning from terminal answer rewards enables search-augmented language models to interleave reasoning and retrieval without step-level annotations, yet outcome supervision assigns identical credit to every search in a trajectory regardless of whether it retrieved useful information. This limitation is especially severe when all sampled rollouts for a question fail, collapsing group-centered outcome advantages to zero even when trajectories gather markedly different evidence. We present Evidence-based Credit Assignment (ECA), which augments terminal outcome rewards with a training-only auxiliary signal measuring how retrieved documents improve the likelihood of a reference answer. Rather than scoring the full dialogue prefix, ECA queries a frozen snapshot of the current policy under an evidence-only prompt containing solely the question and the deduplicated documents observed so far, isolating the utility of retrieved passages from surface variations in generated thoughts or search queries. Within this framework, we compare two return formulations, ECA-Step (which accumulates step-clipped potential increments) and ECA-Horizon (which preserves the telescoping horizon gain in potential), paired with cross-fold group normalization, novelty gating, and search-local token assignment. Across seven Search QA benchmarks, ECA-Horizon improves strict macro exact match over an outcome-only GRPO baseline by 1.03 and 4.80 percentage points on Qwen2.5-3B- and 7B-Instruct, respectively, outperforming both RLOO and GiGPO. Zero-shot evaluation on MultiHop-RAG further demonstrates strong transfer to multi-hop inference queries.

## PGPO: Prototype-Guided Progressive Obfuscation for Privacy-Preserving LLM-Enhanced Recommendation

DevFeed: [PGPO: Prototype-Guided Progressive Obfuscation for Privacy-Preserving LLM-Enhanced Recommendation](<https://devfeed.tech/articles/pgpo-prototype-guided-progressive-obfuscation-for-privacy-preserving-llm-enhanced-recommendation-143114.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23236>)

Author: Shengxiang Lin, Jiajie Su, Pengyang Zhou, Xiang Chen, Xiaolin Zheng, Feng Tian, Chaochao Chen

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [Security & Privacy](<https://devfeed.tech/topics/security-privacy.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [personalization](<https://devfeed.tech/topics/personalization.md>)

Tags: [arxiv](<https://devfeed.tech/tags/arxiv.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [llm](<https://devfeed.tech/tags/llm.md>), [obfuscation](<https://devfeed.tech/tags/obfuscation.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [prototype](<https://devfeed.tech/tags/prototype.md>), [recommendation](<https://devfeed.tech/tags/recommendation.md>)

### AI overview

PGPO aims to protect sensitive item text in LLM-enhanced recommendation embeddings from inversion attacks while preserving the relative semantic relationships that support recommendation. It learns topology-aware obfuscation anchors for influential prototype words and propagates them through a static semantic graph. GRPO optimization uses individual dissimilarity, structural consistency, and semantic constraint rewards. The abstract reports strong inversion defense and near-upper-bound recommendation performance on two benchmarks; it does not provide benchmark names or quantitative results.

### Source excerpt

arXiv:2611.23236v1 Announce Type: cross Abstract: LLM-enhanced recommender systems encode item side information into semantic embeddings, but these embeddings may leak sensitive textual attributes through inversion attacks. Existing input-side defenses rely on trusted collection or local token perturbation, often disrupting the inter-item semantic topology needed for recommendation. To address it, we introduce relational fidelity as the key objective, requiring semantic shift for privacy while preserving relative item relations for utility. Based on this principle, we propose prototype-guided progressive obfuscation (PGPO), which learns topology-aware obfuscation anchors for high-influence prototype words and propagates them through a static semantic graph. PGPO is optimized with GRPO using individual dissimilarity, structural consistency, and semantic constraint rewards. Experiments on two benchmarks show strong inversion defense with near-upper-bound recommendation performance. The code is available at https://github.com/Shengxiang-Lin/PGPO.

## GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation

DevFeed: [GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation](<https://devfeed.tech/articles/grto-bootstrapped-group-relative-tool-optimization-for-referring-segmentation-143268.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.23500>)

Author: Mario Markov (INSAIT, Sofia University "St. Kliment Ohridski"), Stefan Maria Ailuro (INSAIT, Sofia University "St. Kliment Ohridski"), Mohammad Mahdi (INSAIT, Sofia University "St. Kliment Ohridski"),

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [grpo](<https://devfeed.tech/topics/grpo.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [improvements](<https://devfeed.tech/tags/improvements.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [optimize](<https://devfeed.tech/tags/optimize.md>), [optimizing](<https://devfeed.tech/tags/optimizing.md>), [policy](<https://devfeed.tech/tags/policy.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>), [robust](<https://devfeed.tech/tags/robust.md>), [shift](<https://devfeed.tech/tags/shift.md>), [tool](<https://devfeed.tech/tags/tool.md>)

### AI overview

GRTO jointly trains a vision-language model's reasoning policy and its differentiable segmentation tool, addressing the domain-shift risk of approaches that optimize the policy while treating the tool as perfect. It first bootstraps the tool toward the theoretically optimal policy without costly on-policy rollouts; it then extends GRPO to refine both policy and tool. The authors say prealignment yields a more robust reward signal and improves policy convergence. On three challenging referring segmentation benchmarks, GRTO reportedly improves on plain GRPO and alternative tool-training procedures, matching or surpassing domain-specific state-of-the-art methods. The supplied text is an abstract, so it does not provide benchmark names, quantitative results, or implementation details.

### Source excerpt

arXiv:2605.23500v3 Announce Type: replace-cross Abstract: Referring segmentation requires reasoning to connect language with visual evidence and identify the intended target in challenging scenarios. This makes GRPO a natural choice for training VLMs together with a segmentation tool. However, existing approaches optimize only the VLM, assuming that the tool is perfect and leaving it untouched. This assumption is particularly limiting when the tool faces domain shift, as segmentation errors can undermine even correct reasoning. We therefore argue that the segmentation tool should be optimized together with the reasoning policy. However, this is challenging because policy optimization and differentiable tool learning rely on fundamentally different objectives. In this work, we introduce group relative tool optimization (GRTO), a mathematically grounded two-stage framework for jointly optimizing a policy and its differentiable tool. The key idea is to first align the tool with the policy expected after joint training, rather than the initial policy. The prealigned tool then leads to a more robust reward signal and superior policy convergence in subsequent policy training. In the first stage, GRTO bootstraps the tool toward the theoretically optimal policy, capturing readily attainable gains without costly on-policy rollouts. This alignment bootstraps the second stage, where GRTO extends GRPO to jointly refine the policy and tool. Across three challenging referring segmentation benchmarks, GRTO achieves substantial improvements over plain GRPO and alternative tool training procedures, matching or surpassing domain-specific state-of-the-art methods. This demonstrates the value of unifying reinforcement learning with differentiable auxiliary objectives for reasoning-intensive segmentation.

## Process Advantage Signal Shaping: Advantage Construction for Process-Supervised Reinforcement Learning in LLM Reasoners

DevFeed: [Process Advantage Signal Shaping: Advantage Construction for Process-Supervised Reinforcement Learning in LLM Reasoners](<https://devfeed.tech/articles/process-advantage-signal-shaping-advantage-construction-for-process-supervised-reinforcement-learning-in-llm-reasoners-139785.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.29296>)

Author: Chao Wang, Hongtao Tian, Tao Yang, Yunsheng Shi, Ting Yao, Wenbo Ding

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [construction](<https://devfeed.tech/tags/construction.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [group-relative-policy-optimization](<https://devfeed.tech/tags/group-relative-policy-optimization.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [horizon](<https://devfeed.tech/tags/horizon.md>), [llm](<https://devfeed.tech/tags/llm.md>), [mathematical-reasoning](<https://devfeed.tech/tags/mathematical-reasoning.md>), [mathematics](<https://devfeed.tech/tags/mathematics.md>), [model](<https://devfeed.tech/tags/model.md>), [process](<https://devfeed.tech/tags/process.md>), [qa](<https://devfeed.tech/tags/qa.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [residual](<https://devfeed.tech/tags/residual.md>), [reward](<https://devfeed.tech/tags/reward.md>), [reward-model](<https://devfeed.tech/tags/reward-model.md>), [signal](<https://devfeed.tech/tags/signal.md>), [token](<https://devfeed.tech/tags/token.md>), [trace](<https://devfeed.tech/tags/trace.md>), [units](<https://devfeed.tech/tags/units.md>)

### AI overview

PASS shapes process-supervision scores into policy advantages for GRPO by separately normalizing and combining reward channels, forming batch-shared credit units from changes in the fused score trace, and scaling return-to-go by the remaining unit count. With learned process reward feedback on math reasoning, it raises average pass@1 by 4.62 percentage points over outcome-only GRPO across seven benchmarks, with responses about 2% shorter. On multi-hop QA using teacher-derived token feedback, it beats process-signal controls under two normalizers. Token-level accumulation slightly favors pass@1, while PASS yields higher pass@8/16 with shorter responses. Comparisons with Step-GRPO and component ablations support the shared credit units and residual normalization.

### Source excerpt

arXiv:2606.29296v3 Announce Type: replace Abstract: Dense process supervision provides local evaluations, but leaves open how these scores should become policy advantages. We introduce PASS (Process Advantage Signal Shaping), an advantage-construction method for group-relative policy optimization (GRPO). PASS separately normalizes and fuses reward channels, constructs batch-shared credit units from variation in the fused score trace, and normalizes return-to-go by the remaining unit count. These operations connect feedback scale, credit resolution, and accumulation horizon in a single computation. We evaluate learned process reward model (PRM) feedback for mathematical reasoning and teacher-derived token feedback for multi-hop question answering (QA). Across seven mathematics benchmarks, PASS improves average pass@1 by 4.62 percentage points over outcome-only GRPO with approximately $2\%$ shorter responses. Matched comparisons with Step-GRPO and component ablations further support the choice of shared credit units and residual normalization. On multi-hop QA, PASS improves over process-signal controls across two normalizers; token-level ablations reveal an accuracy--length trade-off: token-level accumulation slightly favors pass@1, while PASS achieves higher pass@8/16 with shorter responses. The results establish the conversion from local process scores to policy advantages as a consequential design choice across distinct supervision sources.

## TAILROLL: Tail-Aware Rollout Acceleration for RL-Based LLM Post-Training

DevFeed: [TAILROLL: Tail-Aware Rollout Acceleration for RL-Based LLM Post-Training](<https://devfeed.tech/articles/tailroll-tail-aware-rollout-acceleration-for-rl-based-llm-post-training-140384.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22206>)

Author: Niloofar Gholipour, Marcos Assuncao, Timothy Yu, Gursimran Singh, Rajkumar Buyya, Julien Gascon-Samson, Zhenan Fan, Yong Zhang

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [acceleration](<https://devfeed.tech/tags/acceleration.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [bubble](<https://devfeed.tech/tags/bubble.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [engine](<https://devfeed.tech/tags/engine.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generation](<https://devfeed.tech/tags/generation.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm](<https://devfeed.tech/tags/llm.md>), [masking](<https://devfeed.tech/tags/masking.md>), [migration](<https://devfeed.tech/tags/migration.md>), [models](<https://devfeed.tech/tags/models.md>), [online](<https://devfeed.tech/tags/online.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [rl](<https://devfeed.tech/tags/rl.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

TAILROLL targets two rollout bottlenecks in RL post-training: uneven request placement and waiting for the longest response. Its online scheduler migrates decoding requests between engines at chunk boundaries, resuming them with one re-prefill, and can terminate a response once its GRPO group has enough completed samples for advantage estimation; partial output is masked from training loss. Across ten configurations, it reports 21-74% lower rollout bubble ratio and 44-79% lower cross-engine KV-cache imbalance. End-to-end results include 24-34% less rollout time, 21-22% shorter full steps on long-CoT RL, 28-37% lower p95 latency, and up to 33% higher generation throughput, with comparable or higher final reward.

### Source excerpt

arXiv:2611.22206v1 Announce Type: new Abstract: RL post-training of large language models (LLMs) is bottlenecked by rollout: heavy-tailed responses leave engines idle as each step waits for the longest response. Asynchronous training masks, but does not eliminate, idle time. We distinguish imbalance waste (uneven placement and decode contention) from straggler waste (waiting for the longest response). Tail length, not average load, often sets the step bound. We propose TAILROLL, an online scheduler integrated into an open-source RL framework. It combines stateless in-flight migration and sufficiency-gated termination. Migration moves decoding requests between engines at chunk boundaries and resumes them with one re-prefill under a freeness-based, cost-aware tail-balancing policy. Termination stops a response when its GRPO group has enough completed samples for advantage estimation, masking its partial output from the training loss. Across ten configurations, TAILROLL reduces rollout bubble ratio by 21% to 74% and cross-engine KV-cache imbalance by 44% to 79%. In end-to-end RL training, it cuts rollout time by 24% to 34%, full step time by 21% to 22% on long-CoT RL, and p95 request latency by 28% to 37%, while raising generation throughput by up to 33% with comparable or higher final reward.

## From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment

DevFeed: [From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment](<https://devfeed.tech/articles/from-strategy-search-to-policy-learning-mcts-guided-adversarial-co-evolution-for-llm-safety-alignment-139751.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2511.19218>)

Author: Xurui Li, Kaisong Song

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Adversarial attacks](<https://devfeed.tech/topics/adversarial-attacks.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [adversarial](<https://devfeed.tech/tags/adversarial.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [bypass](<https://devfeed.tech/tags/bypass.md>), [credit-assignment](<https://devfeed.tech/tags/credit-assignment.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [curriculum](<https://devfeed.tech/tags/curriculum.md>), [defender](<https://devfeed.tech/tags/defender.md>), [defense](<https://devfeed.tech/tags/defense.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [framework](<https://devfeed.tech/tags/framework.md>), [jailbreak](<https://devfeed.tech/tags/jailbreak.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [learning](<https://devfeed.tech/tags/learning.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-safety](<https://devfeed.tech/tags/llm-safety.md>), [policy](<https://devfeed.tech/tags/policy.md>), [safety](<https://devfeed.tech/tags/safety.md>), [search](<https://devfeed.tech/tags/search.md>)

### AI overview

MACE uses Monte Carlo Tree Search to turn jailbreak discovery into learning signals for both attackers and defenders. GS-MCTS searches multi-step rewriting strategies and records attack success frequency and path depth. AC-TGPO uses those statistics for difficulty-aware credit assignment and combines current interactions with asymmetric attacker-defender pairings and persistent attacks that survive defense updates. In ablations, removing difficulty weighting, asymmetric samples, or persistent hard samples increased defense ASR-LR by 3.7, 6.0, and 5.0 percentage points, respectively. The attacker achieved the highest ASR-LR in all six evaluated model-dataset settings; the defender reduced average ASR-LR from the best baseline's 11.9% to 8.8%.

### Source excerpt

arXiv:2511.19218v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) increasingly power web-facing applications, including conversational assistants and content services, where open user interaction exposes them to diverse safety threats and adaptive attacks. In particular, jailbreak attackers repeatedly reformulate harmful requests to bypass safety alignment. Although existing studies have concentrated on defending dynamic jailbreaks, they often retain discovered attacks as flat samples and discard information about how those attacks were found. In this paper, we introduce MACE, an MCTS-guided adversarial co-evolution framework that turns search into policy-learning signals for both an attacker and a defender. Its search component, GS-MCTS, explores multi-step compositions of rewriting operators and associates discovered attacks with empirical success frequency and path depth. Its optimization component, AC-TGPO, uses these statistics for difficulty-aware group-relative credit assignment and constructs a temporally asymmetric curriculum from current interactions, pairings between the latest attacker and preceding defender, and attacks that survive a defense update. Thus, MACE connects search structure to both within-round policy optimization and cross-round memory, rather than using MCTS only to generate attacks or treating each round as a flat adversarial dataset. Removing difficulty-aware weighting, asymmetric samples, or persistent hard samples increases defense ASR-LR by 3.7 %, 6.0 %, and 5.0 %, respectively. Across evaluated models, the resulting attacker achieves the highest ASR-LR in all six model--dataset settings, while the defender reduces average ASR-LR from 11.9 % for the best baseline to 8.8 %.

## Efficient Generative Retrieval for E-commerce Search with Semantic Cluster IDs and Expert-Guided RL

DevFeed: [Efficient Generative Retrieval for E-commerce Search with Semantic Cluster IDs and Expert-Guided RL](<https://devfeed.tech/articles/efficient-generative-retrieval-for-e-commerce-search-with-semantic-cluster-ids-and-expert-guided-rl-139770.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.14434>)

Author: Jianbo Zhu, Xing Fang, Jing Wang, Mingmin Jin, Bokang Wang, Guangxin Song, Zhenyu Xie, Junjie Bai

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [information retrieval](<https://devfeed.tech/topics/information-retrieval.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [AI-driven information retrieval](<https://devfeed.tech/topics/ai-driven-information-retrieval.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [VAE](<https://devfeed.tech/topics/vae.md>)

Tags: [beam](<https://devfeed.tech/tags/beam.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [contrastive-learning](<https://devfeed.tech/tags/contrastive-learning.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [e-commerce](<https://devfeed.tech/tags/e-commerce.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generative](<https://devfeed.tech/tags/generative.md>), [group-relative-policy-optimization](<https://devfeed.tech/tags/group-relative-policy-optimization.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [ids](<https://devfeed.tech/tags/ids.md>), [learning](<https://devfeed.tech/tags/learning.md>), [offline](<https://devfeed.tech/tags/offline.md>), [online](<https://devfeed.tech/tags/online.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [policy](<https://devfeed.tech/tags/policy.md>), [production](<https://devfeed.tech/tags/production.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [rl](<https://devfeed.tech/tags/rl.md>), [search](<https://devfeed.tech/tags/search.md>), [semantic](<https://devfeed.tech/tags/semantic.md>)

### AI overview

The framework positions generative retrieval as a recall-stage supplement to existing e-commerce search, addressing catalog scale, latency, and downstream ranking alignment. CQ-SID combines category-aware and query-item contrastive learning with Residual Quantized VAEs to create hierarchical semantic IDs and reduce beam-search complexity. On TmallAPP search logs, it improves semantic and personalized click hitrate by up to 26.76% and 11.11% over RQ-VAE baselines while halving beam-search size. EG-GRPO uses ground-truth samples to stabilize reinforcement learning under sparse rewards and further improves multi-objective performance. Online A/B tests report GMV gains of 1.15% and UCTCVR gains of 0.40%; in production, the generative recall channel accounts for 50.25% of exposures, 58.96% of clicks, and 72.63% of purchases.

### Source excerpt

arXiv:2605.14434v2 Announce Type: replace-cross Abstract: Generative retrieval offers a promising alternative by unifying the fragmented multi-stage retrieval process into a single end-to-end model. However, its practical adoption in industrial e-commerce search remains challenging, given the massive and dynamic product catalogs, strict latency requirements, and the need to align retrieval with downstream ranking goals. In this work, we propose a retrieval framework tailored for real-world recall scenarios, positioning generative retrieval as a recall-stage supplement rather than an end-to-end replacement. Our method, CQ-SID (Category-and-Query constrained Semantic ID), employs category-aware and query-item contrastive learning along with Residual Quantized VAEs to encode items into hierarchical semantic cluster identifiers, significantly reducing beam search complexity. Additionally, we develop EG-GRPO (Expert-Guided Group Relative Policy Optimization), a reinforcement learning approach that aligns generative recall with downstream ranking under sparse rewards by injecting ground-truth samples to stabilize training. Offline experiments on TmallAPP search logs show that CQ-SID achieves up to 26.76% and 11.11% relative gains in semantic and personalized click hitrate over RQ-VAE baselines, while halving beam search size. EG-GRPO further improves multi-objective performance. Online A/B tests confirm gains in GMV (+1.15%) and UCTCVR (+0.40%). The generative recall channel now contributes substantially in production, accounting for over 50.25% of exposures, 58.96% of clicks, and 72.63% of purchases, demonstrating a viable path for deploying generative retrieval in real-world e-commerce systems.

## CORE: Concept-Oriented Reinforcement for Bridging the Definition-Application Gap in Mathematical Reasoning

DevFeed: [CORE: Concept-Oriented Reinforcement for Bridging the Definition-Application Gap in Mathematical Reasoning](<https://devfeed.tech/articles/core-concept-oriented-reinforcement-for-bridging-the-definition-application-gap-in-mathematical-reasoning-139754.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2512.18857>)

Author: Zijun Gao, Zhikun Xu, Xiao Ye, Ben Zhou

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [rlvr](<https://devfeed.tech/topics/rlvr.md>)

Tags: [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [concepts](<https://devfeed.tech/tags/concepts.md>), [core](<https://devfeed.tech/tags/core.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llms](<https://devfeed.tech/tags/llms.md>), [math](<https://devfeed.tech/tags/math.md>), [mathematical-reasoning](<https://devfeed.tech/tags/mathematical-reasoning.md>), [problem-solving](<https://devfeed.tech/tags/problem-solving.md>), [quizzes](<https://devfeed.tech/tags/quizzes.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [rl](<https://devfeed.tech/tags/rl.md>), [rlvr](<https://devfeed.tech/tags/rlvr.md>), [sft](<https://devfeed.tech/tags/sft.md>), [supervision](<https://devfeed.tech/tags/supervision.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

CORE addresses a gap in reinforcement learning with verifiable rewards (RLVR), which trains final-answer correctness but supplies little direct feedback about applying concepts. The framework uses textbook-linked exercises and concise definitions to create concept-aligned quizzes, and can inject concept snippets into training rollouts. It reinforces concept use through trajectory replacement after group failures, a lightweight forward-KL constraint aligning unguided and concept-primed policies, or standard GRPO on concept-aligned quizzes. The abstract reports consistent gains over vanilla and supervised fine-tuning baselines across in-domain concept exercises and diverse out-of-domain math benchmarks. It presents CORE as compatible with different algorithms and verifiers; detailed experimental results are not included in the supplied excerpt.

### Source excerpt

arXiv:2512.18857v4 Announce Type: replace Abstract: Large language models (LLMs) often solve challenging math exercises yet fail to apply the concept right when the problem requires genuine understanding. Popular Reinforcement Learning with Verifiable Rewards (RLVR) pipelines reinforce final answers but provide little fine-grained conceptual signal, so models improve at pattern reuse rather than conceptual applications. We introduce CORE (Concept-Oriented REinforcement), an RL training framework that turns explicit concepts into a controllable supervision signal. Starting from a high-quality, low-contamination textbook resource that links verifiable exercises to concise concept descriptions, we run a sanity probe showing LLMs can restate definitions but fail concept-linked quizzes, quantifying the conceptual reasoning gap. CORE then (i) synthesizes concept-aligned quizzes, (ii) injects brief concept snippets during rollouts to elicit concept-primed trajectories, and (iii) reinforces conceptual reasoning via trajectory replacement after group failures, a lightweight forward-KL constraint that aligns unguided with concept-primed policies, or standard GRPO directly on concept-aligned quizzes. Across several models, CORE delivers consistent gains over vanilla and SFT baselines on both in-domain concept-exercise suites and diverse out-of-domain math benchmarks. CORE unifies direct training on concept-aligned quizzes and concept-injected rollouts under outcome regularization. It provides fine-grained conceptual supervision that bridges problem-solving competence and genuine conceptual reasoning, while remaining algorithm- and verifier-agnostic.

## ReMaRL: Learning to Unmask and Remask in Frozen Diffusion Language Models

DevFeed: [ReMaRL: Learning to Unmask and Remask in Frozen Diffusion Language Models](<https://devfeed.tech/articles/remarl-learning-to-unmask-and-remask-in-frozen-diffusion-language-models-141788.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22576>)

Author: Egor Shibaev, Danila Biktimirov, Ivan Klimov, Artem Makoian, Viacheslav Meshchaninov, Dmitry Vetrov

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [code](<https://devfeed.tech/tags/code.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [group-relative-policy-optimization](<https://devfeed.tech/tags/group-relative-policy-optimization.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [learning](<https://devfeed.tech/tags/learning.md>), [math](<https://devfeed.tech/tags/math.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>)

### AI overview

ReMaRL trains a roughly 0.3M-parameter policy with GRPO and final-answer rewards to choose which tokens a frozen diffusion language model should unmask, retain, or remask. Unlike reward-trained unmasking policies that cannot revise earlier predictions, ReMaRL can remask tokens so they can be corrected using newly generated context. With LLaDA-8B-Instruct, it achieved the highest scores among the compared policies in the tested budgets: 4-32 model calls for math and 8-16 for code.

### Source excerpt

arXiv:2611.22576v1 Announce Type: new Abstract: Masked diffusion language models generate tokens in parallel, and the order in which tokens are unmasked affects final-answer accuracy. Several methods learn unmasking policies from task rewards, but these policies cannot remask earlier predictions. Remasking lets the model correct those predictions using newly generated context. Heuristic rules guide both unmasking and remasking, while supervised remaskers learn to detect token errors. Neither approach directly optimizes these decisions for final-answer correctness. To address this gap, we introduce ReMaRL (Remasking with Reinforcement Learning), a lightweight policy that learns which tokens to unmask, retain, or remask. We train this roughly 0.3M-parameter policy with Group Relative Policy Optimization (GRPO) from final-answer rewards while keeping the diffusion model frozen. With LLaDA-8B-Instruct, ReMaRL achieves the highest scores among the compared unmasking and remasking policies within the tested inference budgets of 4-32 model calls on math and 8-16 calls on code.

## Learning to Edit IFC Building Models: Verifiable Checkers as the Training Signal for Dependency-Aware Editing Agents

DevFeed: [Learning to Edit IFC Building Models: Verifiable Checkers as the Training Signal for Dependency-Aware Editing Agents](<https://devfeed.tech/articles/learning-to-edit-ifc-building-models-verifiable-checkers-as-the-training-signal-for-dependency-aware-editing-agents-140010.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22307>)

Author: Jiaxin Zhang, Yao Luo, Yuze Dan, Yunqin Li, Bowen Wang

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [rlvr](<https://devfeed.tech/topics/rlvr.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [AI Development](<https://devfeed.tech/topics/ai-development.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [after](<https://devfeed.tech/tags/after.md>), [agents](<https://devfeed.tech/tags/agents.md>), [all](<https://devfeed.tech/tags/all.md>), [allowed](<https://devfeed.tech/tags/allowed.md>), [also](<https://devfeed.tech/tags/also.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bim](<https://devfeed.tech/tags/bim.md>), [building](<https://devfeed.tech/tags/building.md>), [changes](<https://devfeed.tech/tags/changes.md>), [classes](<https://devfeed.tech/tags/classes.md>), [code](<https://devfeed.tech/tags/code.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [dependency](<https://devfeed.tech/tags/dependency.md>), [ifc](<https://devfeed.tech/tags/ifc.md>), [learning](<https://devfeed.tech/tags/learning.md>), [models](<https://devfeed.tech/tags/models.md>), [signal](<https://devfeed.tech/tags/signal.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

IfcGym trains IFC editing agents with verifiable task outcomes: four checkers assess validity, intent, dependency handling and preservation, and all must pass for strict success. IfcEdit first uses supervised fine-tuning on verified teacher episodes, then GRPO with strict success as its reward. On 1,990 IfcEditBench tasks across 13 held-out projects, the Qwen3.5-9B student rises from 79.15% strict success after SFT to 86.78% after GRPO; its paired confidence interval versus the teacher includes zero. The same recipe reaches 91.76% on a 27B model and exceeds the teacher. On IfcEditBench, students beat prompted models except the teacher with matched tool access, while both evaluated frameworks score lower. Using all four checkers as reward yields 4.83 points more strict success than rewarding intent and validity alone. Scores on BIM-Edit and BIBIMBAP are 55.19 and 54.50.

### Source excerpt

arXiv:2611.22307v1 Announce Type: new Abstract: Editing a building information model in the Industry Foundation Classes (IFC) schema requires propagating changes to dependent elements while preserving unrelated entities. Existing language-model agents for IFC editing use prompted models without learning from edit outcomes. Target-file comparisons do not specify which alternative edits are allowed or which unrelated entities must remain unchanged. We present IfcGym, a CodeAct environment in which a language model edits IFC through executable code and four checkers verify validity relative to the input model, intent, dependency and preservation against a task specification. Their conjunction is strict success. IfcEdit distils verified teacher episodes into a Qwen3.5-9B student through supervised fine-tuning (SFT), then applies Group Relative Policy Optimization (GRPO) with strict success as the reward. On IfcEditBench, 1,990 editing tasks on 13 held-out real projects, the final student reaches 86.78% strict success, from 79.15% after SFT. The paired confidence interval for the difference from its teacher includes zero; the same recipe on a 27B model reaches 91.76% and exceeds the teacher. On the public benchmarks BIM-Edit and BIBIMBAP the final student scores 55.19 and 54.50. On IfcEditBench, the students outperform all prompted models except the teacher under matched tool access; both evaluated frameworks also score lower. Rewarding all four checkers yields 4.83 points more strict success than rewarding intent and validity alone.

## MedGuardBench: A Patient-Oriented Benchmark for Personalized Guardrail Development

DevFeed: [MedGuardBench: A Patient-Oriented Benchmark for Personalized Guardrail Development](<https://devfeed.tech/articles/medguardbench-a-patient-oriented-benchmark-for-personalized-guardrail-development-139955.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22069>)

Author: Sara Khosravi, Xiaofei Wen, Majid Sarrafzadeh, Muhao Chen

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [guardrail](<https://devfeed.tech/tags/guardrail.md>), [personal](<https://devfeed.tech/tags/personal.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [risk](<https://devfeed.tech/tags/risk.md>), [safety](<https://devfeed.tech/tags/safety.md>)

### AI overview

MedGuardBench evaluates patient-conditioned medication safety across 5,001 drug-disjoint scenarios covering 87 medications, predicting a safety verdict and 17 risk categories from a patient profile and prescription. The authors report that standard guardrails often respond to surface cues instead of patient facts. Their personalized reasoning approach links risks to relevant profile fields; Context Unrolling prompts models to check fields individually and reduces false alarms on most tested backbones, while simply supplying the check does not provide the same benefit. Guided GRPO improves unsafe-case recall and detection of complex risks such as allergies and adverse reactions on some backbones, but raises false alarms on others. The models improve accuracy over base models and matched no-profile controls, and profile removal and swapping suggest the gains depend on patient information.

### Source excerpt

arXiv:2611.22069v1 Announce Type: new Abstract: Safety is personal: the same action can be safe for one person but unsafe for another. Yet current guardrails judge content without considering who is affected. Using medication safety, we introduce \textit{patient-conditioned medication guarding}, which predicts a safety verdict and 17 risk categories from a patient profile and prescription. We release \textbf{MedGuardBench}, a drug-disjoint benchmark of 5{,}001 scenarios across 87 medications. We find that standard guardrails rely on surface cues, such as wording and profile presence, rather than patient facts. To address this, we develop \textit{personalized reasoning guardrails} with three components. \textbf{Personalized Reasoning Distillation} links risks to relevant patient fields. \textbf{Context Unrolling (CU)} makes models check each field themselves, reducing false alarms on most backbones; controls show that providing the check does not yield the same benefit. A \textbf{rule-based reward (guided GRPO)} improves unsafe recall and complex-risk detection, including allergies and adverse reactions, on some backbones, but increases false alarms on others. Our models outperform base models and matched no-profile controls in accuracy, while reducing false alarms relative to no-profile controls. Profile removal and swapping confirm that gains come from patient information, not fine-tuning alone.

## When Terminal-Agent Training Stalls: Demystifying Data Generation and Verification Challenge

DevFeed: [When Terminal-Agent Training Stalls: Demystifying Data Generation and Verification Challenge](<https://devfeed.tech/articles/when-terminal-agent-training-stalls-demystifying-data-generation-and-verification-challenge-139829.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.02405>)

Author: Xi Qin, Isabel Kurth, Xin Cui, Elin Park, Alexander Schaefer, Yaad Oren

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>), [Tool](<https://devfeed.tech/topics/tool.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-behavior](<https://devfeed.tech/tags/agent-behavior.md>), [agent-training](<https://devfeed.tech/tags/agent-training.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [audits](<https://devfeed.tech/tags/audits.md>), [automated](<https://devfeed.tech/tags/automated.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [bias](<https://devfeed.tech/tags/bias.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [community](<https://devfeed.tech/tags/community.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [data](<https://devfeed.tech/tags/data.md>), [generation](<https://devfeed.tech/tags/generation.md>), [terminal](<https://devfeed.tech/tags/terminal.md>), [terminal-agent](<https://devfeed.tech/tags/terminal-agent.md>), [training](<https://devfeed.tech/tags/training.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

A runnable container and executable tests do not ensure that terminal-agent rewards are faithful or informative. The paper presents a pipeline for generating containerized tasks and verifiers, and separates four evaluation requirements: structural validity, verifier faithfulness, policy solvability, and infrastructure reliability. In GRPO training, Qwen3.5-9B overfit a relatively easy dataset within 20 steps; in a controlled comparison, adding hard tasks reduced mean rollout pass@2 from 81.3% to 20.6% without changing the training configuration. The authors identify near-duplicate tasks, incomplete anti-shortcut safeguards, Dockerfile failures, asynchronous pass@k bias, and concurrency-induced environment errors as measurement threats, and call for evaluation combining structural checks, rollout-derived solvability, and verifier audits.

### Source excerpt

arXiv:2610.02405v2 Announce Type: replace Abstract: Automated terminal-agent training relies on generated environments and verifiers to turn open-ended interaction into reward. Yet a runnable Docker image and executable test suite do not guarantee that the resulting reward is faithful, informative, or even attributable to agent behavior. We present a pipeline that generates containerized terminal tasks and their executable verifiers, and use it to demystify failure modes in task generation and live-environment evaluation. Our analysis separates four requirements that are often conflated: structural validity, verifier faithfulness, policy solvability, and infrastructure reliability. In a GRPO training of Qwen3.5-9B, it overfits on a relatively easy dataset within 20 steps. Conversely, in a controlled Qwen3.5-9B comparison, adding hard tasks lowers mean rollout pass@2 from 81.3% to 20.6%, restoring informative failures without changing the training configuration. We further identify near-duplicate task generation, incomplete anti-shortcut safeguards, Dockerfile failures, asynchronous pass@k bias, and concurrency-induced environment errors as distinct threats to measurement. These findings should motivate the community for a terminal agent specific evaluation harness that pairs structural checks and rollout-derived solvability with extensive verifier audits.

## Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Models

DevFeed: [Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Models](<https://devfeed.tech/articles/video-hopchain-multi-hop-questions-and-confidence-gated-exploration-for-video-reasoning-models-139819.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.25773>)

Author: Trung Nguyen Quang, Yuhao Dong, Shuo Sun, Shuai Liu, Shulin Tian, Kim-Hui Yap, Ziwei Liu

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [rlvr](<https://devfeed.tech/topics/rlvr.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [budget](<https://devfeed.tech/tags/budget.md>), [chain-of-thought](<https://devfeed.tech/tags/chain-of-thought.md>), [checkpoint](<https://devfeed.tech/tags/checkpoint.md>), [code](<https://devfeed.tech/tags/code.md>), [compute](<https://devfeed.tech/tags/compute.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [data](<https://devfeed.tech/tags/data.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [errors](<https://devfeed.tech/tags/errors.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generation](<https://devfeed.tech/tags/generation.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [models](<https://devfeed.tech/tags/models.md>), [questions](<https://devfeed.tech/tags/questions.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reasoning-models](<https://devfeed.tech/tags/reasoning-models.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

Video-HopChain uses multi-step questions about moments in a video to train video reasoning models with verifiable rewards. Its 22,550 questions span 13,378 videos, with a held-out benchmark of 1,000; each chains three to six yes/no questions and scores the summed integer answers by exact match. For Qwen3-VL-8B, a second GRPO training stage on this dataset raises the mean across eight benchmarks from 55.4 to 57.9, improving all eight. The authors address GRPO groups with no reward variation using Confidence-Gated Exploration: after four initial rollouts are all correct or all incorrect, four more are sampled with the policy's most confident reasoning token masked, and those masked positions are excluded from the loss. At the same compute budget, CGE raises the mean to 59.3. The abstract says the dataset, checkpoint, and generation and training code are released.

### Source excerpt

arXiv:2609.25773v3 Announce Type: replace-cross Abstract: HopChain has shown on still images that multi-hop data synthesis improves vision-language reasoning, because long chain-of-thought reasoning exposes errors that compound across steps, while most data used for reinforcement learning with verifiable rewards (RLVR) rarely demands a chain of visual evidence, so these weaknesses are likely to stay unexposed. We observe the same problem in video, where this framework has not yet been explored. We therefore build Video-HopChain, a dataset of 22,550 multi-hop video questions over 13,378 videos, together with a held-out benchmark of 1,000 questions. Each question chains three to six yes/no questions about moments in one video, and each yields one of two integers depending on its answer. The final answer is the sum of these integers, so an exact match on that sum gives the verifiable reward that RLVR needs. We first train Qwen3-VL-8B with GRPO on a standard video dataset, and a second stage on Video-HopChain then raises the mean over eight video understanding and reasoning benchmarks from 55.4 to 57.9 and improves every one of them. Training on such a dataset, however, exposes a known limitation of GRPO: its learning signal comes from the reward variance within a group, so hard questions whose rollouts are all incorrect and easy questions whose rollouts are all correct both leave the group with no gradient. To recover these groups at the same compute budget, we introduce Confidence-Gated Exploration (CGE). With 8 rollouts per question, CGE samples the first 4 as usual. If these 4 are either all correct or all incorrect, it samples the last 4 with the policy's most confident token masked inside the reasoning span, and removes the masked positions from the loss while all 8 rollouts enter the advantage. With CGE, the mean rises further to 59.3. We release the dataset, the checkpoint, and the data generation and training code.

[Next page](<https://devfeed.tech/topics/grpo.md?cursor=WyIyMDI2LTExLTE4VDA1OjAwOjAwKzAwOjAwIiwgIjE4MDFjNWE0LWMxMDgtNDU3MS05YTk1LWRmOTZhZjQ0NzdmNyJd>)