# RLiD: Reinforcement Learning in the Decision Plane for Language Agents and LLMs

DevFeed: [RLiD: Reinforcement Learning in the Decision Plane for Language Agents and LLMs](<https://devfeed.tech/articles/rlid-reinforcement-learning-in-the-decision-plane-for-language-agents-and-llms-115309.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.02455>)

Author: Ziyi Chen, Yan Zhang, Jianhui Wei, Jianing Yin, Daoan Zhang

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [decision-making](<https://devfeed.tech/topics/decision-making.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [arc](<https://devfeed.tech/tags/arc.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [decision-model](<https://devfeed.tech/tags/decision-model.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [llms](<https://devfeed.tech/tags/llms.md>), [memory](<https://devfeed.tech/tags/memory.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>)

## AI overview

RLiD trains language agents by optimizing probabilities over executable actions instead of generated response tokens. It presents observations, optional compact memory, and a labeled action menu to a large decision model, which chooses an action in one forward pass; this avoids autoregressive generation, parsing, and invalid action outputs. The authors argue that marginalizing over different text realizations of the same action removes a zero-mean gradient component without changing the expected update, then distill trained decision policies into generative LLMs using action trajectories. Across ARC, Countdown, and Sokoban, they report improved performance or lower computation in several settings. On ALFWorld, training and distillation reached 99.3% success in 12.7 H100 GPU-hours, versus 86.6% in 17.3 hours for token-level RL of the same LLM. With compiled memory, Qwen2.5-1.5B decision and distilled generative models achieved 100% on the 134 unseen and 140 seen evaluation games.

## Source excerpt

arXiv:2611.02455v1 Announce Type: new Abstract: Reinforcement learning (RL) has improved the capabilities of large language models (LLMs) across a wide range of tasks. However, token-level RL assigns an action's advantage to the entire generated response, including tokens specifying the parsed action, reasoning chains, and malformed content. Although the environment executes only the parsed action, variation in how the action is expressed in natural language can introduce additional variance into policy-gradient estimates. We propose RLiD (Reinforcement Learning in the Decision plane), a framework that directly optimizes the probabilities of executable action options. RLiD compiles observations, an optional compact memory, and a labeled action menu into the model input, and uses a large decision model (LDM) to select an action in a single forward pass. By eliminating autoregressive token generation and bypassing action parsing, RLiD avoids invalid action outputs and improves RL rollout efficiency. Our analysis shows that, within a fixed text policy, marginalizing over alternative realizations of the same action removes a zero-mean gradient component without changing the expected update. We further distill trained LDM policies into generative LLMs through supervision on their action trajectories. Experiments on ARC, Countdown, and Sokoban demonstrate improved performance or lower computational cost across several settings. On ALFWorld, the complete training and distillation pipeline reaches 99.3% success in 12.7 H100 GPU-hours, compared with 86.6% in 17.3 GPU-hours for token-level RL of the same LLM. With compiled memory, our strongest Qwen2.5-1.5B LDM and distilled generative LLM achieve 100% success on all 134 unseen and 140 seen ALFWorld evaluation games.