# Co-Evolving Policy and Reward for Code Generation: A Mechanistic Study of Single-Adapter Drift

DevFeed: [Co-Evolving Policy and Reward for Code Generation: A Mechanistic Study of Single-Adapter Drift](<https://devfeed.tech/articles/co-evolving-policy-and-reward-for-code-generation-a-mechanistic-study-of-single-adapter-drift-93626.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.26416>)

Author: Amarsaikhan Batjargal, Noriko Tomuro

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [rlvr](<https://devfeed.tech/topics/rlvr.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>)

Tags: [adapter](<https://devfeed.tech/tags/adapter.md>), [code](<https://devfeed.tech/tags/code.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [diagnostics](<https://devfeed.tech/tags/diagnostics.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [execution](<https://devfeed.tech/tags/execution.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [learning](<https://devfeed.tech/tags/learning.md>), [model](<https://devfeed.tech/tags/model.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [policy](<https://devfeed.tech/tags/policy.md>), [problems](<https://devfeed.tech/tags/problems.md>), [recycling](<https://devfeed.tech/tags/recycling.md>), [reflection](<https://devfeed.tech/tags/reflection.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>), [rlvr](<https://devfeed.tech/tags/rlvr.md>), [self-improvement](<https://devfeed.tech/tags/self-improvement.md>)

## AI overview

In experiments with Qwen2.5-Coder-3B-Instruct, SPARK-style rollout recycling improved training performance but did not yield sustained gains on held-out code problems, and the results did not support added benefits from execution-grounded reflection. At the same training stage, rollout recycling produced roughly 34 times the KL divergence from the reference policy seen with execution-only GRPO. As training pass rate rose, trained tokens per GRPO sequence fell from 185 to 86 while held-out performance declined or failed to sustain early gains. The authors describe this as single-adapter policy drift: shared parameters move away from pretrained code-generation behavior without corresponding generalization. They caution that higher training reward alone does not demonstrate self-improvement and emphasize held-out evaluation and drift diagnostics.

## Source excerpt

arXiv:2610.26416v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) is particularly well suited to code generation because generated programs can be objectively evaluated through execution. Execution also provides behavioral information that can be reused as additional supervision. The Synergistic Policy and Reward Co-Evolving Framework (SPARK), originally proposed for reasoning tasks, exploits this idea by recycling training rollouts and their evaluation results to train a model not only to generate solutions but also to evaluate and improve them. We investigate whether this policy-reward co-evolution approach transfers effectively to code generation and whether execution-grounded reflection provides an additional source of self-improvement. We compare SPARK-style rollout recycling with execution-only Group Relative Policy Optimization (GRPO) using Qwen2.5-Coder-3B-Instruct. Our results do not support either hypothesis under the configuration tested. Although co-evolution improves training performance, these gains do not generalize to held-out problems. Compared at the same stage of training, rollout recycling produces roughly 34x greater KL divergence from the reference policy than execution-only GRPO. Over training, trained tokens per GRPO sequence fall from 185 to 86 as training pass rate increases, while held-out performance declines or fails to sustain early gains. We characterize these patterns as single-adapter policy drift: the shared trainable parameters increasingly depart from pretrained code-generation behavior without corresponding gains in generalization. The results show that higher training reward alone is not evidence of generalizable code self-improvement and highlight the importance of held-out evaluation and drift diagnostics.