# R$^2$Orch: Recursive Self-Improvement for Agent Orchestration via Self-Play within a Skill-Internalization Loop

DevFeed: [R$^2$Orch: Recursive Self-Improvement for Agent Orchestration via Self-Play within a Skill-Internalization Loop](<https://devfeed.tech/articles/r-2-orch-recursive-self-improvement-for-agent-orchestration-via-self-play-within-a-skill-internalization-loop-97998.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.22740>)

Author: Mingda Zhang, Yun Huang, Qiang Huang, Yanjin Li, Rui Qian, Qichao Zhao, Tiesunlong Shen, Na Zhao, Jian Wang, Xiaoying Tang

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [agent orchestration](<https://devfeed.tech/topics/agent-orchestration.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [rlvr](<https://devfeed.tech/topics/rlvr.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [agent-orchestration](<https://devfeed.tech/tags/agent-orchestration.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [learning](<https://devfeed.tech/tags/learning.md>), [math](<https://devfeed.tech/tags/math.md>), [play](<https://devfeed.tech/tags/play.md>), [question](<https://devfeed.tech/tags/question.md>), [recursive-self-improvement](<https://devfeed.tech/tags/recursive-self-improvement.md>), [skills](<https://devfeed.tech/tags/skills.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

R$^2$Orch trains an agent orchestrator through nested self-play: an inner loop improves it on tasks chosen by a Proposer, while an outer loop improves that task-selection process. Its Recursive Learnability objective favors tasks whose rollout outcomes remain varied as skills are withdrawn, addressing the stated problem of collapsed learning signals when outcomes converge. In Skill Withdrawal, a skill is removed when controlled evaluation shows the orchestrator performs as well without it. Failures can then produce new skills, which are internalized into model parameters; withdrawals prompt the Proposer to seek new failures, aiming for deployment without explicit skills. The abstract reports better results than baselines across 12 datasets in question answering, math, code, and interactive tasks; it provides no detailed metrics or baseline comparisons.

## Source excerpt

arXiv:2610.22740v2 Announce Type: replace Abstract: In recent years, agentic orchestration has been widely applied to solve complex human tasks. However, existing orchestration learning still faces key challenges, including hand-designed curricula, advantage collapse when a task's rollouts all reach the same outcome, and unreliable evaluation signals that judge each skill by one noisy score. To address these challenges, we propose R$^2$Orch, a new paradigm of Recursive Self-Play Orchestration: an inner loop of self-play that improves the orchestrator on self-chosen tasks, and an outer loop that improves how the inner loop improves. To support this paradigm, we introduce Recursive Learnability, a training objective that rewards a Proposer for choosing tasks whose rollouts keep reaching different outcomes as skills are withdrawn. Built on this objective, we further propose Skill Withdrawal, in which the outer loop withdraws each skill once a controlled evaluation within self-play shows that the orchestrator performs as well without it. Moreover, the two loops form one cycle: failures are distilled into skills, skills into parameters, and withdrawals lead the Proposer to new failures, so the orchestrator moves toward zero-skill deployment. Experimental results on twelve datasets show that R$^2$Orch consistently outperforms baselines across question answering, math, code, and interactive tasks. Our code is available at https://github.com/beita6969/R2Orch.