# Predict the Reward, Not the World: World Reward Models for ML Agents

DevFeed: [Predict the Reward, Not the World: World Reward Models for ML Agents](<https://devfeed.tech/articles/predict-the-reward-not-the-world-world-reward-models-for-ml-agents-108079.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.38439>)

Author: Yuxiao Lee, Zherui Yang, Fan Liu, Hao Liu

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [World models](<https://devfeed.tech/topics/world-models.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [based](<https://devfeed.tech/tags/based.md>), [best](<https://devfeed.tech/tags/best.md>), [better](<https://devfeed.tech/tags/better.md>), [but](<https://devfeed.tech/tags/but.md>), [can](<https://devfeed.tech/tags/can.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [decision-model](<https://devfeed.tech/tags/decision-model.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [error](<https://devfeed.tech/tags/error.md>), [execution](<https://devfeed.tech/tags/execution.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [ml](<https://devfeed.tech/tags/ml.md>), [models](<https://devfeed.tech/tags/models.md>), [reward](<https://devfeed.tech/tags/reward.md>), [world](<https://devfeed.tech/tags/world.md>)

## AI overview

World Reward Models (WRMs) choose directly which candidate program will perform better, avoiding the intermediate execution-outcome predictions that can mislead search. The authors' ForeCritic adapts a 9B typed decision model using 5,000 executed program pairs. On 12,438 held-out pairs across 26 ML tasks, it selects the better program 79.4% of the time, compared with 53-62% for general-purpose LLM judges, and its probabilities are calibrated. Integrated into AIDE, ML-Master, and ForeAgent, it reduces mean task time by 4.4-7.5 times while keeping mean scores within 0.01 of baseline or improving them. These are abstract-reported results; the excerpt does not provide experimental details.

## Source excerpt

arXiv:2610.38439v1 Announce Type: new Abstract: Search-based machine-learning (ML) agents improve a solution by writing candidate programs, executing them, and keeping the best. Execution gives the search reliable feedback, but a single run can take hours, and most runs end in programs the agent discards. Recent agents try to save these runs by letting a large language model act as a world model that predicts how each program would perform. However, a wrong prediction feeds the search false information. We argue that an agent does not need to predict the outcome of execution, only the decision that execution would support. We propose World Reward Models (WRMs), which learn from real execution results to choose directly which candidate will perform better, without predicting any intermediate outcome. We prove that a selector which predicts execution from the same inputs cannot outperform the best direct selector, that predicted outcomes add an error which can make a likelihood-trained world model prefer the worse program on every instance, and that cross-entropy on executed winners drives a WRM toward the optimum. Our WRM, ForeCritic, adapts a 9B typed decision model to 5k executed program pairs. On 12,438 held-out pairs from 26 ML tasks, it picks the better program 79.4% of the time, while general-purpose LLM judges reach 53--62%, and its probabilities are calibrated. Inside AIDE, ML-Master, and ForeAgent, it cuts mean task time by 4.4--7.5 times while keeping their mean scores within 0.01 or raising them.