# CastWorld: Benchmarking Context-Aware Agentic Time Series Forecasting Beyond Predictive Accuracy

DevFeed: [CastWorld: Benchmarking Context-Aware Agentic Time Series Forecasting Beyond Predictive Accuracy](<https://devfeed.tech/articles/castworld-benchmarking-context-aware-agentic-time-series-forecasting-beyond-predictive-accuracy-132117.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15262>)

Author: Bokai Pan, Ruijie Yang, Mingyue Cheng, Jiahao Wang, Xiaoyu Tao

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Time Series](<https://devfeed.tech/topics/time-series.md>), [Agentic AI](<https://devfeed.tech/topics/what-is-agentic-ai.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>), [versioning](<https://devfeed.tech/topics/versioning.md>), [Predictive AI](<https://devfeed.tech/topics/predictive-ai.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [series](<https://devfeed.tech/tags/series.md>), [time](<https://devfeed.tech/tags/time.md>), [time-series](<https://devfeed.tech/tags/time-series.md>), [time-series-forecasting](<https://devfeed.tech/tags/time-series-forecasting.md>)

## AI overview

CastWorld benchmarks context-aware, agentic time series forecasting beyond predictive accuracy. Its suite covers 25 real-world datasets across 8 domains, including external variables, events, and historical cases. A forecasting-focused harness provides 49 tools and tracks forecast versions through an adaptive loop for gathering evidence, revising forecasts, and selecting a final version. Evaluation measures forecast quality, process execution, behavioral trustworthiness, and resource efficiency, alongside system-level predictive accuracy. Experiments found that agents with similar accuracy can behave differently; more interactions do not always improve forecasts, and agents may fail to choose their best earlier prediction. Allowing evidence gathering during refinement improved task-aware scores for all three evaluated models.

## Source excerpt

arXiv:2611.15262v1 Announce Type: cross Abstract: Agentic time series forecasting extends conventional approaches beyond one-shot numerical prediction toward autonomous, context-aware interaction. However, existing benchmarks offer limited integrated support for diverse contextual information, forecasting-specific adaptive interactions, and systematic evaluation of forecasting agents. In this paper, we introduce CastWorld, an agentic benchmark for context-aware time series forecasting. First, its context-rich dataset suite spans 8 domains and 25 real-world datasets, covering external variables, events, and historical cases under a unified context specification. Second, a lightweight, modular forecasting-native harness provides 49 core tools and a state-adaptive loop that tracks forecast versions and enables iterative evidence gathering and forecast revision. Intermediate forecasts can expose information gaps, while newly acquired evidence guides subsequent revisions and final version selection. Third, model-level evaluation assesses forecast quality, process execution, behavioral trustworthiness, and resource efficiency through decision trajectories and forecast-version histories. Also, systems-level evaluation compares predictive accuracy across forecasting systems. Experiments reveal distinct decision behaviors among agents with comparable accuracy. Additional interactions do not necessarily improve forecasts, and agents may fail to select their best previously generated predictions. Allowing evidence acquisition during forecast refinement improves task-aware forecast scores across all three evaluated models. CastWorld provides a reusable foundation for evaluating agentic time series forecasting under controlled conditions.