# TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks

DevFeed: [TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks](<https://devfeed.tech/articles/terminalworld-benchmarking-agents-on-real-world-terminal-tasks-120033.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.22535>)

Author: Zhaoyang Chu, Jiarui Hu, Xingyu Jiang, Pengyu Zou, Han Li, Chao Peng, Peter O'Hearn, Earl T. Barr, Mark Harman, Federica Sarro, He Ye

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [Frontier AI](<https://devfeed.tech/topics/frontier-ai.md>), [Terminal](<https://devfeed.tech/topics/terminal.md>), [Phoronix Test Suite](<https://devfeed.tech/topics/phoronix-test-suite.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [commands](<https://devfeed.tech/tags/commands.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [terminal](<https://devfeed.tech/tags/terminal.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

## AI overview

TerminalWorld automatically derives terminal tasks from real-world recordings: processing 80,870 recordings produced 1,530 validated tasks across 18 categories, with 1,280 commands; a 200-task subset received manual review. On that Verified subset, eight models and six agents achieved at most a 62.5% pass rate, indicating difficulty with authentic terminal workflows. Scores had only weak correlation with Terminal-Bench (Pearson r=0.20), suggesting the benchmarks measure distinct capabilities. The supplied text is an abstract, so it does not detail the task-generation method or individual model and agent results.

## Source excerpt

arXiv:2605.22535v2 Announce Type: replace Abstract: We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings. Processing 80,870 terminal recordings, the engine yields a full benchmark of 1,530 validated tasks, spanning 18 real-world categories, ranging from short everyday operations to workflows exceeding 50 steps, and covering 1,280 unique commands. From these, we curate a Verified subset of 200 diverse, manually reviewed tasks. Comprehensive benchmarking on TerminalWorld-Verified across eight frontier models and six agents reveals that current systems still struggle with authentic terminal workflows, achieving a maximum pass rate of only 62.5%. Moreover, TerminalWorld captures real-world terminal capabilities distinct from existing expert-curated benchmarks (e.g., Terminal-Bench), with only a weak correlation to their scores (Pearson r=0.20). The automated engine makes TerminalWorld authentic and scalable by construction, enabling it to evaluate agents in real-world terminal environments as developer practices evolve.