# screensuite

Benchmarking suite for evaluating graphical user interface agents.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents

DevFeed: [ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents](<https://devfeed.tech/articles/shopgym-an-integrated-framework-for-realistic-simulation-and-scalable-benchmarking-of-e-commerce-web-agents-134983.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.16116>)

Author: Chinmay Savadikar, Mingyu Zhao, Yuanzheng Zhu, Han Li, Shuang Xie, Alberto Castelo, Tianfu Wu, Lingyun Wang

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [screensuite](<https://devfeed.tech/topics/screensuite.md>), [gui-agents](<https://devfeed.tech/topics/gui-agents.md>)

Tags: [agent-evaluation](<https://devfeed.tech/tags/agent-evaluation.md>), [agents](<https://devfeed.tech/tags/agents.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [categories](<https://devfeed.tech/tags/categories.md>), [commerce](<https://devfeed.tech/tags/commerce.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [data](<https://devfeed.tech/tags/data.md>), [e-commerce](<https://devfeed.tech/tags/e-commerce.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [failure](<https://devfeed.tech/tags/failure.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generation](<https://devfeed.tech/tags/generation.md>), [graph](<https://devfeed.tech/tags/graph.md>), [simulation](<https://devfeed.tech/tags/simulation.md>), [web](<https://devfeed.tech/tags/web.md>)

### AI overview

ShopGym addresses the reproducibility and coverage tradeoff in evaluating e-commerce web agents by pairing ShopArena, which turns live seed storefronts into validated, resettable sandbox shops, with ShopGuru, which generates tasks across seven skill categories grounded in each shop's catalog, navigation, policies, and interactions. In tests covering 224 tasks across six sandbox shops, synthetic shops retained key structural properties, and agent success rates on synthetic twins closely tracked live shops, including per-skill profiles and failure modes. Generating ten shops cost about $60 per shop, with human effort declining as the pipeline scaled.

### Source excerpt

arXiv:2605.16116v2 Announce Type: replace Abstract: Developing and evaluating e-commerce web agents requires environments that preserve meaningful task structure while enabling controllable, reproducible, and scalable scientific comparison. Existing methods force a tradeoff: live storefronts provide realism but are non-stationary, difficult to inspect, and irreproducible, while hand-built sandbox benchmarks provide control but cover only a narrow range of layouts, catalogs, policies, and interaction patterns. We argue that the core bottleneck is methodological: the field lacks a scalable way to construct evaluation settings that are simultaneously realistic, diverse, controllable, inspectable, and reproducible. We introduce ShopGym, an integrated framework for realistic simulation and scalable benchmarking of e-commerce web agents. Its simulation layer, ShopArena, converts live seed storefronts into self-contained sandbox shops through anonymized shop specifications and a staged, validated generation process. On top of these simulated storefronts, ShopGuru synthesizes benchmark tasks across seven skill categories, grounding each task in the shop's catalog, navigation structure, policies, and interaction affordances. Together, ShopArena and ShopGuru produce self-contained, resettable, inspectable, and stable evaluation artifacts that preserve structural properties and agent-evaluation signals relevant to shopping tasks. We validate the framework through graph-based structural analysis and agent-based behavioral evaluation with 224 generated tasks across six sandbox shops (three with synthetic data, three with real data). Results show that synthetic shops preserve key structural properties of live storefronts, and that agent success rates on synthetic twins closely track live storefronts, with matching per-skill profiles and failure modes. Generating ten shops shows the pipeline scales at about $60 per shop with declining human effort.

## Towards Robust Long-Lived Agents: Benchmarking Experience Lifecycles in Evolving Environments

DevFeed: [Towards Robust Long-Lived Agents: Benchmarking Experience Lifecycles in Evolving Environments](<https://devfeed.tech/articles/towards-robust-long-lived-agents-benchmarking-experience-lifecycles-in-evolving-environments-118328.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05450>)

Author: Shijue Huang, Hangyu Guo, Hengheng Zhang, Guo An, Zhitao He, Qi Tian, Yiren Feng

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [screensuite](<https://devfeed.tech/topics/screensuite.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [agents](<https://devfeed.tech/tags/agents.md>), [approach](<https://devfeed.tech/tags/approach.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [builds](<https://devfeed.tech/tags/builds.md>), [but](<https://devfeed.tech/tags/but.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [change](<https://devfeed.tech/tags/change.md>), [changes](<https://devfeed.tech/tags/changes.md>), [complete](<https://devfeed.tech/tags/complete.md>), [construction](<https://devfeed.tech/tags/construction.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [effective](<https://devfeed.tech/tags/effective.md>), [environment](<https://devfeed.tech/tags/environment.md>), [environments](<https://devfeed.tech/tags/environments.md>), [experience](<https://devfeed.tech/tags/experience.md>), [robust](<https://devfeed.tech/tags/robust.md>)

### AI overview

ReScopeBench evaluates how long-lived agents acquire, reuse, and revise experience across task streams in 18 interactive environments, using silent environmental changes to test whether saved knowledge stays current. It separates structural experience about how environments are organized and accessed from rule experience about reusable principles and constraints. Across 20 models, retained experience improved task performance, but agents struggled to keep it current as environments changed. Correctly recording required experience did not always lead to task success, pointing to both selective maintenance and effective use as challenges. The evidence provided is an abstract, so it does not detail model-specific results or benchmark limitations.

### Source excerpt

arXiv:2611.05450v1 Announce Type: new Abstract: As agents become more capable, they are moving beyond one-off tasks toward sustained assistance across successive user goals. Such assistance requires agents to discover how unfamiliar environments work, reuse what they learn across tasks, and keep that experience current as environments evolve. Existing benchmarks increasingly study experience reuse and adaptation, but provide limited insight into which parts of accumulated experience agents revise, retain, or lose. To address this gap, we introduce ReScopeBench, a benchmark for the experience lifecycle across 18 interactive environments. It distinguishes structural experience about how an environment is organized and accessed from rule experience about reusable principles and constraints that guide decisions across tasks. To study how experience is acquired, reused, and revised, we propose an experience-first construction approach. It builds task streams around shared experience and introduces silent environmental changes. We evaluate task performance and track changes in saved experience throughout each stream. Across 20 models, we find that retained experience improves task performance, but agents struggle to keep it current as environments change. Even when the required experience is correctly recorded, agents do not always complete tasks successfully. These findings identify selective maintenance and effective use as complementary challenges for long-lived agents.

## MemU-Bench: Benchmarking GUI Agents on Long-Horizon User Interactions

DevFeed: [MemU-Bench: Benchmarking GUI Agents on Long-Horizon User Interactions](<https://devfeed.tech/articles/memu-bench-benchmarking-gui-agents-on-long-horizon-user-interactions-114996.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.00191>)

Author: Yizhou Liu, Zhengxi Lu, Fei Tang, Yiwen Qiu, Yong Du, Tongbo Chen, Bofan Chen, Tao Jiang, Wenhao Xu, Yunfeng Shu, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [screensuite](<https://devfeed.tech/topics/screensuite.md>), [gui-agents](<https://devfeed.tech/topics/gui-agents.md>), [computer-use](<https://devfeed.tech/topics/computer-use.md>)

Tags: [bench](<https://devfeed.tech/tags/bench.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [gui](<https://devfeed.tech/tags/gui.md>), [gui-agents](<https://devfeed.tech/tags/gui-agents.md>), [horizon](<https://devfeed.tech/tags/horizon.md>), [memory](<https://devfeed.tech/tags/memory.md>), [mobile](<https://devfeed.tech/tags/mobile.md>), [structured](<https://devfeed.tech/tags/structured.md>), [training](<https://devfeed.tech/tags/training.md>), [user-interface](<https://devfeed.tech/tags/user-interface.md>)

### AI overview

MemU-Bench evaluates whether mobile GUI agents can recover omitted details from long interaction histories. It contains 118 vague tasks based on 29 user trajectories; outcomes are checked against serialized app databases, with Interaction Efficiency and Success weighted by Path Length (SPL) complementing Success Rate to measure costly cross-app searching. Without history, Gemini-3.1-Pro reaches 16.4% SPL, while end-to-end GUI agents score below 5%. The training-free AppMem framework stores history in a structured, app-centric memory bank, tracks interrupted intents across app switches, and retrieves memories based on inferred target apps. On GPT-5.5, AppMem raises SPL from 15.7% to 66.9%, 9.3 percentage points above a naive summary baseline; the abstract reports gains across evaluated backbones.

### Source excerpt

arXiv:2611.00191v1 Announce Type: new Abstract: Existing Graphical User Interface (GUI) benchmarks predominantly rely on precise and stateless user instructions. In everyday usage, however, instructions are often \emph{vague and history-dependent}: users omit the exact details of their past operations because they no longer remember them, leaving the agent to recover the missing context from long and noisy interaction histories. To bridge this evaluation gap, we introduce \textbf{MemU-Bench}, a mobile benchmark of \textbf{118 vague tasks} grounded in 29 long-horizon user trajectories, where agents must distill raw interaction streams into memory and retrieve the missing context on demand, with task outcomes verified against serialized application databases. Alongside Success Rate, we incorporate Interaction Efficiency (IE) and Success weighted by Path Length (SPL) to expose agents that succeed by exhaustively searching across different applications rather than by \emph{genuinely recalling} the history. Evaluations show that without user history, state-of-the-art systems struggle under such ambiguity: the best-performing Gemini-3.1-Pro achieves an SPL of only 16.4\%, and end-to-end GUI agents fall below 5\%. To address this, we further propose \textbf{AppMem}, a training-free framework that distills user history into a \emph{structured, app-centric} memory bank, tracks interrupted intents across application switches, and selectively retrieves relevant memories via target-app inference. AppMem raises the SPL of GPT-5.5 from 15.7\% to 66.9\% (\textbf{+51.2\%}), outperforms a naive summary baseline by 9.3\%, and delivers consistent gains across evaluated backbones.

## ANGLER: Verifier-Grounded Benchmarking with an Adaptive Attacker for Web Agents

DevFeed: [ANGLER: Verifier-Grounded Benchmarking with an Adaptive Attacker for Web Agents](<https://devfeed.tech/articles/angler-verifier-grounded-benchmarking-with-an-adaptive-attacker-for-web-agents-111217.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.41272>)

Author: Yini Huang, Aiwen LU, Zhen Sun, Yi Han, Yuzhi Zhao, Jiaheng Wei

Published: 2026-11-02T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [screensuite](<https://devfeed.tech/topics/screensuite.md>), [agentic browsing](<https://devfeed.tech/topics/agentic-browsing.md>), [ai-assisted attacks](<https://devfeed.tech/topics/ai-assisted-attacks.md>), [AI Bots](<https://devfeed.tech/topics/ai-bots.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [agents](<https://devfeed.tech/tags/agents.md>), [attack](<https://devfeed.tech/tags/attack.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [red-teaming](<https://devfeed.tech/tags/red-teaming.md>), [reef](<https://devfeed.tech/tags/reef.md>), [security](<https://devfeed.tech/tags/security.md>), [security-vulnerabilities](<https://devfeed.tech/tags/security-vulnerabilities.md>), [verification](<https://devfeed.tech/tags/verification.md>), [web](<https://devfeed.tech/tags/web.md>)

### AI overview

REEF evaluates web-agent security through state-based verification across 9 websites and 5 attack types, pairing benign tasks with malicious objectives delivered at user or environment level. Its verifiers feed execution outcomes into ANGLER, which adaptively mutates and selects LLM-generated lures, prioritizing near-miss attacks instead of using a fixed set. Across 8 VLM-based agents, ANGLER achieved a 20.36% attack success rate versus 15.24% without adaptation. On the same benign tasks, generated attacks lowered benign task success from 61.9% in attack-free runs to 44.4%.

### Source excerpt

arXiv:2610.41272v1 Announce Type: new Abstract: Web agents are increasingly deployed to complete tasks autonomously on real websites, where adversarially injected content can induce them to perform unauthorized actions. Existing evaluations primarily use fixed attacks or focus on a single attack type, and evaluate attacks at the text level rather than through real state changes, thus underestimating how vulnerable agents are to diverse attacks. To address the gap, we introduce REEF, a benchmark of red-teaming executable environments with faithful verification for measuring the security vulnerabilities of web agents. REEF covers 9 verifier-grounded websites spanning travel, commerce, productivity, media, and communication, and pairs every authentic benign task with a malicious objective delivered through 5 attack types at both the user and the environment level. REEF's state-based verifiers provide the execution feedback that drives ANGLER, an adaptive near-miss-guided lure evolution framework for red-teaming. Rather than evaluating a fixed attack set, ANGLER uses outcomes from prior rollouts to guide LLM-based attack mutation and candidate selection, prioritizing near-miss seeds during refinement. Across 8 VLM-based web agents, ANGLER achieves an end-to-end attack success rate (ASR) of 20.36%, compared with 15.24% for the non-adaptive setting. Compared with attack-free execution on the same benign tasks, attacks generated by ANGLER reduce benign task success rate (BSR) from 61.9% to 44.4%.

## AgentProofBench: Benchmarking Reusable Verifiers for Web Agents

DevFeed: [AgentProofBench: Benchmarking Reusable Verifiers for Web Agents](<https://devfeed.tech/articles/agentproofbench-benchmarking-reusable-verifiers-for-web-agents-101783.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.34634>)

Author: Jingjie Ning, Xueqi Li, Yi Lu, Yibo Kong

Published: 2026-10-27T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [screensuite](<https://devfeed.tech/topics/screensuite.md>), [Web](<https://devfeed.tech/topics/web.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ma](<https://devfeed.tech/tags/cs-ma.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [execution](<https://devfeed.tech/tags/execution.md>), [verification](<https://devfeed.tech/tags/verification.md>), [web](<https://devfeed.tech/tags/web.md>), [web-services](<https://devfeed.tech/tags/web-services.md>)

### AI overview

AgentProofBench evaluates reusable Web-agent verifiers by linking tests of a frozen acceptance rule to executions from identical saved openings. Across 97 tasks and four services, DeepSeek Flash and Pro retained 82/96 and 79/96 source programs in complete-panel and decision-replay tests; paired reuse-versus-rebuild BA intervals were [-5.47, 1.57] and [-10.16, 0.00] percentage points. In 22 execution families, both saved openings were delivered correctly for 32/33 assignments with error-free panels, compared with 1/11 where panels had errors; all reference-defined gate regressions and withheld deliveries were in the latter group. Neutral continuation had the highest correct delivery under the tested generic feedback. The released instructions, states, programs, reference rules, and trajectories link verifier-test failures to execution outcomes.

### Source excerpt

arXiv:2610.34634v1 Announce Type: cross Abstract: Reusable Web-agent verification requires a rule to follow changing requirements and support subsequent execution. We introduce AgentProofBench, which connects paired tests of a frozen acceptance rule to execution from identical saved openings. The resource spans 97 tasks, 79 families, and four Web services. Experiments with DeepSeek Flash/Pro establish two linked measurements. Across 48 families, complete-panel and decision-replay tests retain 82/96 Flash and 79/96 Pro source programs; paired reuse-rebuild BA intervals are [-5.47, 1.57] and [-10.16, 0.00] percentage points. Across 22 execution families, deployment-state panel correctness separates program assignments whose two saved openings are both delivered correctly: 32/33 assignments with error-free panels, versus 1/11 with panel errors. All reference-defined program-gate regressions and withheld deliveries occur in the latter group. Neutral continuation attains the highest correct delivery under the tested generic feedback, providing a concrete baseline for measuring the value added by a gate. Released instructions, states, programs, reference rules, and trajectories connect each test failure to its execution consequences. AgentProofBench makes reusable verification assessable at both the requirement and policy levels.

## ScreenSuite - The most comprehensive evaluation suite for GUI Agents!

DevFeed: [ScreenSuite - The most comprehensive evaluation suite for GUI Agents!](<https://devfeed.tech/articles/screensuite-the-most-comprehensive-evaluation-suite-for-gui-agents-7468.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/screensuite>)

Author: Amir Mahla; Aymeric Roucher; Thomas Wolf

Published: 2025-06-06T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [screensuite](<https://devfeed.tech/topics/screensuite.md>), [gui-agents](<https://devfeed.tech/topics/gui-agents.md>), [computer-use](<https://devfeed.tech/topics/computer-use.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [Claude](<https://devfeed.tech/topics/claude.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [computer-use](<https://devfeed.tech/tags/computer-use.md>), [docker](<https://devfeed.tech/tags/docker.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [gui](<https://devfeed.tech/tags/gui.md>), [gui-agents](<https://devfeed.tech/tags/gui-agents.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [screensuite](<https://devfeed.tech/tags/screensuite.md>), [smolagents](<https://devfeed.tech/tags/smolagents.md>), [virtual-machines](<https://devfeed.tech/tags/virtual-machines.md>), [vlm](<https://devfeed.tech/tags/vlm.md>), [vlms](<https://devfeed.tech/tags/vlms.md>)

### AI overview

ScreenSuite is an open benchmarking and evaluation suite for GUI agents. It unifies 13 benchmarks covering perception, grounding, single-step actions, and multi-step agentic capabilities for Vision Language Models, with support for remote desktop sandboxes and Ubuntu or Android virtual machines launched in Docker.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.