WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point

ICLR 2026 Conference Submission13249 Authors

Keywords: Computer Agent Benchmark, Graphical User Interface Automation, Computer-Agent Interaction, Multimodal Large Language Model

TL;DR: We present a novel GUI benchmark called WorldGUI, which designs GUI tasks with various initial states to simulate authentic human-computer interactions.

Abstract: GUI agents have achieved outstanding performance in GUI element grounding. However, planning remains highly challenging, especially due to the sensitivity to the initial state of the environment. Specifically, slight differences in the initial state-such as the target software not being open or the interface not being in its default state, often lead to planning errors. This issue is widespread in real application scenarios, but existing benchmarks fail to evaluate it. To address this gap, we introduce WorldGUI, a comprehensive GUI benchmark containing tasks across ten widely used desktop and web applications (e.g., PowerPoint, VSCode, Acrobat), each instantiated with diverse initial states to simulate authentic human–computer interactions. Complementing this, we propose GUI-Thinker, a universal framework that unifies three core modules: Planner-Critic for high-level plan refinement, Step-Check for intermediate verification, and Actor-Critic for action-level optimization to proactively detect and correct errors. Experimental evaluation shows that WorldGUI-Agent outperforms the outstanding existing model (Claude-3.5-Sonnet Claude Computer Use) by 12.4% in success rate on WorldGUI, and achieves a 31.2% overall success rate on WindowsAgentArena, surpassing the prior state-of-the-art by 11.7%. Our analysis further reveals that dynamic augmentation tasks and desktop environments pose substantial hurdles, underscoring the necessity of adaptive planning and feedback-driven execution for advancing real-world GUI automation.

Primary Area: datasets and benchmarks

Submission Number: 13249