Computer-Use Agents as Judges for Automatic GUI Design

Kevin Qinghong Lin, Siyuan Hu, Linjie Li, Zhengyuan Yang, Lijuan Wang, Philip Torr, Mike Zheng Shou

Keywords: Computer-Use, Coding, Human-Computer Interface, Multi-agent

TL;DR: Pioneering Agent-centric UI design via Coder–Computer-use agents collaboration

Abstract:

Computer-Use Agents (CUA) are becoming increasingly capable of autonomously operating digital environments through Graphical User Interfaces (GUI). Yet, most GUI remain designed primarily for humans—prioritizing aesthetics and usability—forcing agents to adopt human-oriented behaviors that are unnecessary for efficient task execution.
At the same time, rapid advances in coding-oriented language models (Coder) have transformed automatic GUI design. This raises a fundamental question: \textit{Can CUA as judges to assist Coder for automatic GUI design}?
To investigate, we introduce \textbf{\bench}, a benchmark for \underline{A}utomatic G\underline{UI} development spanning 52 applications across diverse domains. Using language models, we synthesis 1560 tasks that simulate real-world scenarios. To ensure task reliability, we further develop a verifier that programmatically checks whether each task is executable within its environment.
Building on this, we propose a \textbf{Coder–CUA in Collaboration} framework: the Coder acts as Designer, generating and revising websites, while the CUA serves as Judge, evaluating functionality and refining designs. Success is measured not by visual appearance, but by task solvability and CUA navigation success rate.
To turn CUA feedback into usable guidance, we design an \textbf{CUA Dashboard} that compresses multi-step navigation histories into concise visual summaries, offering interpretable guidance for iterative redesign.
By positioning agents as both designers and judges, our framework shifts interface design toward agent-native efficiency and reliability.
Our work takes a step toward shifting agents from passive use toward active participation in digital environments.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 5626