AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs

ACL ARR 2025 February Submission6573 Authors

Abstract:

User interface understanding with vision-language models (VLMs) has received much attention due to its potential for enhancing software automation.
However, existing datasets used to build UI-VLMs either only contain large-scale context-free element annotations or contextualized functional descriptions for elements at a small scale.
In this work, we propose the AutoGUI pipeline for automatically annotating UI elements with detailed functionality descriptions at scale.
Specifically, we leverage large language models (LLMs) to infer element functionality by comparing UI state changes before and after simulated interactions. To improve annotation quality, we propose LLM-aided rejection and verification, eliminating invalid annotations without human labor.
We construct a high-quality AutoGUI-704k dataset using the proposed pipeline, featuring diverse and detailed functionality annotations that are hardly provided by previous datasets.
Human evaluation shows that we achieve annotation correctness comparable to a trained human annotator. Extensive experiments show that our dataset remarkably enhances VLM's UI grounding capabilities and exhibits significant scaling effects. We also show the interesting potential use of our dataset in UI agent tasks. Please view our data at this anonymous URL: https://huggingface.co/AutoGUI.

Paper Type: Long

Research Area: Multimodality and Language Grounding to Vision, Robotics and Beyond

Research Area Keywords: Dataset & benchmarks,GUI understanding,vision-language model

Contribution Types: NLP engineering experiment, Data resources

Languages Studied: English

Submission Number: 6573