Beyond Clicking: A Step Towards Generalist GUI Grounding via Text Dragging

Zeyi Liao, Yadong Lu, Boyu Gou, Huan Sun, Ahmed Hassan Awadallah

Keywords: GUI grounding, text dragging

TL;DR: We enhance and evaluate the text dragging capabilities of existing grounding models through an automated data collection pipeline and a comprehensive benchmark.

Abstract:

Graphical user interface (GUI) grounding, the process of mapping human instructions to GUI actions, serves as a fundamental basis to autonomous GUI agents. While existing grounding models achieve promising performance to simulate the mouse click action on various click-based benchmark, another essential mode of mouse interaction, namely dragging, remains largely underexplored. Yet, dragging the mouse to select and manipulate textual content represents a prevelant and important usage in practical GUI scenarios.
To narrow this gap, we first introduce GUIDrag, a diverse dataset of 161K text dragging examples synthesized through a scalable pipeline. To support systematic and robust evaluation, we further construct ScreenDrag, a benchmark with 5,333 examples spanning three levels of interface context, together with three dedicated metrics designed for assessing text dragging capability. Models trained on GUIDrag with an efficient continual training strategy achieve substantial improvements on ScreenDrag, while preserving the original click-based performance on ScreenSpot, ScreenSpot-v2, and OSWorld-G. Our work encourages further research on broader GUI grounding beyond just clicking and paves way toward a truly generalist GUI grounding model.

Primary Area: datasets and benchmarks

Submission Number: 23835