GUIrilla: A Scalable Framework for Automated Desktop UI Exploration

Sofiya Garkot, Maksym Shamrai, Ivan Synytsia, Mariya Hirna

Keywords: Data-Centric Foundation Models, Large-Scale Multimodal Dataset Generation, Automated Data Collection Pipelines, GUI Interaction Data for Vision-Language Models, Human-Computer Interaction for Agent Training

TL;DR: GUIRILLA is a scalable framework that automatically explores GUIs and builds structured interaction datasets, enabling foundation models and GUI agents to learn realistic desktop behavior from large-scale accessibility-driven data.

Abstract: The performance and generalization of foundation models for interactive systems critically depend on the availability of large-scale, realistic training data. While recent advances in large language models (LLMs) have improved GUI understanding, progress in desktop automation remains constrained by the scarcity of high-quality, publicly available desktop interaction data, particularly for macOS. We introduce GUIRILLA, a scalable data crawling framework for automated exploration of desktop GUIs. GUIRILLA is not an autonomous agent; instead, it systematically collects realistic interaction traces and accessibility metadata intended to support the training, evaluation, and stabilization of downstream foundation models and GUI agents. The framework targets macOS, a largely underrepresented platform in existing resources, and organizes explored interfaces into hierarchical MacApp Trees derived from accessibility states and user actions. As part of this work, we release these MacApp Trees as a reusable structural representation of macOS applications, enabling downstream analysis, retrieval, testing, and future agent training. We additionally release macapptree, an open-source library for reproducible accessibility-driven GUI data collection, along with the full framework implementation to support open research in desktop autonomy.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 115