MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Users

WenHao Wang, Mengying Yuan, Zijie Yu, Guangyi Liu, Rui Ye, Tian Jin, Siheng Chen, Yanfeng Wang

Keywords: GUI Agent, Mobile Agent, User Intention, Federated Learning

Abstract:

The advancement of mobile GUI agents has opened new opportunities for automating tasks on mobile devices. Training these agents requires large-scale high-quality data, which is prohibitively expensive when relying on human labor. Given the vast population of global mobile phone users, if automated data collection from them becomes feasible, the resulting data volume and the subsequently trained mobile agents could reach unprecedented levels. Nevertheless, two major challenges arise: (1) extracting user instructions without human intervention and (2) utilizing distributed user data while preserving privacy.

To tackle these challenges, we propose **MobileA3gent**, a collaborative framework that trains mobile GUI **A**gents using decentralized self-sourced data from diverse users. The framework comprises two components, each targeting a specific challenge: (1) **A**uto-Annotation, which enables the automatic collection of high-quality datasets during users' routine phone usage with minimal cost. (2) FedVLM-**A**, which enhances federated VLM training under non-IID distributions by incorporating adapted global aggregation based on both episode-level and step-level variability. Extensive experiments prove that MobileA3gent achieves superior performance over traditional approaches at only 1% of the cost, highlighting its potential for real-world applications. Our code is publicly available at: https://github.com/wwh0411/MobileA3gent.

Submission Number: 41