Hi-Agent: Hierarchical Vision-Language Agents for Mobile Device Control

ICLR 2026 Conference Submission19119 Authors

Keywords: Reinforcement Learning, Device Control, LLM/VLM Agent, Mobile Agent

Abstract:

Building agents that autonomously operate mobile devices has attracted increasing attention. While Vision-Language Models (VLMs) show promise, most existing approaches rely on direct state-to-action mappings, which lack structured reasoning and planning, and thus generalize poorly to novel tasks or unseen UI layouts. We introduce $\textbf{Hi-Agent}$, a trainable hierarchical vision-language agent for mobile control, featuring a high-level reasoning model and a low-level action model that are jointly optimized. For efficient training, we reformulate multi-step decision-making as a sequence of single-step subgoals and propose a foresight advantage function, which leverages execution feedback from the low-level model to guide high-level optimization. This design alleviates the path explosion issue encountered by Group Relative Policy Optimization (GRPO) in long-horizon tasks and enables stable, critic-free joint training.
Hi-Agent achieves a new State-Of-The-Art (SOTA) $\textbf{87.9\%}$ task success rate on the Android-in-the-Wild (AitW) benchmark, significantly outperforming prior methods across three paradigms: prompt-based (AppAgent: 17.7\%), supervised (Filtered BC: 54.5\%), and reinforcement learning-based (DigiRL: 71.9\%). It also demonstrates competitive zero-shot generalization on the ScreenSpot-v2 benchmark. On the more challenging AndroidWorld benchmark, Hi-Agent also scales effectively with larger backbones, showing strong adaptability in high-complexity mobile control scenarios.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 19119