From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces

Published: 21 Sept 2023, Last Modified: 02 Nov 2023NeurIPS 2023 spotlightEveryoneRevisionsBibTeX
Keywords: instruction following, web tasks, user interface tasks, vision and language, representation learning, reinforcement learning, imitation learning, tree search, language grounding, web agents, computer control
TL;DR: We study GUI-based instruction following with a general observation and action space consisting of pixel-based inputs and low-level actions.
Abstract: Much of the previous work towards digital agents for graphical user interfaces (GUIs) has relied on text-based representations (derived from HTML or other structured data sources), which are not always readily available. These input representations have been often coupled with custom, task-specific action spaces. This paper focuses on creating agents that interact with the digital world using the same conceptual interface that humans commonly use — via pixel-based screenshots and a generic action space corresponding to keyboard and mouse actions. Building upon recent progress in pixel-based pretraining, we show, for the first time, that it is possible for such agents to outperform human crowdworkers on the MiniWob++ benchmark of GUI-based instruction following tasks.
Submission Number: 15350