Personalizing GUI Agents with Human-like Contrastive Interaction Trace

ACL ARR 2026 January Submission10565 Authors

Keywords: GUI Automation, Personalized Agent, Large Language Model

Abstract: Personalized GUI agents show strong potential for assisting users in complex digital environments, yet learning fine-grained user preferences from limited interaction data remains a significant challenge. Existing approaches often rely on manual prompt engineering or synthetic interaction data that fail to capture realistic human behaviors, limiting their robustness and generalization. In this work, we propose PACUT, a novel approach for personalizing GUI agents through human-like contrastive interaction traces. PACUT introduces a multi-agent self-refinement process that iteratively infers and consolidates user profiles from observed GUI interactions, enabling the synthesis of behaviorally realistic yet preference-divergent contrastive traces. Building on this supervision, we fine-tune GUI agents using supervised learning and Direct Preference Optimization to explicitly distinguish preferred behaviors from profile-inconsistent alternatives. To support systematic evaluation, we collect a new benchmark dataset with user-specific mobile GUI interaction traces. Extensive experiments across diverse LLM backbones demonstrate that PACUT consistently outperforms strong baselines and ablation variants.

Paper Type: Long

Research Area: AI/LLM Agents

Research Area Keywords: Autonomous agents, multi-agent systems, environment interaction, grounded agents

Languages Studied: English

Submission Number: 10565