"Are We Done Yet?": A Vision-Based Judge for Autonomous Task Completion of Computer Use Agents

AAAI 2026 Workshop TrustAgent Submission14 Authors

Keywords: Computer Use Agents, Autonomous evaluation, GUI automation, Task Completion

TL;DR: A Vision-Language Model looks at the final screenshot and task description to judge if a Computer Use Agent (CUA) actually completes the task, if not, then the feedback is fed back to the CUA for retry.

Abstract: Computer Use Agents (CUAs) are designed to autonomously operate digital interfaces, yet they often fail to reliably determine whether a given task has been successfully completed. We present an autonomous evaluation and feedback framework that leverages Vision–Language Models (VLMs) to assess task completion directly from screenshots and task descriptions. Our dataset covers $42$ built-in macOS applications and $1{,}260$ human-labeled tasks, covering a wide range of scenarios. Our framework achieves up to $73\%$ percent classification accuracy in task success detection and yields an average relative improvement of $27\%$ percent in the overall task success rate of CUAs when evaluator feedback is applied. These results demonstrate that vision-based evaluation can serve as an actionable feedback mechanism that significantly improves the reliability and self-correction of autonomous computer-use agents.

Submission Number: 14