Programming with Pixels: Can Computer-Use Agents do Software Engineering?

ICLR 2026 Conference Submission19204 Authors

Keywords: computer-use agents, evaluation, benchmark, code-generation, multimodal

TL;DR: We introduce ProgrammingwithPixels (PwP), an environment for evaluating computer use agents (CUAs) for software engineering. Our evaluations on the 15-task PwP-Bench reveals current CUA limitations and potential directions for their improvement.

Abstract:

Computer-use agents (CUAs) hold the promise of performing a wide variety of general tasks, but current evaluations have primarily focused on simple scenarios.
It therefore remains unclear whether such generalist agents can automate more sophisticated and specialized work such as software engineering (SWE).
To investigate this, we introduce Programming with Pixels (PwP), the first comprehensive computer-use environment for software engineering, where agents visually control an IDE to perform diverse software engineering tasks.
To enable holistic evaluation, we also introduce PwP-Bench, a benchmark of 15 existing and new software-engineering tasks spanning multiple modalities, programming languages, and skillsets.
We perform an extensive evaluation of state-of-the-art open-weight and closed-weight CUAs and find that when interacting purely visually, they perform significantly worse than specialized coding agents.
However, when the same CUAs are given direct access to just two APIs—file editing and bash operations—performance jumps, often reaching the levels of specialized agents despite having a task-agnostic design.
Furthermore, when given access to additional IDE tools via text APIs, all models show further gains.
Our analysis shows that current CUAs fall short mainly due to limited visual grounding and the inability to take full advantage of the rich environment, leaving clear room for future improvements.
PwP establishes software engineering as a natural domain for benchmarking whether generalist computer-use agents can reach specialist-level performance on sophisticated tasks.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 19204