Coding Agents with Multimodal Browsing are Generalist Problem Solvers

Aditya Bharat Soni, Boxuan Li, Xingyao Wang, Valerie Chen, Graham Neubig

Submission Track: Paper Track (up to 8 pages)

Keywords: LLM Agents, AI Agents, Generalizability, Specialist Agents, Coding Agents, Web Agents

TL;DR: OpenHands-Versa is a generalist agent that is capable of solving diverse tasks and demonstrates competitive or superior performance compared to specialist agents on three benchmarks: SWE-Bench Multimodal, GAIA, and The Agent Company.

Abstract: Modern human labor is characterized by specialization; we train for years and develop particular tools that allow us to perform well across a variety of tasks. In addition, AI agents have been specialized for domains such as software engineering, web navigation, and workflow automation. However, this results in agents that are good for one thing and fail to generalize beyond their intended scope because agent developers provide a highly specialized set of tools or make architectural decisions optimized for a specific use case or benchmark. In this work, we ask the question: what is the minimal set of general tools that can be used to achieve high performance across a diverse set of tasks? Our answer is OpenHands-Versa, a generalist agent built with a modest number of general tools: code editing and execution, web search, multimodal web browsing and file access. Unlike existing multi-agent systems that fail to generalize, OpenHands-Versa is a single-agent system that demonstrates superior or competitive performance over leading specialized agents across three diverse and challenging benchmarks: SWE-Bench Multimodal, GAIA, and The Agent Company, outperforming the best-performing previously published results with absolute improvements in success rate of 9.1, 1.3, and 9.1 points respectively. These results demonstrate the feasibility of developing a generalist agent to solve diverse tasks and establish OpenHands-Versa as a strong baseline for future research.

Camera Ready Modification Summary: Added results on GAIA test split, added results for claude-sonnet-4 for all 3 benchmarks, provided details about inference costs and refined the figures and draft.

Submission Number: 22