How can we assess human-agent interactions? Case studies in software agent design

ICLR 2026 Conference Submission13759 Authors

Keywords: evaluation, human-AI interaction, coding agent

TL;DR: Benchmarks don't tell the full picture about agent utility to humans.

Abstract: LLM-powered agents are both a promising new technology and a source of complexity, where choices about models, tools, and prompting can affect their usefulness. While numerous benchmarks measure agent accuracy across domains, they mostly assume full automation, failing to represent the collaborative nature of real-world use cases. In this paper, we make two major steps towards the rigorous assessment of human-agent interactions. First, we propose a framework for more efficient human-centric evaluation of agent designs, which comprises collecting user feedback, training an ML model to predict user satisfaction, and computing results by combining human satisfaction ratings with model-generated pseudo-labels. Second, we deploy the framework on a large-scale web platform built around the open-source software agent OpenHands, collecting in-the-wild usage data across over 15k users. We conduct case studies around how three agent design decisions---choice of LLM backbone, planning strategy, and memory mechanisms---impact developer satisfaction rates, yielding practical insights for software agent design. We also show how our framework can lead to more robust conclusions about agent design, reducing confidence intervals by 40\% compared to a standard A/B test. Finally, we find substantial discrepancies in-the-wild results with benchmark performance (e.g., the anti-correlation between results comparing claude-sonnet-4 and GPT-5, underscoring the limitations of benchmark-driven evaluation. Our findings provide guidance for evaluations of LLM agents with humans and identify opportunities for better agent designs.

Primary Area: datasets and benchmarks

Submission Number: 13759