Lita: Light Agent Uncovers the Agentic Coding Capabilities of LLMs

ICLR 2026 Conference Submission16581 Authors

Keywords: LLM, code agent, LLM evaluation, large language model

TL;DR: We showed that complex agent design hide the true ability of LLMs and argue the necessity of lite agents for a fair and truthful evaluation

Abstract:

Large language models (LLMs) are increasingly being applied to programming tasks, ranging from single-turn code completion to autonomous agents.
Current code agent designs frequently depend on complex, hand-crafted workflows and tool sets. However, this reliance on elaborate scaffolding presents several challenges: agent performance becomes overly dependent on prompt tuning and custom design choices, heavy human intervention obscures a model's true underlying capabilities, and intricate pipelines are costly to build and maintain. Furthermore, optimizing complex task prompts increases the risk of data leakage.
Currently, when introducing new models, LLM providers like OpenAI and Anthropic often publish benchmark scores to demonstrate their models' coding proficiency, but keep their proprietary evaluation frameworks confidential.
To address these limitations, we introduce \textit{Lita} (\textbf{Lit}e \textbf{A}gent), which operationalizes \textit{liteness}, a principle of minimizing manual design while retaining the essential elements of a fully autonomous agent. Lita enables a more faithful and unified evaluation without elaborate scaffolding.
Experiments on the Aider polyglot and SWEbench with frontier models demonstrate that Lita achieves competitive or superior performance compared to workflow-based and agentic baselines. Crucially, Lita also consumes fewer tokens and requires significantly less design effort.
Our results suggest that Lita is sufficient to reveal the underlying coding competence of modern LLMs. Finally, we propose the \textbf{Agent Complexity Law}: \textit{the performance gap between agents of varying complexity, from simple to sophisticated designs, will shrink as the core model improves, ultimately converging to a negligible difference.}

Primary Area: foundation or frontier models, including LLMs

Submission Number: 16581