More Capable, Less Cooperative? When LLMs Fail at Zero-Cost Collaboration

Advait Yadav, Sidney Black, Oliver Sourbut

Keywords: Alignment, Multi-Agent Systems, Safety, LLM Evaluation, Cooperative AI

TL;DR: When instructed to maximize group performance, many LLMs withhold information anyway, revealing that cooperative alignment doesn't automatically emerge from capability and requires explicit incentives or protocols.

Abstract: Large language model (LLM) agents increasingly coordinate in multi-agent systems, yet we lack understanding of where and why cooperation failures may arise. In many real-world coordination problems—from knowledge sharing in organizations to code documentation—helping others carries negligible personal cost while generating substantial collective benefits. However, whether LLM agents cooperate when helping neither benefits nor harms the helper, despite being given explicit instructions to do so, remains unknown. We build a turn-based multi-agent setup designed to study competitive and cooperative behavior in a frictionless multi-agent setup, removing all strategic complexity from cooperation. We find that capability does not predict cooperation: OpenAI o3 achieves only 17\% of optimal collective performance while OpenAI o3-mini reaches 50\%, despite identical instructions to maximize group revenue. Through a causal decomposition that automates one side of agent communication, we separate cooperation failures from competence failures. Testing targeted interventions, we find that explicit protocols double performance for low-competence models, and tiny sharing incentives improve models with weak cooperation. These results demonstrate that even when helping is free and strategically trivial, many LLMs fail to follow the instructed cooperative objectives, requiring interventions based on specific failure modes. Our findings suggest that scaling intelligence alone will not solve coordination problems in multi-agent systems and will require deliberate cooperative design, even when helping costs nothing.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 24502