What Suppresses Nash Equilibrium Play in Large Language Models? Mechanistic Evidence and Causal Control

TMLR Paper9209 Authors

Abstract:

LLM agents are known to deviate from Nash equilibria in strategic interactions, but nobody has looked inside the model to understand why,
or asked whether the deviation can be reversed. We do both. Working with four open-source models (Llama-3 and Qwen2.5, 8B to 72B
parameters) playing four canonical two-player games, we first establish the behavioral picture through self-play and cross-play experiments,
then open up the 32-layer Llama-3-8B model and examine what actually happens during a strategic decision. The mechanistic findings are clear. Opponent history is encoded with near-perfect fidelity at the very first layer (96\% probe accuracy) and consumed progressively by later
ones, while Nash action encoding is weak throughout, never exceeding 56\%. There is no dedicated Nash module. Instead, the model privately favors the Nash action through most of its forward pass, but a prosocial override (a bias toward cooperative, other-regarding
behavior rooted in pretraining on human text and further modulated by RLHF) concentrated in the final layers reverses this, reaching 84\% probability of cooperation at layer 30. When we inject a learned Nash direction into the residual stream, the behavior shifts bidirectionally and causally, confirmed through concept clamping. The behavioral experiments surface six scale- and architecture-dependent findings in self-play, the most notable being that chain-of-thought reasoning worsens Nash play in small models but achieves near-perfect Nash play in models above 70B parameters. The cross-play experiments reveal three phenomena invisible in self-play: a small model can unravel the cooperation of any partner simply by defecting early; two large models reinforce each other's cooperative instincts indefinitely; and who moves first in a coordination game determines which Nash equilibrium the system lands on. The central finding is that LLMs do not lack Nash-playing competence. They compute it, then suppress it.

Submission Type: Regular submission (no more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=ESiE14F635&noteId=ESiE14F635

Changes Since Last Submission: Removed all anonymity-breaking content: arXiv preprint link from page 1, author names and affiliations, and acknowledgments section. Two prior desk rejections were purely procedural. No other changes.

Assigned Action Editor: ~Mengmi_Zhang1

Submission Number: 9209