Who’s the Impostor? Multi‑Agent Social Deduction for Evaluating LLM Social Reasoning

Xiang Fu

Keywords: LLM evaluation, multi‑agent, social reasoning, social‑deduction games

TL;DR: We evaluate 9 LLMs across 90,720 games; vote‑network centrality predicts coalition conversion, and randomized speaking order shows late speakers reduce impostor odds (−1.11 pp); mode effects are suggestive under conservative uncertainty.

Abstract: We present The Impostor Game, a controlled social‑deduction benchmark for evaluating interactive social reasoning in large language models via multi‑agent play. In each four‑player game, three agents share a majority word $w_m$ and one agent receives a related impostor word $w_i$; agents describe their words and then vote to identify the impostor. Across 90{,}720 games (9 models, 5 modes), vote‑network position best explains realized influence: coalition efficiency increases with centrality ($r \approx 0.87$). Performance varies substantially (impostor win 27.8–69.0%), and recognition (detection accuracy) shows a positive cross‑model trend with outcomes ($r \approx 0.61$, $p=0.17$). Speaking order is randomized; a pseudo‑arm ITT indicates that middle/late speaking modestly reduces impostor odds ($-1.11$ pp; 95% CI $[-1.64, -0.55]$), while seat‑index contrasts are descriptive. Despite having more information, team‑aware coordination underperforms team‑blind. Together, interaction signals—votes, outcomes, and the topology of the induced voting network—reveal limitations in social reasoning and coordination that are not captured by single‑agent evaluations.

Submission Number: 199