The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation

Zarreen Reza

Keywords: Multi-agent LLMs, Multi-agent Debate, LLM Evaluation, Psychometric Evaluation, Agent-agent collaboration

Abstract: As Large Language Models (LLMs) transition from static tools to autonomous agents, traditional evaluation benchmarks are insufficient for capturing the emergent social and cognitive dynamics of agentic interaction. To address this gap, we introduce a multi-agent debate framework as a controlled 'social laboratory' to discover and quantify these behaviors. In our framework, LLM agents with distinct personas and incentives deliberate on challenging topics, enabling analysis via a new suite of psychometric and semantic metrics. Our experiments reveal a powerful emergent tendency for agents to seek consensus, consistently reaching high semantic agreement ($\mu > 0.88$). We show that assigned personas induce stable psychometric profiles, with 'Evidence-Driven Analysts' reporting higher cognitive effort and 'Values-Focused Ethicists' showing greater cognitive dissonance; and our qualitative analysis reveals critical dynamics, from the successful de-biasing of toxic topics to unexpected polarization on seemingly benign ones. Finally, we find the moderator's persona can significantly alter debate outcomes by structuring the environment, a key finding for AI alignment. This work provides a blueprint for dynamic, psychometrically-grounded evaluation protocols for the next generation of AI agents.

Submission Number: 186