The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies

ICLR 2026 Conference Submission99 Authors

Keywords: Large Language Model, Multi-Agent System, Social Simulation, Social Science

Abstract: Large Language Models (LLMs) are increasingly used for social simulation, where populations of agents are expected to reproduce human-like collective behavior. However, we find that many recent studies adopt experimental designs that systematically undermine the validity of their claims. From a survey of over 40 papers, we identify six recurring methodological flaws: agents are often homogeneous (Profile), interactions are absent or artificially imposed (Interaction), memory is discarded (Memory), prompts tightly control outcomes (Minimal-Control), agents can infer the experimental hypothesis (Unawareness), and validation relies on simplified theoretical models rather than real-world data (Realism). For instance, GPT-4o and Qwen-3 infer the underlying social-experiments in 52.9% of cases—for example, identifying that their herd behaviors are being tested—despite not being provided with such information, thereby violating the Unawareness principle. We formalize these six requirements as the PIMMUR principles and argue they are necessary conditions for credible LLM-based social simulation. To demonstrate their impact, we re-run five representative studies using a framework that enforces PIMMUR and find that the reported social phenomena frequently fail to emerge under more rigorous conditions. Our work establishes methodological standards for LLM-based multi-agent research and provides a foundation for more reliable and reproducible claims about "AI societies."

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 99