PoliSim: Evaluating Large Language Model-based Agents in Politician Simulation
Abstract: The capabilities of Large Language Models (LLMs) to model and imitate humans offer new perspectives for simulating politicians and society. However, the specific aspects and extent to which LLMs can effectively simulate politicians remain unexplored. Previous evaluations have primarily focused on fictional characters and superficial characteristics, such as linguistic styles, while ignoring LLMs' capacity to accurately replicate individuals' complex features, such as their opinions and actions. This paper introduces PoliSim, a novel benchmark designed to comprehensively and objectively assess the effectiveness of politician simulation by LLM-driven agents. Grounded in cognitive behavior theory, PoliSim evaluates simulations across cognition, attitude, and behavior. By utilizing data from 1,000 politicians, PoliSim transforms the information into a unified evaluation framework consisting of multiple-choice and generation questions. We apply PoliSim to various LLMs and simulation schemas to offer insights and directions for future research in realistic agent-based simulations.
Paper Type: Long
Research Area: Resources and Evaluation
Research Area Keywords: benchmarking, LLM-based agent, social simulation
Contribution Types: Model analysis & interpretability, Data resources
Languages Studied: English
Submission Number: 856
Loading