Political-LLM: Large Language Models in Political Science

TMLR Paper5781 Authors

Abstract: Political science is undergoing a significant transition as large language models (LLMs) gain traction in tasks such as election forecasting, policy assessment, and misinformation detection. While LLMs advance political research, they also pose challenges, including but not limited to societal biases (e.g., partisan skew in political sentiment analysis), ethical concerns (e.g., misinformation propagation in automated legislative summarization), and scalability limitations (e.g., inefficiencies in adapting general LLMs for real-time election forecasting). In this work, we—an interdisciplinary team bridging computer science and political science—take an initial step towards systematically understanding how LLMs can be integrated in political science by introducing the principled conceptual framework named Political-LLM. Specifically, our approach begins with a taxonomy that divides normative political science (NPS) and positive political science (PPS), a method of classification that is deeply rooted in the foundation of classical political science research. By grounding the framework in this perspective, we provide a structured view for organizing previous work, pinpointing critical challenges, and uncovering opportunities to promote both empirical research and responsible applications of LLMs. As a case study, we perform empirical experiments using the ANES benchmark to evaluate state-of-the-art LLMs through a voting simulation task, focusing on their abilities to generate relevant political features and expose inherent biases. This study highlights how to employ our principled taxonomy as the guidance of specific research problems in this interdisciplinary field, while also provides an vivid and understandable example for general audience to deepen their comprehension on Political-LLM framework. Finally, we outline key challenges and future directions, emphasizing domain-specific dataset development, careful attention to issues such as bias and opaque modeling processes, acknowledgment of non-scalability constraints, the value of expert involvement, and the importance of proprietary evaluation criteria that meet the needs of this field. Political-LLM is intended as a Guidebook for researchers seeking to apply Artificial Intelligence in political science with care and impact.

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Manuel_Gomez_Rodriguez1

Submission Number: 5781