Go to ACL ARR 2025 July homepageA Detailed Factor Analysis for the Political Compass Test: Navigating Ideologies of Large Language Models
Abstract: Political Compass Test (PCT) or similar questionnaires have long been used to quantify LLM's political leanings. Building on a recent line of work that examines the validity of PCT tests, we demonstrate that variation in standard generation parameters does not significantly impact a model's PCT scores. However, external factors such as prompt variations and fine-tuning individually and in combination affect the same. Finally, we demonstrate that when models are fine-tuned on text datasets with higher political content than others, the PCT scores are not differentially affected. We also generalize these findings to a similar popular test. This calls for a thorough investigation into the validity of PCT and similar tests, as well as the mechanism by which political leanings are encoded in LLMs.
Paper Type: Short
Research Area: Ethics, Bias, and Fairness
Research Area Keywords: Political leaning, LLM, Political Compass Test, 8 values
Contribution Types: Model analysis & interpretability
Languages Studied: English
Submission Number: 1035
Loading