A Detailed Factor Analysis for the Political Compass Test: Navigating Ideologies of Large Language Models

ACL ARR 2025 May Submission5198 Authors

Abstract:

Political Compass Test (PCT) or similar ques-
tionnaires have been used to quantify LLM’s
political leanings. Expanding on a recent line
of work that examine the validity of PCT tests,
we show that variation in standard generation
parameters, perhaps unexpectedly, does not sig-
nificantly affect the models’ PCT scores. How-
ever, external factors such as prompt variations
and fine-tuning individually and in combination
affect the same. Finally, we show that when
models are fine-tuned on text datasets that have
higher political content than others, the PCT
scores are not affected differentially. This calls
for a thorough investigation in the validity of
PCT and similar tests, as well as the mechanism
of encoding of political leanings in LLMs.

Paper Type: Short

Research Area: Ethics, Bias, and Fairness

Research Area Keywords: Political Compass Test, LLM, Political Bias, Political Leaning

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 5198