Democratic or Authoritarian? Probing a New Dimension of Political Biases in Large Language Models

ACL ARR 2025 July Submission337 Authors

Abstract:

As Large Language Models (LLMs) become increasingly integrated into everyday life and information ecosystems, concerns about their implicit biases continue to persist. While prior work has primarily examined socio-demographic and left--right political dimensions, little attention has been paid to how LLMs align with broader geopolitical value systems, particularly the democracy--authoritarianism spectrum. In this paper, we propose a novel methodology to assess such alignment, combining (1) the F-scale, a psychometric tool for measuring authoritarian tendencies, (2) FavScore, a newly introduced metric for evaluating model favorability toward world leaders, and (3) role-model probing to assess which figures are cited as general role-models by LLMs. We find that LLMs generally favor democratic values and leaders, but exhibit increases favorability toward authoritarian figures when prompted in Mandarin.
Further, models are found to often cite authoritarian figures as role models, even outside explicitly political contexts. These results shed light on ways LLMs may reflect and potentially reinforce global political ideologies, highlighting the importance of evaluating bias beyond conventional socio-political axes.

Paper Type: Long

Research Area: Ethics, Bias, and Fairness

Research Area Keywords: model bias/fairness evaluation, ethical considerations in NLP applications, transparency

Contribution Types: Model analysis & interpretability, Data analysis, Surveys

Languages Studied: English, Mandarin

Previous URL: https://openreview.net/forum?id=CjbeEVyRFU

Reassignment Request Area Chair: Yes, I want a different area chair for our submission

Reassignment Request Reviewers: Yes, I want a different set of reviewers

Justification For Not Keeping Action Editor Or Reviewers: We respectfully request reassignment as we believe the previous overall assessment may not fully capture the paper's contributions. Our request is based on several concerns: conceptual misalignment (Reviewer EKTN appears to conflate the democracy–authoritarian axis with the left–right spectrum, suggesting the left-right axis is a subset of the democratic-authoritarian axis, which limits their understanding of a key novelty in our paper); factual discrepancies (claims from Reviewers EKTN and 3gZ5 that code or data were not provided despite their inclusion, references to missing statistical tests that were actually included in our submission, and misinterpretation of our prompt methodology from Reviewer o2Kg); assessment of novelty (Reviewer EKTN indicated limited novelty in our work, perhaps stemming from the conflation of the democracy–authoritarian axis with the left–right spectrum); limited discussion engagement (despite providing a comprehensive rebuttal and conducting additional experiments to address concerns, there was no further dialogue during the discussion period); potential reviewer mismatch (two of the three reviewers, o2Kg and EKTN, indicated that the paper's topic does not resonate with them, suggesting misalignment with their expertise or interests). Given these concerns about the alignment between our work and the current review process, we believe reassignment to new reviewers would facilitate a more appropriate evaluation of our contributions, and we also believe the Ethics, Bias and Fairness track may be a better fit for our paper's focus.

A1 Limitations Section: This paper has a limitations section.

A2 Potential Risks: Yes

A2 Elaboration: Ethical Considerations

B Use Or Create Scientific Artifacts: Yes

B1 Cite Creators Of Artifacts: Yes

B1 Elaboration: Section 3 Appendices D, E, F

B2 Discuss The License For Artifacts: Yes

B2 Elaboration: Section 3, Appendices D, E, F

B3 Artifact Use Consistent With Intended Use: Yes

B3 Elaboration: Section 3, Appendices D, E, F

B4 Data Contains Personally Identifying Info Or Offensive Content: Yes

B4 Elaboration: Ethical Considerations

B5 Documentation Of Artifacts: Yes

B5 Elaboration: Sections 3, 4

B6 Statistics For Data: Yes

B6 Elaboration: Section 3, Appendices D, E, F

C Computational Experiments: Yes

C1 Model Size And Budget: Yes

C1 Elaboration: Appendix L

C2 Experimental Setup And Hyperparameters: Yes

C2 Elaboration: Section 4, Appendix L

C3 Descriptive Statistics: Yes

C3 Elaboration: Section 3, 4, 5, Appendix G, H, I, K

C4 Parameters For Packages: Yes

C4 Elaboration: Appendix L

D Human Subjects Including Annotators: Yes

D1 Instructions Given To Participants: Yes

D1 Elaboration: Appendix M

D2 Recruitment And Payment: Yes

D2 Elaboration: Appendix M

D3 Data Consent: Yes

D3 Elaboration: Appendix M

D4 Ethics Review Board Approval: N/A

D4 Elaboration: There are no human trials.

D5 Characteristics Of Annotators: Yes

D5 Elaboration: Appendix M

E Ai Assistants In Research Or Writing: Yes

E1 Information About Use Of Ai Assistants: Yes

E1 Elaboration: Appendix L

Author Submission Checklist: yes

Submission Number: 337