Through the LLM Looking Glass: A Socratic Self-Assessment of Donkeys, Elephants, and Markets

ACL ARR 2025 February Submission7448 Authors

Abstract:

While detecting and avoiding bias in LLM-generated text is becoming increasingly important, media bias often remains subtle and subjective, making it particularly difficult to identify and mitigate. In this study, we assess media bias in LLM-generated content and their ability to detect subtle ideological bias using two datasets, PoliGen and EconoLex, respectively, covering political and economic discourse. We evaluate eight widely used LLMs by prompting them to generate articles and analyze their ideological preferences via self-assessment, eliminating
interpretations regarding the subjectivity of media bias. Our results reveal a consistent Democratic preference over Republican across all models. Conversely, in economic topics, biases vary among Western LLMs, while those developed in China lean more strongly toward socialism.

Paper Type: Short

Research Area: Ethics, Bias, and Fairness

Research Area Keywords: media-bias, generated, self-assessment, socratic

Contribution Types: Model analysis & interpretability, Data resources

Languages Studied: English

Submission Number: 7448