Reasoning Boosts Opinion Alignment in LLMs

ICLR 2026 Conference Submission17872 Authors

Keywords: Opinion modeling, Alignement, LLMs, Reasoning

TL;DR: Using reasoning and reinforcement learning, we improve LLM opinion modeling, highlight remaining biases, and provide a baseline for future research.

Abstract: Opinion modeling aims to capture individual or group political preferences, enabling applications such as digital democracies, where models could help shape fairer and more popular policies. Given their versatility, strong generalization capabilities, and demonstrated success across diverse text-to-text applications, large language models (LLMs) are natural candidates for this task. However, due to their statistical nature and limited causal understanding, they tend to produce biased opinions when prompted naively. In this work, we study whether reasoning can improve opinion alignment. Motivated by the recent advancement in mathematical reasoning enabled by reinforcement learning (RL), we train models to produce profile-consistent answers through structured reasoning. We evaluate our approach on three datasets covering U.S., European, and Swiss politics. Results indicate that reasoning enhances opinion modeling and is competitive with strong baselines, but does not fully remove bias, highlighting the need for additional mechanisms to build faithful political digital twins using LLMs. By releasing both our method and datasets, we establish a solid baseline to support future research on LLM opinion alignment.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 17872