ChOiRe: Characterizing and Predicting Human Opinions with Chain of Opinion Reasoning

ACL ARR 2024 April Submission150 Authors

Abstract: Characterizing and predicting human opinions with language models (LMs) is a challenging yet vital task to enhance their grasp of human values, preferences, and beliefs. While prior studies demonstrate the potential to solve this task by adopting personae, the personae often include excessive and irrelevant information that can harm the models’ performance. Therefore, how to effectively employ the personae for LMs remains a significant challenge. We introduce ChOiRe, a novel four-step framework addressing the above challenge by differentially modeling the user’s explicit personae (i.e. demographic or ideological attributes) that are manually declared, and implicit personae inferred from user historical opinions. ChOiRe consists of (i) an LM analyzing the user’s explicit personae to filter out irrelevant attributes; (ii) the LM ranking the implicit persona opinions into a preferential list; (iii) Chain-of-Opinion (CoO) reasoning, where the LM sequentially analyzes the explicit personae and the most relevant implicit personae to perform opinion prediction; (iv) and where ChOiRe executes Step (iii)’s CoO multiple times with increasingly larger lists of implicit personae to overcome insufficient personae information to infer a final result. ChOiRe achieves new state-of-the-art effectiveness with limited inference calls, improving previous techniques significantly by 3.22%. Moreover, ChOiRe’s Steps (i) and (ii) can significantly better fine-tune opinion-aligned models, by up to 18.44%.

Paper Type: Long

Research Area: Efficient/Low-Resource Methods for NLP

Research Area Keywords: Large Language Models, Human-AI Alignment, Prompting

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Approaches low compute settings-efficiency

Languages Studied: English

Submission Number: 150