Evaluating Large Language Models for Belief Inference: Mapping Belief Networks at Scale

ACL ARR 2025 May Submission4127 Authors

Abstract: Beliefs are interconnected, influencing how people process and update what they think. To study the interconnectedness of beliefs at scale, we introduce a novel analytical pipeline leveraging a finetuned GPT-4o model to infer belief structures from large-scale social media data. We evaluate the model's performance by (1) comparing it to human annotated data (2) and its inferences to human-generated survey data. Our results show that a fine-tuned GPT-4o model can effectively recover belief structures, allowing for a level of scalability and efficiency that is impossible using traditional survey methods of data collection. This work demonstrates the potential for large language models to perform belief inference tasks and provides a framework for future research on the analysis of belief structures.

Paper Type: Short

Research Area: Computational Social Science and Cultural Analytics

Research Area Keywords: stance detection; NLP tools for social analysis; quantitative analyses of news and/or social media;

Contribution Types: NLP engineering experiment, Data resources, Data analysis

Languages Studied: English

Submission Number: 4127