Fact or Fiction? Can LLMs be Reliable Annotators for Political Truths?
Track: Technical
Keywords: Large language model, Political Bias
TL;DR: Our work explores using open-source large language models to automate and improve the annotation of political bias, reducing costs and biases associated with manual fact-checking.
Abstract: Political misinformation poses significant challenges to democratic processes, shaping public opinion and trust in media. Manual fact-checking methods face issues of scalability and annotator bias, while machine learning models require large, costly labelled datasets. This study investigates the use of state-of-the-art large language models (LLMs) as reliable annotators for detecting political factuality in news articles. Using open-source LLMs, we create a politically diverse dataset, labelled for bias through LLM-generated annotations. These annotations are validated by human experts and further evaluated by LLM-based judges to assess the accuracy and reliability of the annotations. Our approach offers a scalable and robust alternative to traditional fact-checking, enhancing transparency and public trust in media.
Submission Number: 5
Loading