Leveraging Large Language Models for Fact-Checking Farsi News Headlines

Shirin Dehghani, Mohammadmasiha Zahedivafa, Zahra Baghshahi, Darya Zare, Sara Yari, Zeynab Samei, Mohammadhadi Aliahmadi, Mahdis Abbasi, Sara Mirzamojtahedi, Sarvenaz Ebrahimi, Meysam Alizadeh

Abstract: The proliferation of misinformation demands the development of automated fact-checking systems. Large language models (LLMs), are increasingly being used for academic, legal, and journalistic content writing. This underscores the critical importance of LLMs in distinguishing between factual accuracy and inaccuracy. Hence, understanding the capacities and limitations of LLMs in fact-checking tasks is essential for their usage in information space. While previous research showed the potential of LLMs in fact-checking English news headlines, the extent to which LLMs work well in other languages are mostly unexplored. In this paper, using data from a local fact-checking website, we investigate the performance of close- and open-source LLMs in fact-checking Farsi news headlines. Our results show that in none of the model combinations, the fact-checking accuracy of LLMs exceeds 55%, which is pretty low compared to results reported for English news. While fine-tuning shows promising results for performance gain, and should be explored further in future research, our results underscore the weakness of LLMs in low-resource languages such as Farsi, even when fine-tuned.