A Comparative Study of Translation Bias and Accuracy in Multilingual Large Language Models for Cross-Language Claim Verification

NeurIPS 2024 Workshop ATTRIB Submission27 Authors

Release Opt Out: No, I don't wish to opt out of paper release. My paper should be released.

Keywords: Multilingual Large Language Models, Translation Bias, Cross-Language Claim Verification, XFACT Dataset, mBERT, Pre-translation, Self-translation, Low-Resource Languages, Accuracy, Political Misinformation, Language Families, Machine Translation, Model Performance, Computational Resources, Large-Scale Experiments

Abstract: The rise of digital misinformation has heightened interest in using multilingual Large Language Models (LLMs) for fact-checking. This study systematically evaluates translation bias and the effectiveness of LLMs for cross-lingual claim verification across fifteen languages from five language families: Romance, Slavic, Turkic, Indo-Aryan, and Kartvelian. Using the XFACT dataset to assess their impact on accuracy and bias, we investigate two distinct translation methods: pre-translation and self-translation. We use mBERT’s performance on the English dataset as a baseline to compare language-specific accuracies. Our findings reveal that low-resource languages exhibit significantly lower accuracy in direct inference due to underrepresentation in the training data. Furthermore, larger models demonstrate superior performance in self-translation, improving translation accuracy and reducing bias. These results highlight the need for balanced multilingual training, especially in low-resource languages, to promote equitable access to reliable fact-checking tools and minimize the risk of spreading misinformation in different linguistic contexts.

Submission Number: 27