Multi-FAct: Assessing Factuality of Multilingual LLMs using FActScore

Sheikh Shafayat, Eunsu Kim, Juhyun Oh, Alice Oh

Research Area: Evaluation, Science of LMs, LMs for everyone, LMs and the world

Keywords: multilinguality, factuality, knowledge representation

TL;DR: We fact check multilingual generations of LLMs

Abstract:

Evaluating the factuality of long-form large language model (LLM)-generated text is an important challenge. Recently there has been a surge of interest in factuality evaluation for English, but little is known about the factuality evaluation of multilingual LLMs, specially when it comes to long-form generation.
This paper systematically evaluates multilingual LLMs' factual accuracy across languages and geographic regions.
We introduce a simple pipeline for multilingual factuality evaluation, by applying FActScore \citep{min2023factscore} for diverse languages. In addition to evaluating multilingual factual generation, we evaluate the factual accuracy of long-form text generation in topics that reflect regional diversity. We also examine the feasibility of running the FActScore pipeline using non-English Wikipedia and provide comprehensive guidelines on multilingual factual evaluation for regionally diverse topics.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html

Author Guide: I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html

Submission Number: 1450