Injecting Knowledge from Social Science Journals to Improve Indonesian Cultural Understanding by LLMs

Adimulya Kartiyasa, Bao G. Cao, Boyang Li

Keywords: Cultural understanding of LLM, Indonesia, Indonesian

TL;DR: We present a novel text dataset from Indonesian social science journals, extract the facts related to Indonesian culture and apply RAG, achieving strong performance gains over strong baselines on the IndoCulture benchmark and a new SOTA of 81.4%.

Abstract: Recently there have been intensifying efforts to improve the understanding of Indonesian cultures by large language models (LLMs). An attractive source of cultural knowledge that has been largely overlooked is local journals of social science, which likely contain substantial cultural studies from a native perspective. We present a novel text dataset of journal article passages, created from 151 open-source Indonesian social science journals, called IndoSoSci. We demonstrate an effective recipe for injecting Indonesian cultural knowledge therein into LLMs: extracting the facts related to Indonesian culture, and apply retrieval-augmented generation (RAG) with LLM- generated hypothetical documents as queries during retrieval. The proposed recipe yields strong performance gains over several strong baselines on the IndoCulture benchmark. Additionally, by combining IndoSoSci with Indonesian Wikipedia, we set a new state-of-the-art accuracy on the IndoCulture benchmark.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 6