Attributing Culture-Conditioned Generations to Pretraining Corpora
Keywords: culture bias, pretraining data, memorization, generalization
TL;DR: We uncover reasons of biased culture-centric generations by attributing generated entities to memorization or generalization from pretraining data.
Abstract: In open-ended generative tasks such as narrative writing or dialog interaction, large language models are known to manifest culture biases, showing inadequate knowledge and producing templated generations on less prevalent cultures. Previous works suggest that such biased generations are due to the uneven representation of each culture in pretraining corpora of the language models. In this
work, we study how pretraining data lead to biased culture-conditioned generations via the lens of LLM memorization and generalization, in order to provide more insights on improving the pretraining data and the pretraining procedure of LLMs. We introduce the MEMOed framework (MEMOrization from pretraining document) which determines whether a generation for a culture is due to memorization or generalization. On culture-conditioned generations about food and clothing entities for 110 cultures, we find that for a culture with high frequency in pretraining data, the model can recall more memorized knowledge about the culture; for cultures appearing least frequently, none of their generations contain any entities memorized from pretraining. In addition, we discover that the model prefers generating about entities with extraordinarily high frequency regardless of the conditioned-culture, an indication of overmemorization, where the model demonstrates biases towards frequent terms in pretraining data regardless of its correctness. Our findings show that current LLM generations majorly consist of memorization and un-founded overmemorization. We hope that the MEMOed framework and our insights will inspire more works on attributing model performance on pretraining data.
Primary Area: interpretability and explainable AI
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 13342
Loading