Exposure Bias Mitigation for Self Information Updating of Large Language Models
Keywords: Large Language Models, Knowledge Update, Exposure Bias
Abstract: Current large language models (LLMs) have demonstrated remarkable capabilities in addressing users' requests for various types of information. However, these models are limited by the most recent data available in their pretraining corpora, rendering them incapable of providing up-to-date information. While periodically updating LLM pretraining corpora is possible, the optimal updating strategy remains underexplored. Retraining LLMs from scratch is cost-prohibitive, and the effectiveness of continual fine-tuning on new corpora has not been thoroughly examined. Additionally, current update procedures typically demand significant human involvement to convert the information into more structured format, such as knowledge triples, conversational data or responses with human feedback. In this study, we conduct a comprehensive examination of a novel \problem{} task in LLMs, which only requires the provision of informative text corpora without additional human intervention. For instance, we can use the latest news articles to update the LLMs' existing knowledge. We define the \problem{} task and assess the continual fine-tuning approach for this purpose. We formulate this task as a self knowledge distillation task where the teacher model is the original LLM with a new corpus as the context. We observe that the na\"ive distillation method can be problematic due to LLMs' exposure bias, which prioritizes existing information over new information that we aim to incorporate. When fine-tuned to accommodate instructions related to new information, LLMs tend to rely on pre-existing knowledge, neglecting recent facts and leading to incorrect reasoning chains that ultimately diminish the efficacy of information updates. Based on our theoretical analysis, we propose a straightforward yet effective method to mitigate exposure bias by incorporating the selected relevant facts into training losses. To validate our hypothesis, we develop two datasets to evaluate information updates, one derived from news articles published in March and April 2023 (the latest available news by the time of dataset collection) and the other derived from the Natural Questions benchmark. The latter has been chosen due to its provided link between questions and relevant passages from Wikipedia, which are utilized as the corpus for information updates and evaluation, respectively. Experimental results demonstrate that our proposed approach significantly increases the factual consistency score (on a scale from 0 to 1) by up to 0.16. Furthermore, we perform a preliminary investigation into the forgetting issue associated with this task, unveiling that our method, with a compact replay buffer of only 2.3\% of the training tokens, can significantly alleviate the forgetting problem. This study thus marks a significant stride towards optimizing the procedures for updating LLMs with the latest information, promising enhanced accuracy and efficacy.
Primary Area: representation learning for computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3998
Loading