On Re-Encoding Short-Term Memory of Large Language Models in Conversations
Keywords: LLM, misinformation correction, zero-shot self-correction
TL;DR: We present the KEIC task for LLMs to update their knowledge based on user corrections, construct a 1,781 human-labeled dataset under this framework, and propose a structured approach, including a theoretical algorithm for self-correction.
Abstract: Large language models (LLMs), such as GPT-4, are adept at generating coherent and fluent responses within conversational contexts.
However, there has been a paucity of comprehensive research exploring LLMs to dynamically update their knowledge in response to corrections of misinformation provided by users during dialogue sessions.
In this paper, we present a novel framework termed Knowledge Editing In Conversation (KEIC), along with an accompanying dataset, devised to assess the efficacy of LLMs in aligning the user update in an in-context setting, given the previous chat history containing a false statement that conflicts with the subsequent user update.
Through in-depth investigations, we observe that the contemporary LLMs exhibit a modicum of proficiency in this task.
To enhance their in-context knowledge editing abilities, we propose a structured strategy to handle the information update for LLMs in a multi-turn conversation.
We demonstrate that our approach is effective and suggest insights for research communities in this emerging and essential issue.
Supplementary Material: zip
Primary Area: datasets and benchmarks
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 454
Loading