KEIC: A Framework and Dataset to Self-Correcting Large Language Models in Conversations

ICLR 2026 Conference Submission14915 Authors

Keywords: LLMs, self-correction, zero-shot, framework and systematic evaluation for misinformation correction

TL;DR: We present and formalize the KEIC task for LLMs to update their knowledge based on user corrections, construct a 1,781 human-labeled dataset, and propose a structured approach to this task, including an iterative algorithm for self-correction.

Abstract: Large language models (LLMs) are adept at generating coherent and fluent responses within conversational contexts. Recent studies also demonstrate that LLMs can follow the user preference in an extremely long-term setting. Nevertheless, there is still lack of comprehensive research exploring LLMs to dynamically update their knowledge in response to corrections of misinformation provided by users during dialogue sessions. In this paper, we present a unified framework termed Knowledge Editing In Conversation (KEIC), along with a 1,781 human-annotated dataset, devised to assess the efficacy of LLMs in aligning the user update in an in-context setting, wherein the previous chat containing a false statement that conflicts with the subsequent user update. Through systematic investigations on more than 25 LLMs using various prompting and retrieval-augmented generation (RAG) methods, we observe that the contemporary LLMs exhibit a modicum of proficiency in this task. To enhance their self-correction abilities, we propose a structured strategy to handle the information update in a multi-turn conversation. We demonstrate that our approach is effective and suggest insights for research communities in this emerging and essential issue.

Primary Area: datasets and benchmarks

Submission Number: 14915