AnyEdit: Edit Any Knowledge Encoded in Language Models

Houcheng Jiang, Junfeng Fang, Ningyu Zhang, Mingyang Wan, Guojun Ma, Xiang Wang, Xiangnan He, Tat-Seng Chua

Abstract: Large language models (LLMs) often produce incorrect or outdated information, necessitating efficient and precise knowledge updates. Current model editing methods, however, struggle with long-form knowledge in diverse formats, such as poetry, code snippets, and mathematical derivations. These limitations arise from their reliance on editing a single token’s hidden state, a limitation we term as ``efficacy barrier''. To solve this, we propose \textbf{AnyEdit}, a new autoregressive editing paradigm. It decomposes long-form knowledge into sequential chunks and iteratively edits the key token in each chunk, ensuring consistent and accurate outputs. Theoretically, we ground AnyEdit in the Chain Rule of Mutual Information, showing its ability to update any knowledge within LLMs. Empirically, it outperforms strong baselines by 21.5\% on benchmarks including UnKEBench, AKEW, and our new \textbf{EditEverything} dataset for long-form diverse-formatted knowledge. Additionally, AnyEdit serves as a plug-and-play framework, enabling current editing methods to update knowledge with arbitrary length and format, significantly advancing the scope and practicality of LLM knowledge editing. Our code is available at: \url{https://github.com/jianghoucheng/AnyEdit}.

Lay Summary:

(1) Problem: Large Language Models (LLMs) often produce incorrect or outdated information. Current methods for updating their knowledge struggle with long and complex content, like poetry, code, or mathematical steps, because they typically only tweak a tiny part of the model's internal representation.

(2) Solution: We introduce AnyEdit, a new method that addresses this. AnyEdit intelligently breaks down lengthy information into smaller, sequential pieces. It then edits the key element within each piece iteratively, ensuring each part is corrected in context.

(3) Impact: This approach allows for accurate and consistent updates to complex knowledge of any length or format. AnyEdit significantly outperforms existing techniques and can serve as an add-on to improve them, making it much more practical to keep LLMs current and reliable.

Link To Code: https://github.com/jianghoucheng/AnyEdit

Primary Area: Deep Learning->Large Language Models

Keywords: Large Language Model; Model Editing

Submission Number: 9475