Latent Knowledge Scalpel: Precise and Massive Knowledge Editing for Large Language Models

ACL ARR 2025 February Submission1705 Authors

Abstract: Large Language Models (LLMs) often retain inaccurate or outdated information from pre-training, leading to incorrect predictions or biased outputs during inference. While model editing methods have been proposed to address this challenge, existing approaches struggle with editing large amounts of factual information simultaneously and may compromise the general capabilities of the models. In this paper, we demonstrate that it is feasible to edit the internal representations of LLMs in a manner akin to editing natural language inputs. Based on this insight, we introduce the Latent Knowledge Scalpel (LKS), an LLM editor that manipulates the latent knowledge of specific entities via a hypernetwork to enable precise and large-scale editing. Experiments conducted on Llama-2 and Mistral show that even with the number of simultaneous edits reaching 10,000, LKS effectively preserves the general abilities of the edited LLMs while surpassing other editors in terms of edit performance.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: model editing

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Submission Number: 1705