Balancing Knowledge Updates: Toward Unified Modular Editing in LLMs

刘嘉豪, Zijian Wang, Zhao Kuo, 胡冬

Abstract:

Knowledge editing has emerged as an efficient approach for updating factual knowledge in large language models (LLMs), typically achieved by first locating key knowledge-storage modules and then modifying their parameters.
However, most existing methods focus exclusively on updating the weights of Multi-Layer Perceptron (MLP) modules, which are commonly identified as the primary repositories of factual information.
Other important components, such as attention (Attn) modules—one of the core modules in LLMs—are often ignored during editing.
This biased allocation of updates can leave residual outdated knowledge in the model and limit the effectiveness of knowledge editing.
In this paper, we conduct comprehensive and systematic knowledge localization experiments on advanced LLMs, revealing that Attn modules play a substantial role in factual knowledge storage and retrieval, especially in earlier layers.
Building on these insights, we propose \textit{IntAttn-Edit}, a novel method that extends the associative memory paradigm to jointly update both MLP and Attn modules.
Our approach employs a knowledge balancing strategy that proportionally allocates update magnitudes based on each module’s measured contribution to knowledge storage.
Extensive experiments on popular benchmarks demonstrate that \textit{IntAttn-Edit} consistently achieves superior results over existing methods, delivering higher edit success, improved generalization, and robust knowledge preservation.
Further empirical analysis shows that our knowledge balancing strategy enables the editing performance to remain within the optimal range across different settings.

Submission Number: 88