Knowledge Editing with Subspace-Aware Key-Value Mappings

ICLR 2026 Conference Submission17066 Authors

Keywords: Knowledge Editing, Large Language Model, Subspace

TL;DR: Subspace Knowledge Edit (SUIT) identifies and modifies only the subspace of critical features relevant to the edit, dramatically improves knowledge preservation over strong baselines while maintaining high edit efficacy.

Abstract: Knowledge editing aims to efficiently correct factual errors in Language Models (LMs). The popular locate-then-edit approach modifies an MLP layer by finding an optimal mapping between its input vector (key) and output vector (value) that leads to the expression of the edited knowledge. However, existing methods without any constraints on the key and value vectors cause significant perturbations to the edited model. To address this, we propose Subspace Knowledge Edit (SUIT), a method that identifies and modifies only the subspace of critical features relevant to the edit. Our empirical results on LLaMA-3-8B, GPT-J-6B, and Qwen2.5-7B models show that SUIT dramatically improves knowledge preservation over strong baselines while maintaining high edit efficacy. This effectiveness confirms that SUIT successfully identifies the critical subspace for the edit. Further analyses provide additional validation for our approach. The source code and data will be released to the public upon publication of the paper.

Primary Area: interpretability and explainable AI

Submission Number: 17066