Multilingual Knowledge Editing with Language-Agnostic Factual Neurons
Abstract: Multilingual knowledge editing (MKE) aims to simultaneously revise factual knowledge across multilingual languages within large language models (LLMs).
However, most existing MKE methods just adapt existing monolingual editing methods to multilingual scenarios, overlooking the deep semantic connections of the same factual knowledge between different languages, thereby limiting edit performance.
To address this issue, we first investigate how LLMs represent multilingual factual knowledge and discover that the same factual knowledge in different languages generally activates a shared set of neurons, which we call language-agnostic factual neurons.
These neurons represent the semantic connections between multilingual knowledge and are mainly located in certain layers.
Inspired by this finding, we propose a new MKE method by locating and modifying Language-Agnostic Factual Neurons (LAFN) to simultaneously edit multilingual knowledge.
Specifically, we first generate a set of paraphrases for each multilingual knowledge to be edited to precisely locate the corresponding language-agnostic factual neurons.
Then we optimize the update values for modifying these located neurons to achieve simultaneous modification of the same factual knowledge in multiple languages.
Experimental results on Bi-ZsRE and MzsRE benchmarks demonstrate that our method outperforms existing MKE methods and achieves remarkable edit performance, indicating the importance of considering the semantic connections among multilingual knowledge.
Paper Type: Long
Research Area: Multilingualism and Cross-Lingual NLP
Research Area Keywords: multilingualism, multilingual representations
Contribution Types: Model analysis & interpretability, NLP engineering experiment
Languages Studied: English, Chinese, Czech, German, Dutch, Spanish, French, Portuguese, Russian, Thai, Turkish, Vietnamese
Submission Number: 4229
Loading