WilKE: Wise-Layer Knowledge Editor for Lifelong Knowledge Editing

Anonymous

Abstract: Knowledge editing aims to rectify inaccuracies in large language models (LLMs) without costly retraining for outdated or erroneous knowledge. However, current knowledge editing methods primarily focus on single editing, failing to meet the requirements for lifelong editing\footnote{In this paper, lifelong editing is synonymous with lifelong knowledge editing.}. This study reveals a performance degradation encountered by knowledge editing in lifelong editing, characterized by toxicity buildup and toxicity flash, with the primary cause identified as pattern unmatch. We introduce a knowledge editing approach named WilKE, which selects editing layer based on the pattern matching degree of editing knowledge across different layers. Experimental results demonstrate that, in lifelong editing, WilKE exhibits an average improvement of 46.2\% and 67.8\% on editing GPT2-XL and GPT-J relative to state-of-the-art knowledge editing methods.

Paper Type: long

Research Area: NLP Applications

Contribution Types: Model analysis & interpretability, Data analysis

Languages Studied: English

Preprint Status: We are considering releasing a non-anonymous preprint in the next two months (i.e., during the reviewing process).

A1: yes

A1 Elaboration For Yes Or No: In Section 8

A2: yes

A2 Elaboration For Yes Or No: In Section 9

A3: yes

A3 Elaboration For Yes Or No: Abstract and Section 1 Introduction

B: yes

B1: yes

B1 Elaboration For Yes Or No: In Section 4, Section 5 and Section 6

B2: yes

B2 Elaboration For Yes Or No: In Section 4, Section 5 and Section 6

B3: yes

B3 Elaboration For Yes Or No: In Section 4, Section 5 and Section 6. MIT License gives permission to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software.

B4: no

B4 Elaboration For Yes Or No: The dataset we applied is a commonly used open-source benchmarks datasets in the field of knowledge editing.

B5: yes

B5 Elaboration For Yes Or No: In Section 6.1

B6: yes

B6 Elaboration For Yes Or No: In Section 6.1

C: yes

C1: yes

C1 Elaboration For Yes Or No: In Section 6.1

C2: yes

C2 Elaboration For Yes Or No: In Section 6.1 and Section 6.3

C3: yes

C3 Elaboration For Yes Or No: In Section 6.2

C4: yes

C4 Elaboration For Yes Or No: In Section 6.1

D: no

E: yes

E1: no

E1 Elaboration For Yes Or No: ChatGPT is mainly used to help understand past related work and its code, so it is not cited in this paper.