Editing Large Language Models: Problems, Methods, and Opportunities

Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, Ningyu Zhang

Submission Type: Regular Long Paper

Submission Track: Theme Track: Large Language Models and the Future of NLP

Submission Track 2: Efficient Methods for NLP

Keywords: Model Editing, Large Language Model, Editing Factual Knowledge

TL;DR: This paper embarks on a deep exploration of the problems, methods, and opportunities relating to model editing for LLMs.

Abstract:

Despite the ability to train capable LLMs, the methodology for maintaining their relevancy and rectifying errors remains elusive.
To this end, the past few years have witnessed a surge in techniques for editing LLMs, the objective of which is to alter the behavior of LLMs \textbf{efficiently} within a specific domain without negatively impacting performance across other inputs. This paper embarks on a deep exploration of the problems, methods, and opportunities related to model editing for LLMs.
In particular, we provide an exhaustive overview of the task definition and challenges associated with model editing, along with an in-depth empirical analysis of the most progressive methods currently at our disposal. We also build a new benchmark dataset to facilitate a more robust evaluation and pinpoint enduring issues intrinsic to existing techniques.
Our objective is to provide valuable insights into the effectiveness and feasibility of each editing technique, thereby assisting the community in making informed decisions on the selection of the most appropriate method for a specific task or context\footnote{Code and datasets are in the supplementary and will be released.}.

Submission Number: 1409