Eva-KELLM: A New Benchmark for Evaluating Document-based Knowledge Editing of LLMs

Anonymous

Abstract: Since the knowledge of large language models (LLMs) may become outdated or contain inaccuracies, knowledge editing for LLMs and evaluating their effectiveness attract increasing attention. However, current knowledge editing methods often rely on manually annotated triples or question-answer pairs, limiting their applicability. In this paper, we explore a more general knowledge editing scenario where LLMs only use raw documents for editing. Given the absence of benchmarks for document-based knowledge editing, we propose a new benchmark Eva-KELLM, which includes raw documents for editing and corresponding test datasets evaluated from multiple perspectives. In addition to conventional evaluations assessing the model's memory of altered knowledge and retention of unrelated knowledge, we also evaluate the updated LLM's performance in reasoning with altered knowledge and cross-lingual knowledge transfer. Furthermore, we propose a document-based knowledge editing method aimed at addressing challenges associated with noise and unidirectional auto-regressive learning. Experimental results on the benchmark showcase the effectiveness of our method in achieving improved performance.

Paper Type: long

Research Area: Resources and Evaluation

Contribution Types: Data resources

Languages Studied: English,Chinese