MEMIT-Merge: Addressing MEMIT's Key-Value Conflicts in Same-Subject Batch Editing for LLMs

ACL ARR 2025 February Submission417 Authors

Abstract:

As large language models (LLMs) continue to scale up, knowledge editing techniques that modify models' internal knowledge without full retraining have gained significant attention.
MEMIT, a prominent batch editing algorithm, stands out for its capability to perform mass knowledge modifications.
However, we uncovers a critical limitation that MEMIT's editing efficacy significantly deteriorates when processing batches containing multiple edits sharing the same subject.
Our analysis reveals the root cause lies in
MEMIT’s key-value modeling framework:
when multiple facts with the same subject in a batch are modeled through MEMIT's key-value mechanism, identical keys (derived from the shared subject) are forced to represent different values (corresponding to distinct knowledge), resulting in update conflicts during editing.
Addressing this issue, we propose MEMIT-Merge, an enhanced approach that merges value computation processes for facts sharing the same subject, effectively resolving the performance degradation in same-subject batch editing scenarios.
Experimental results demonstrate that at a batch size of 5, while the original MEMIT's success rate drops to 46\%, MEMIT-Merge maintains a 98\% editing success rate, showcasing remarkable robustness to subject entity collisions.

Paper Type: Short

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: Knowledge Editing, knowledge tracing

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 417