CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learners

ACL ARR 2025 February Submission149 Authors

Abstract: Knowledge Editing (KE) enables the modification of outdated or incorrect information in large language models (LLMs). While existing KE methods can only update isolated facts, they struggle to generalize these updates to multi-hop reasoning tasks that depend on the modified knowledge. Through an analysis of reasoning circuits---the neural pathways LLMs use for knowledge-based inference, we identify that current layer-localized KE approaches fail to effectively integrate updated information into these reasoning pathways. To address this limitation, we propose CaKE (Circuit-aware Knowledge Editing), a novel method that facilitates more effective integration of updated knowledge in LLMs. CaKE leverages meticulously curated data that enforces the model to utilize the modified knowledge, guiding the model to develop appropriate reasoning circuits for newly integrated knowledge. Experimental results show that CaKE enables more accurate and consistent use of updated knowledge across related reasoning tasks, leading to an average 20% improvement in multi-hop accuracy on MQuAKE compared to existing KE methods.

Paper Type: Long

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: knowledge tracing/discovering/inducing; model editing; knowledge editing; circuit analysis

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 149