BalancEdit: Dynamically Balancing the Generality-Locality Trade-off in Multi-modal Model Editing

Dongliang Guo, Mengxuan Hu, Zihan Guan, Thomas Hartvigsen, Sheng Li

Abstract: Large multi-modal models inevitably decay over time as facts update and previously learned information becomes outdated. Traditional approaches such as fine-tuning are often impractical for updating these models due to their size and complexity. Instead, direct knowledge editing within the models presents a more viable solution. Current model editing techniques, however, typically overlook the unique influence ranges of different facts, leading to compromised model performance in terms of both generality and locality. To address this issue, we introduce the concept of the generality-locality trade-off in multi-modal model editing. We develop a new model editing dataset named OKEDIT, specifically designed to effectively evaluate this trade-off. Building on this foundation, we propose \textbf{BalancEdit}, a novel method for balanced model editing that dynamically achieves an optimal balance between generality and locality. BalancEdit utilizes a unique mechanism that generates both positive and negative samples for each fact to accurately determine its influence scope and incorporates these insights into the model's latent space using a discrete, localized codebook of edits, without modifying the underlying model weights. To our knowledge, this is the first approach explicitly addressing the generality-locality trade-off in multi-modal model editing. Our comprehensive results confirm the effectiveness of BalancEdit, demonstrating minimal trade-offs while maintaining robust editing capabilities. Our code and dataset are available at https://github.com/donglgcn/BalancEdit/tree/MMOKVQA.

Lay Summary:

Large language models and vision-language models often struggle to keep up with evolving facts over time. Traditional small-scale model updates, such as fine-tuning, usually require significant computational resources and data. In contrast, model editing offers a more efficient solution by locally modifying model weights to inject new knowledge.

However, existing methods tend to overemphasize improving generality after editing, while overlooking the preservation of locality. This "overgeneralization" may cause collateral damage to other correct knowledge, ultimately degrading the model's overall performance.
To address this, we propose BalancEdit, a model editing method that dynamically balances generality and locality. By constructing both supporting and counterfactual examples for the target fact, BalancEdit efficiently estimates the influence boundary of the edit, allowing the model to focus on the intended change without interfering with unrelated knowledge. Moreover, we introduce a codebook module to store and adjust the modified weights, enabling repeated edits with minimal cumulative impact.

Experiments on both existing datasets and our newly constructed dataset demonstrate that BalancEdit effectively injects new knowledge while preserving the model’s original capabilities.

Link To Code: https://github.com/donglgcn/BalancEdit/tree/MMOKVQA

Primary Area: Deep Learning->Foundation Models

Keywords: Model editing, Multi-modal model

Submission Number: 14934