From One to Many: Trajectory Invariant Learning for Multimodal Large Language Model Editing

Jiajie Su, Haoyuan Wang, XiaoHua Feng, Yunshan Ma, Xiaobo Xia, Yuyuan Li, Chaochao Chen, Xiaolin Zheng, Jianwei Yin

Keywords: knowledge editing, multimodal learning, multimodal large language models

TL;DR: We propose a novel multimodal model editing method named ODEdit to promote editing robustness across diverse cross-modal prompting environments.

Abstract:

Knowledge editing emerges as a crucial technique for efficiently correcting incorrect or outdated knowledge in large language models (LLM). Existing editing methods for unimodal LLM rely on a rigid parameter-to-output mapping, which causes causal-underfit and causal-overfit in cascaded reasoning for Multimodal LLM (MLLM). In this paper, we reformulate MLLM editing as an out-of-distribution (OOD) generalization problem, where the goal is to discern semantic shift with factual shift and thus achieve robust editing among diverse cross-modal prompting. The key challenge of this OOD problem lies in identifying invariant causal trajectories that generalize accurately while suppressing spurious correlations. To address it, we propose ODEdit,a plug-and-play invariant learning based framework that optimizes the tripartite OOD risk objective to simultaneously enhance editing reliability, locality, and generality.We further introduce an edit trajectory invariant learning method, which integrates a total variation penalty into the risk minimization objective to stabilize edit trajectories against environmental variations. Theoretical analysis and extensive experiments demonstrate the effectiveness of ODEdit. Our code is available at
https://anonymous.4open.science/r/ODEdit-2756.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 12508