Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning

Xinlong Li, Weijieying Ren, Wei Qin, Lei Wang, Tianxiang Zhao, Richang Hong

Abstract: Existing continual learning works explored strategies like memory replay, regularization, and parameter isolation, but little analysis was conducted on the optimization behavior of LLMs’ continual fine-tuning. In this work, we investigate the geometric connections of different minima along the continual LLM fine-tuning trajectories. We validate this phenomenon on LLMs and propose a new method called Interpolation-based LoRA (I-LoRA). I-LoRA can strike a balance between plasticity and stability through parameter interpolation, which constructs a dual-memory experience replay framework based on LoRA. Experiments on eight domain-specific benchmarks demonstrate that I-LoRA consistently shows significant improvement over previous approaches with up to 11% performance gains. Our code is available at https://anonymous.4open.science/r/LLMCL-3823.

External IDs:dblp:conf/icassp/LiRQW0H25