Orthogonal Mixture-of-Expert Low-Rank Adapter for Continual Learning

Zhuoran Xie, Shichun Yang, Xu Mohan, Qingyun Ye, Jiong Ding, Chen rui, Fan Zhou

Keywords: Continual Learning, Contrastive Learning, Orthogonal Projection, Low Rank Adaptation

Abstract: Continual Learning (CL) aims to prevent catastrophic forgetting during downstream finetuning. While Parameter-Efficient Fine-Tuning (PEFT) methods mitigate this by shielding pre-trained weights, they still suffer from severe cross-task interference. Existing solutions either use independent routers, causing structural misalignment, or rigid orthogonal constraints, severely limiting model plasticity. We propose the Orthogonal Mixture-of-Expert Low-Rank Adapter (OMoE-LoRA), which integrates an end-to-end contrastive soft router within the down-projection matrix to avoid misalignment, and an orthogonal constraint exclusively on the up-projection matrix to suppress cross-talk without sacrificing plasticity. Experiments on the MTIL benchmark demonstrate OMoE-LoRA achieves comparable accuracy with state-of-the-art method while effectively reducing trainable parameters.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 31