Optimizing low-rank adaptation with decomposed matrices and adaptive rank allocation

Dacao Zhang, Fan Yang, Kunwu Zhang, Xin Li, Si Wei, Richang Hong, Meng Wang

Abstract: In this paper, we argue that the same rank setting of LoRA will inhibit its potential. In response, we proposed two novel PEFT strategies to improve the capability of LoRA in single-task and multi-task scenarios. We also conducted extensive experiments to show the effectiveness of our proposed method.