Going Beyond LoRA Fine-Tuning with Hebb Learning: Blazingly Fast and Accurate

Alexander Demidovskij, Igor Salnikov, Olga Frolova, Aleksei Trutnev, Artyom Tugaryov, Yuri Ignatiev, Vasilisa Blyudova, Egor Zharikov, Irina Novikova

Abstract: Modern Multimodal Large Language Models have increased demands on computational resources required for both pretraining and fine-tuning procedures. This challenge is primarily attributed to the backpropagation step because the computation of gradients is time-consuming and memory-intensive. This paper aims to alleviate the presented issues, and introduces novel fine-tuning strategy. Low-Rank Adaptation with Hebb Rapid Optimization (LoRA-HeRO) effectively combines the gradient-based method of LoRA fine-tuning with a local learning rule. An extra feature of the proposed algorithm is weight importance analysis, that identifies Transformer blocks for vanilla LoRA update. Additionally, it is possible to perform the analysis of model convergence during the fine-tuning process. LoRA-HeRO achieves lossless fine-tuning acceleration for InternVL-1B model by up to 48% and StableDiffusionV1-4 fine-tuning acceleration by 50% compared to conventional LoRA fine-tuning.

External IDs:doi:10.3233/faia251089