InterLoRA: An Adaptive LoRA Structure Based on The Mechanistic Interpretability of Transformer

Jihao Gu, Zelin Wang, Yibo Zhang, Ping Gong, Zhisong Bie

Keywords: parameter-efficient-training, low-rank adaptation

TL;DR: This paper proposes an improved LoRA method and based on the mechanistic interpretability of Transformer

Abstract: With the escalating costs associated with fine-tuning large pre-trained models, the significance of parameter-efficient fine-tuning (PEFT) methods has become increasingly evident. Among these methods, we focus on LoRA, which introduces parallel trainable parameters in the multi-head attention component and has demonstrated promising results. However, previous research may have overlooked the mechanistic interpretability of the transformer architecture, especially since PEFT methods are built upon this framework. Drawing on this insight, we propose InterLoRA, which integrates LoRA with feature adaptation mechanism into both the attention layer, considering the varying importance of multiple heads, and the Feed-Forward Network (FFN) layer, acknowledging the memory storage characteristics. Experiments conducted on a variety of complex generation tasks highlight the effectiveness of InterLoRA in jointly fine-tuning both components while efficiently managing parameter memory.

Submission Number: 82