XoRA: Expander Adapted LoRA Finetuning

Amaljith EV, Arindam Biswas, Suryam Arnav Kalra, Pabitra Mitra, BISWAJIT BASU

Keywords: low-rank fine-tuning, expander graphs, sparse fine-tuning

TL;DR: Expander graph based masking of LoRA matrices for efficient fine-tuning

Abstract: Parameter-efficient fine-tuning aims to reduce the computational cost of adapting foundational models to downstream tasks. Low-rank matrix based adaptation (LoRA) techniques are popular for this purpose. We propose XoRA, an efficient fine-tuning scheme, which sparsifies the low-rank matrices even further using expander masks. The mask is generated using extremal expander graphs (Ramanujan graphs) to maintain high edge connectivity even at a very high sparsity. Experimental results demonstrate that this method has comparable performance with the LoRA fine-tuning method while retaining much fewer number of parameters.

Submission Number: 49