Slaying the HyDRA: Parameter-Efficient Hyper Networks with Low-Displacement Rank Adaptation

Xiangyu Chen, Ye Wang, Matthew Brand, Pu Perry Wang, Jing Liu, Toshiaki Koike-Akino

Keywords: LoRA, PEFT, Low Displacement Rank Matrices

Abstract:

Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method to adapt large foundation models on downstream tasks. However, the weight updates are constrained to be low-rank structures, limiting their expressiveness.
Alternatively, low-displacement rank (LDR)-based structured matrices are rank unrestricted, while requiring few parameters and supporting fast matrix-vector multiplication. We propose a new PEFT strategy to construct the weight updates with block-wise LDR matrices by sampling parameters from a hyper network framework. Our method, hyper low-displacement rank adaptation (HyDRA), offers high flexibility for choosing the size of a pool of trainable parameters, while not being restricted by the displacement rank. Our experiments demonstrate that the HyDRA can boost the classification accuracy by up to 3.4% and achieve two-fold improvement in parameter efficiency on an image classification benchmark compared with other PEFT variants.

Submission Number: 87