Parameter-Efficient Fine-Tuning with Circulant and Diagonal Vectors

Xinyu Ding, Siyu Liao, Zhongfeng Wang

Keywords: fine-tuning; circulant matrices; diagonal matrices

Abstract:

Foundation models have achieved tremendous success in different domains.
However, their huge computation and storage complexity make these models difficult to fine-tune and also less applicable in practice.
Recent study shows training in fourier domain can be an effective fine-tuning method in terms of both model performance and number of training parameters.
In this work, we propose to further reduce the complexity by using the product of interleaved circulant and diagonal matrices.
Our method avoids the construction of weight change matrix and applies 1D fast fourier transform (FFT) instead of 2D FFT.
Experimental results show that our method achieves similar or better performance across various tasks with much less floating-point operations (FLOPs).

Primary Area: foundation or frontier models, including LLMs

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3602