PISA: Compressive Sensing Adaptation of Large Language Models

Minxue Xia, Hao Zhu, Yifei Zhang

Keywords: Parameter-Efficient Fine-Tuning (PEFT)，k-sparse approximation，low-rank structure

Abstract:

In this paper, we introduce a novel perspective on Parameter-Efficient Fine-Tuning (PEFT) by viewing the weight update matrix as a k-sparse approximation in the spatial domain, departing from the commonly used low-rank structure assumption. We propose a compressive sensing-based approach that leverages under-complete measurement matrices to analyze the approximation capabilities of the weight update matrix. Our method ensures bounded error in the reconstruction of the weight updates, as guaranteed by theoretical results in compressive sensing.
However, the vectorization of the weight update matrix leads to a high-dimensional problem (d^2), which can potentially result in large error bounds. To address this issue, we introduce a block-structured approximation scheme that partitions the weight update matrix into smaller blocks and applies the k-sparse approximation to each block independently. We theoretically analyze the approximation error bounds of our approach and demonstrate that the block-structured scheme achieves tighter error bounds compared to the non-block approach.
Empirically, we validate the effectiveness of our proposed method on various downstream NLP tasks, showcasing its ability to achieve competitive performance with a reduced number of trainable parameters. Our approach offers a new direction for parameter-efficient fine-tuning of large language models. Notably, our experiments demonstrate competitive performance with only 500 learnable parameters, while offering greater memory and computational efficiency than LoRA in a rank-1 setting.

Primary Area: foundation or frontier models, including LLMs

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2793