PaSEFT: Gradient-based Partial Parameter Selection in Efficient Fine-Tuning for Large Language Models

Shuoran Jiang, Wentao Zhu, Qingcai Chen, Xiangping Wu, Junjie Chen, Yang Xiang, Youcheng Pan, Kai Chen

Keywords: partial parameter, parameter efficient fine-tuning, large language models, multilingual capability

Abstract: As the severe unfairness of language distribution in the pre-training data, large language models (LLMs) require further fine-tuning for multilingual capability. However, existing parameter efficient fine-tuning (PEFT) methods are unable to enhance multilingual translation capabilities in the incremental learning for LLMs. This paper proposes a language specified gradient attribution (LanGA) method to select the trainable subset from the full-parameter LLMs, which is used for the multilingual translation PEFT. Specifically, LanGA first select a language feature specific subset from full parameters for training, and the selected subset is much smaller than full parameters. Next, LanGA reconstruct the AdamW as a sparse optimizer for parameter subset training. Our experiments demonstrate that LanGA outperforms different low-memory fine-tuning methods for multilingual fine-tuning. At the same time, LanGA costs less memory usage that existing PEFT methods. Importantly, LanGA keeps the translation performance on resource-rich languages in highest measure.

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 10628