HiddenKey: Parameter-Efficient FineTuning Meets Dropout under a Unified Framework

Sheng Wang, Liheng Chen, Jiyue Jiang, Lingpeng Kong, Chuan Wu

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Dropout, LoRA, Parameter-Efficient FineTuning, NLU, NLG, NLP

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We investigate the contradiction between limited trainable parameters in PEFT scenario and dropout regularization methods associated with excessive parameter redundancy, and recommend a superior practice named HiddenKey guided by a unified framework.

Abstract: The emerging powerful capabilities exhibited by large language models (LLMs) have established them as a fundamental element in various applications that rely on advanced language understanding. At the same time, fine-tuning has become the standard learning approach to adapting LLMs to a concrete application (e.g., instruction tuning, alignment tuning, and task/user-specific specialization). Due to the high cost associated with full finetuning, parameter-efficient finetuning (PEFT) methods, especially LoRA, have gained popularity due to their lower storage, memory, and computation requirements. However, the possible contradiction between limited trainable parameters and the dropout regularization methods (which aim at alleviating overfitting associated with excessive parameter redundancy), has been largely overlooked. With extensive experiments of LoRA-based PEFT, we first confirm that PEFT is also overfitting-prone. We then revisit transformer-specific dropout methods, and validate their equivalence and differences mathematically and empirically. To facilitate a comprehensive comparison, we introduce a unified framework to instantiate them along dropping position, structural pattern and compensation measure, and uncover their new preferences and performance comparisons in PEFT scenarios. This framework also enables us to integrate the best of all into a new dropout method named HiddenKey, which shows performance superiority over existing methods on both NLU and NLG tasks. Compared to baselines, it also achieves better performance with less finetuning time, and offers continuous improvement with further finetuning. These highlight HiddenKey as the better practice for high-performance and parameter-efficient finetuning of LLMs.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3347