EPT: Explosive Prompt Tuning For Parameter-Efficient with Large Norm Prompt
Abstract: Prompt tuning introduces additional learnable tokens, known as $\textit{soft prompts}$, to frozen pre-trained language models for parameter-efficient tuning. Unlike fine-tuning, only these soft prompts are trained on downstream tasks rather than all model parameters. While recent prompt tuning approaches that introduce a reparameterization network have shown comparable performance to fine-tuning, they still require a large number of parameters for the soft prompts. In this paper, we empirically show the characteristics of the recent prompt tuning methods, such as the large norm of trained soft prompts and their significant similarity to each other. Inspired by these observations, we propose simple yet effective modifications to the reparameterization network for efficient prompt tuning, which involves inducing large norm, replacing overparameterization with under-parameterization, and focusing on a single prompt. This approach preserves the advantageous characteristics of the soft prompts while significantly reducing the number of parameters. Our comprehensive experiments across 21 diverse NLP datasets show that our method called EPT: Explosive Prompt Tuning, significantly outperforms prompt Tuning and achieves comparable performance full fine-tuning or other parameter-efficient tuning, with only 2.3K parameters during training on T5-base.
Paper Type: Long
Research Area: Efficient/Low-Resource Methods for NLP
Research Area Keywords: Efficient/Low-Resource Methods for NLP, Machine Learning for NLP
Contribution Types: Approaches to low-resource settings, Approaches low compute settings-efficiency, Publicly available software and/or pre-trained models
Languages Studied: English
Submission Number: 4444
Loading