SPT: Learning to Selectively Insert Prompts for Better Prompt Tuning
Submission Type: Regular Long Paper
Submission Track: Machine Learning for NLP
Submission Track 2: Language Modeling and Analysis of Language Models
Keywords: Prompt tuning, neural architecture search, parameter efficient tuning
Abstract: Prompt tuning prepends a soft prompt to the input embeddings or hidden states and only optimizes the prompt to adapt pretrained models (PTMs) to downstream tasks. The previous work manually selects prompt layers which are far from optimal and failed to exploit the potential of prompt tuning. In this work, we propose a novel framework, \underline{S}elective \underline{P}rompt \underline{T}uning (SPT), that learns to select the proper prompt layers by inserting a prompt controlled by a learnable probabilistic gate at each intermediate layer. We further propose a novel bi-level optimization framework, SPT-DARTS, that can better optimize the learnable gates and improve the final prompt tuning performances of the learned prompt layer settings. We conduct extensive experiments with ten benchmark datasets under the full-data and few-shot scenarios. The results demonstrate that our SPT framework can perform better than the previous state-of-the-art PETuning baselines with comparable or fewer tunable parameters. \footnote{Codes will be publicly available upon acceptance. }
Submission Number: 2451
Loading