Snapshot Prompt Ensemble for Parameter-Efficient Soft Prompt Transfer

Xinglong Wu, C. L. Philip Chen, Shuzhen Li, Tony Zhang

Abstract: Soft Prompt Transfer(SPT) uses well-trained soft prompts as initialization to improve prompt tuning efficiency. However, most methods in SPT learn only a single and task-specific prompt for each source task. It may not be suitable for the target task and results in poor transferability on target task. To address this issue, we propose Snapshot Prompt Ensemble (SPE) method for parameter-efficient soft prompt transfer. Specifically, SPE extracts multiple soft prompts from each source task via taking snapshots at different training phases of prompt tuning. SPE then adaptively ensembles the multiple soft prompts and obtains a fused and instance-dependent prompt for the target task by a cross-task attention module. SPE can effectively exploit the prompts at different training phases and provide a more suitable starting point for target prompt training. Extensive experiments on various NLU tasks demonstrate that SPE outperforms state-of-the-art methods while SPE tunes less than 0.4% of the parameters compared to full fine-tuning.