AMP:the Attention Mechanism of Multiple Prompts for Transfer Learning

Anonymous

TL;DR: Our work exactly computes how much influence each source task yields for target task during prompt tranfer learning and automatically identify right source tasks which yield positive transfer for target task through learning an attention component

Abstract: Prompt transfer learning can significantly improve the performance of prompt-tuning methods. However, it requires much manual work to find out the proper source tasks which can yield positive transfer for the target task.We propose a two-stage multiple prompts transfer learning approach called AMP to address this drawback. First, we train a source prompt for each task as task embedding. Second, we learn a target prompt for each task which is an attention-weighted sum of source prompts through training an attention component. The attentions control the influence each source task yields for the target task, through which proper source tasks for the target task can be automatically identified. A source prompt is a 2D matrix, but the traditional attention mechanism only receives vectors. The prior methods employ pooling or flattened method to transform the matrix to the vector for computing the attentions between a set of matrices. We propose a method called DAM which can compute attentions between matrices directly without transforming. DAM method can more exactly compute the attentions between matrices. Wide experiments demonstrate that AMP is effective and can improve the performance of prompt-tuning without any prior search.

Paper Type: long

Research Area: Efficient/Low-Resource Methods for NLP

Contribution Types: NLP engineering experiment

Languages Studied: english