Eliciting Knowledge from Pretrained Language Models for Prototypical Prompt Verbalizer

Yinyi Wei, Tong Mo, Yongtao Jiang, Weiping Li, Wen Zhao

Abstract: Recent advances on prompt-tuning cast few-shot classification tasks as a masked language modeling problem. By wrapping input into a template and using a verbalizer which constructs a mapping between label space and label word space, prompt-tuning can achieve excellent results in few-shot scenarios. However, typical prompt-tuning needs a manually designed verbalizer which requires domain expertise and human efforts. And the insufficient label space may introduce considerable bias into the results. In this paper, we focus on eliciting knowledge from pretrained language models and propose a prototypical prompt verbalizer for prompt-tuning. Labels are represented by prototypical embeddings in the feature space rather than by discrete words. The distances between the embedding at the masked position of input and prototypical embeddings are used as classification criterion. To address the problem of random initialization of parameters in zero-shot settings, we elicit knowledge from pretrained language models to form initial prototypical embeddings. Our method optimizes models by contrastive learning. Extensive experimental results on several many-class text classification datasets with low-resource settings demonstrate the effectiveness of our approach compared with other verbalizer construction methods. Our implementation is at https://github.com/Ydongd/prototypical-prompt-verbalizer .