LiST: Lite Self-training Makes Efficient Few-shot Learners

Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, Jianfeng Gao

Keywords: Prompt fine-tuning, Semi-supervised Learning, Few-shot, NLP

Abstract: We present a new method LiST for efficient fine-tuning of large pre-trained language models (PLMs) in few-shot learning settings. LiST significantly improves over recent methods that adopt prompt-tuning using two key techniques. The first one is the use of self-training to leverage large amounts of unlabeled data for prompt tuning to significantly boost the model performance in few-shot settings. We use self-training in conjunction with meta-learning for re-weighting noisy pseudo-prompt labels. However, traditional self-training is also quite expensive as it requires updating all the model parameters repetitively. Therefore, we introduce a second technique for light-weight fine-tuning where we only update a small number of the model parameters. To this end, we introduce a small number of task-specific adapter parameters that are tuned during self-training while keeping the PLM encoder frozen. This also significantly reduces the overall model footprint across several tasks that can now share a common PLM encoder as backbone for inference. Combining the above techniques, LiST not only improves the model performance for few-shot learning on target domains but also reduces the model memory footprint. We present a comprehensive study on six NLU tasks to validate the effectiveness of LiST . The results show that LiST improves by 35% over classic fine-tuning and 6% over prompt-tuning with 96% reduction in the number of trainable parameters when fine-tuned with only 30 labeled examples from the target domain.

One-sentence Summary: Lite self-training on unlabeled data with prompts and adapters make efficient few-shot learners

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2110.06274/code)