P-Distill: Efficient and Effective Prompt Tuning using Knowledge Distillation

Anonymous

Abstract: In the field of natural language processing (NLP), prompt-based learning is widely used for efficient parameter learning. In this study, we propose P-Distill, a novel approach that mitigates this limitation by utilizing knowledge distillation from a teacher model with extensive prompts to a student model with shorter prompts. We introduce two novel methods for prompt compression, including prompt initialization, and prompt distillation. Experiments across various NLP tasks demonstrate that P-Distill exhibits comparable or superior performance compared to other state-of-the-art prompt-based learning methods, even with significantly shorter prompts. We achieve a peak improvement of 1.90% even with the prompt lengths compressed to one-eighth. An additional study further provides insights into the distinct impact of each method on the overall performance of P-Distill. These results highlight the potential of P-Distill in facilitating efficient and effective training for a wide range of NLP models.

Paper Type: long

Research Area: Efficient/Low-Resource Methods for NLP

Contribution Types: NLP engineering experiment, Approaches low compute settings-efficiency

Languages Studied: English