PETapter: Leveraging PET-style classification heads for modular few-shot parameter-efficient fine-tuning
Abstract: Few-shot learning and parameter-efficient fine-tuning (PEFT) are crucial to overcome the challenges of data scarcity and ever growing language model sizes. This applies in particular to specialized domains such as argument mining, where complex and nuanced phrasing makes it difficult even for humans to distinguish not only between the stances but also whether a sentence contains a claim or an argument. We propose PETapter, a novel method that effectively combines PEFT methods with PET-style classification heads to boost few-shot learning capabilities without the significant computational overhead typically associated with full model training. We validate our approach on three established NLP benchmark datasets and one real-world argument mining dataset. We show that PETapter not only achieves comparable performance to full few-shot fine-tuning using pattern-exploiting training (PET), but also provides greater reliability and higher parameter efficiency while enabling higher modularity and easy sharing of the trained modules.
Paper Type: Long
Research Area: Efficient/Low-Resource Methods for NLP
Research Area Keywords: parameter-efficient-training, data-efficient training, argument mining, NLP tools for social analysis, quantitative analyses of news and/or social media
Contribution Types: NLP engineering experiment, Approaches to low-resource settings, Approaches low compute settings-efficiency
Languages Studied: English, German
Submission Number: 3171
Loading