Automated Few-Shot Classification with Instruction-Finetuned Language Models

Rami Aly, Xingjian Shi, Kaixiang Lin, Aston Zhang, Andrew Gordon Wilson

Submission Type: Regular Long Paper

Submission Track: Semantics: Lexical, Sentence level, Document Level, Textual Inference, etc.

Submission Track 2: Theme Track: Large Language Models and the Future of NLP

Keywords: few-shot classification, prompt automation, large language models

TL;DR: Automation of prompts for large instruction-tuned encoder-decoder language models on few-shot classification tasks.

Abstract:

A particularly successful class of approaches for few-shot learning combines language models with prompts - hand-crafted task descriptions that complement data samples. However, designing prompts by hand for each task commonly requires domain knowledge and substantial guesswork. We observe, in the context of classification tasks, that instruction finetuned language models are remarkably robust towards some dimensions of a prompt's design. We subsequently propose a simple method to eliminate the need for handcrafted prompts, named AuT-Few.
This approach consists of (i) a prompt retrieval module that selects suitable task instructions from the instruction-tuning knowledge base, and (ii) the generation of two distinct, semantically meaningful, class descriptions and a selection mechanism via cross-validation. Over 12 datasets, spanning 8 classification tasks, we show that AuT-Few outperforms current state-of-the-art few-shot learning methods. Moreover, AuT-Few is the best ranking method across datasets on the RAFT few-shot benchmark. Notably, these results are achieved without task-specific handcrafted prompts on unseen tasks.

Submission Number: 2225