Anchoring Fine-tuning of Sentence Transformer with Semantic Label Information for Efficient Truly Few-shot Classification

Amalie Brogaard Pauli, Leon Derczynski, Ira Assent

Submission Type: Regular Short Paper

Submission Track: Efficient Methods for NLP

Submission Track 2: Machine Learning for NLP

Keywords: few-shot, sentence transformer, classification, efficiency

Abstract:

Few-shot classification is a powerful technique,
but training requires substantial computing
power and data. We propose an efficient
method with small model sizes and less training
data with only 2-8 training instances per class.
Our proposed method, AncSetFit, targets low data
scenarios by anchoring the task and label
information through sentence embeddings in
fine-tuning a Sentence Transformer model. It
uses contrastive learning and a triplet loss to enforce
training instances of a class to be closest
to its own textual semantic label information
in the embedding space - and thereby learning
to embed different class instances more distinct.
AncSetFit obtains strong performance
in data-sparse scenarios compared to existing
methods across SST-5, Emotion detection, and
AG News data, even with just two examples
per class.

Submission Number: 1586