Tackling Concept Shift in Text Classification using Entailment-style modeling

Sumegh Roychowdhury, Siva Rajesh Kasa, Karan Gupta, Prasanna Srinivasa Murthy, Alok Chandra

Keywords: Concept Shift, Pretrained Language Models, Entailment

TL;DR: We propose a reformulation of the vanilla classification task to help pre-trained language models adapt to sudden concept shift using much lesser data in textual domain leading to 75% savings in re-labeling costs.

Abstract:

Pre-trained language models (PLMs) have seen tremendous success in text classification (TC) problems in the context of Natural Language Processing (NLP). In many real-world text classification tasks, the class definitions being learned do not remain constant but rather change with time - this is known as concept shift. Most techniques for handling concept shift rely on retraining the old classifiers with the newly labelled data. However, given the amount of training data required to fine-tune large DL models for the new concepts, the associated labelling costs can be prohibitively expensive and time consuming.

In this work, we propose a reformulation, converting vanilla classification into an entailment-style problem that requires significantly less data to re-train the text classifier to adapt to new concepts. We demonstrate the effectiveness of our proposed method on both real world & synthetic datasets achieving absolute F1 gains upto 7% and 40% respectively in few-shot settings. Further, upon deployment, our solution also helped save 75% of labeling costs overall.

Submission Number: 2