Intent classification using contextual embeddings

Yunhao Chen, Hélène Rondey

Abstract: Sequence Labelling problems, such as Dialogue Act (DA) and Sentiment/Emotion (S/E) identification, play a large part in Natural Language Understanding tasks, like implementing Conversational Agents. In the past few years, the methods used for vector representation of words and utterances, an essential component of these problems, have been significantly improved thanks to Large Language Models, like BERT (Devlin et al., 2018), which is based on the Transformer Architecture. In this work we experiment on the Sequence Labelling Problem by proposing our own architecture for classification and relying on the fine-tuning of a pre-trained large language model, either RoBERTa (Liu et al., 2019) or DeBERTa (He et al., 2020). We then compare the performance of the two models for our architecture. We also study the difference in performance between a total fine-tuning and a partial fine-tuning of the pre-trained models. We conduct our analysis on the annoted spoken language datasets from the SILICONE benchmark (Chapuis et al., 2020). Our work shows that the two models are quite similar in test accuracy and seem to outperform the models from (Colombo, 2021). We also show that partially fine-tuning provides lesser performances than totally fine-tuning. We conclude by discussing some other leads of exploration to tackle the Intent classification problem, like hierarchical structures and contrasive learning.