Measuring the effect of pretraining a hierarchical autoencoder for binary intent classification on data with similar structure

Emilien Jemelen, Christophe Morau

Abstract:

In this article, we examine the potential of the hierarchical autoencoder presented in Chapuis, Colombo et al. (2020) for classifying movie reviews. Our objective is to determine how effective this model, which is tailored for processing conversations and utterances, is when applied to tasks with different hierarchical structures.

Our findings indicate that pretraining the hierarchical model on a dataset of movie subtitles unexpectedly fails to enhance prediction accuracy and performs worse than the same model without any pretraining. This suggests that the structure learned from subtitle conversations may not always generalize well to written paragraphs. These results prompt a discussion on the requirements that both data and models should meet to qualify for an accuracy boost through hierarchical pretraining.

In conclusion, our study provides novel insights into the potential of hierarchical autoencoders for natural language processing tasks and emphasizes the importance of comprehending the specific characteristics of data to optimize model performance.