Sentence representation and classification using attention and additional language information

Hongli Deng, Lei Zhang, Lituan Wang

Abstract: Sentence representation is one of the foundational tasks in natural language processing, and long short term memory (LSTM) is a widely used tool to deal with the variable-length sentence. In this paper, a new LSTM-based sentence representation model is proposed for sentence classification task. By introducing a self-supervised method in the process of learning the hidden representation of the sentence, the proposed model automatically capture the syntactic and semantic information from the context and used as additional language information to learn better contextual hidden representation. Moreover, instead of using the final hidden representation of LSTM or the max (or average) pooling of the hidden representations over all the time step, we propose to generate the global representation of the sentence by combining all contextual hidden representations in an element-wise attention manner. We evaluate our model on three sentence classification tasks: sentiment classification, question type classification, and subjectivity classification. Experimental results show that the proposed model improves the accuracy of sentence classification compared to other sentence representation methods in all of the three tasks.