Enhancing the Model Robustness and Generalization in Sentence Sentiment Classification Based on Causal Representation Learning Techniques

PENG Yanyi, Xiang Li

Keywords: Causal Representation Learning, Sentiment Classification, Spurious Correlation

Abstract: Sentiment classification is an important task in natural language processing (NLP), aiming to perform sentiment analysis on sentences. One of widely used method based on the causal word detection first estimates the treatment effect between words and sentiment of sentences, and then removing words with low treatment effect in sentiment classification model training. However, the previous works regard whether the specific word appears in the sentence as the binary treatment, which limits the robustness of identify the treatment effect of word, especially for the low-frequency word. To bridge this gap, we propose a novel causal representation learning method that regarding word representation as treatments to ensure the generalization of the sentiment classifier. Specifically, the method begins by clustering words based on their representations obtained from a pre-trained language model. Subsequently, corresponding to the clusters, a multi-head word classifier is trained to estimate the treatment effect of each word to identify whether this word is causally or spurious correlated to the sentiment. To ensure covariate balancing between each treatment cluster, we utilize the integral probability metric (IPM) distance to learn the balanced representation of the context. Then, the balanced representation and estimated treatment effects are used to train a more robust and generalizable sentiment classification model. Extensive experiments on public datasets demonstrate the effectiveness of our method in identifying causal words and improving the performance of sentiment classification.

Submission Number: 33