Lexicon Creation for Interpretable NLP Models

Anonymous

Abstract: Lexica--words and associated scores--are widely used as simple, interpretable, generalizable language features to predict sentiment, emotions, mental health, and personality traits. Applying different feature importance methods to different predictive models yields lexica of varying quality. In this paper, we train diverse sequence classification models, including context-oblivious (SVMs, Feed-forward neural networks) and context-sensitive (RoBERTa, DistilBERT) models, and generate lexica based on different feature importance measurements, including attention, masking, and SHAP (SHapley Additive exPlanations) values. We evaluate the generated lexica on their predictive performance on test sets within the same corpus domain and on their generalization to different but similar domains. We find that simple context-oblivious models produce lexica of similar accuracy within domain and of better accuracy across domains to those from complex context-sensitive models. Based on human evaluator ratings of these lexica, we also find that context-oblivious models generate similar lexica that are more aligned with human judgments.