Explaining the Hardest Errors of Contextual Embedding Based Classifiers

Anonymous

TL;DR: A paper that explores documents that current text classification cannot correctly predict, proposing a taxonomy to accommodate these errors and a quantification and analysis of these documents by category.

Abstract: We seek to explain potential causes for incorrect classification of the most challenging documents, namely, documents that no classifier using state-of-the-art, very semantically-separable contextual embedding representations managed to accurately predict. To do so, we propose a misclassification taxonomy of incorrect predictions, which we used to perform qualitative human evaluation. We posed two (research) questions, achieving a high inter-evaluator agreement of 81.7%. We worked with three sentiment analysis datasets, two in the movie reviews domain and a third one containing product reviews. We quantified answers per category in our taxonomy across all datasets and computed their proportion. Differences were observed between the product and movie review domains, such as the prevalence of ambivalence in product reviews and sarcasm in movie reviews. Our analysis also revealed an unexpectedly high rate of human mislabeling in the datasets and a significant number of model errors that we cannot yet explain. To ensure reproducibility, our documentation, code, and datasets can be accessed on GitHub.

Paper Type: long

Research Area: Machine Learning for NLP

Contribution Types: Model analysis & interpretability, Data analysis

Languages Studied: english