Karenina: Modeling the Complexity of Negative Emotions to Better Serve Industry Goals

Moritz Sudhof, Liam Croteau, Christopher Potts

Keywords: emotion analysis, sentiment analysis, industrial applications

TL;DR: We introduce Karenina, a labeled dataset of 25K consumer healthcare experience texts, we validate it through modeling experiments, and we illustrate how Karenina models can be used to address industry needs.

Abstract: Sentiment analysis systems are widely applied in industry, but standard formulations of the task (e.g., positive/negative/neutral classification) are often not well aligned with real-world goals. For instance, in customer support contexts, negative labels dominate due to the nature of the work, and different negative emotions call for different solutions. To help address this issue, we introduce Karenina, a labeled dataset of 25K consumer healthcare experience comments with labels that support standard sentiment distinctions but also allow for a breakdown into six negative emotions: confused, disappointed, frustrated, angry, stressed, and worried. Each text has 1-4 emotion labels, with over 90% of examples having at least 2 labels. We define strong baselines for this dataset, we seek to motivate a flexible approach to evaluation that takes into account the variable costs for different mistakes in different industrial contexts, and we report on some illustrative analyses using Karenina to understand customer experiences.

URL: https://github.com/sudhof/karenina