Learning From Textual User Feedback — Collect New Datasets Or Extend Existing Ones?

Anonymous

Abstract: Learning from textual user feedback, i.e., user responses that address errors in system utterances, is increasingly important to continuously improve dialogue systems, but datasets that provide the needed annotations, i.e., annotations for causing errors and user responses, are scarce. As creating new datasets involves an immense manual effort, we investigate the extendability of various existing dialogue datasets with annotations for errors and user responses. In order to facilitate the detection of dialogues that contain such data, we propose Textual Feedback Detection (TFD), a semi-automatic approach to identify textual user feedback. Furthermore, we propose two taxonomies optimized to categorize such data, a user response type taxonomy and an error type taxonomy. In our study, we annotate 1,155 dialogues from six different dialogue datasets with both errors and corresponding user responses. Our findings give insights on dataset-specific error and user response types. We show that open-domain and knowledge-grounded dialogue datasets are more appropriate to be extended with annotations for causing errors and user responses than task-oriented dialogue datasets.

Paper Type: long

Research Area: Dialogue and Interactive Systems