Learning From Textual User Feedback – Collect New Datasets Or Extend Existing Ones?

Anonymous

Abstract: Textual user feedback is of growing importance to continuously improve dialogue systems, but appropriately annotated data is scarce, especially for types other than open-domain dialogues. In this work, we investigate the extendibility of six existing datasets from various types, e.g., MultiWoZ (task-oriented), PersonaChat (open-domain), and Wizards-of-Wikipedia (knowledge-grounded) with the required annotations. The results of this corpus study are new insights into dataset composition, including error and user response types, the relation between them, and EURTAD, the first feedback-annotated dataset that includes various dialogue types. For annotation, we propose two new taxonomies for error and user response type classification

Paper Type: long

Research Area: Dialogue and Interactive Systems