To Know What User Concerns: Conceptual Knowledge Reasoning for Task-oriented Dialogue Quality Estimation
Abstract: Dialogue Quality Estimation (DQE) is crucial in assessing the effects of a conversational consultation system, which has wide applications in E-commerce and Social Media. In task-oriented scenarios, users usually seek personalized consultation about the target subjects they are concerned with rather than general knowledge commonly known by populations. It is essential to identify whether a dialogue solves the user’s questions by task-oriented DQE. Existing studies mainly focus on analyzing dialogue semantics and user sentiment, neglecting to understand what the user is concerned about when requesting a consultation. It may cause fatal errors when the response is emotionally friendly but non-informative. In this paper, we propose a knowledge-enhanced DQE model named CoReT, which introduces the Conceptual Knowledge Reasoning for Task-oriented DQE. We first design a simple yet efficient entity linking and relation selection module enabling conceptual reasoning from a knowledge graph. Then, we propose a multi-turn textual encoder to capture the contextual information in dialogues. Finally, we introduce a knowledge enhancement module to fuse conceptual reasoning features into contextual embeddings to produce DQE results. For evaluation, we conduct experiments on two real-world datasets in e-commerce consultation systems, the results demonstrate the effectiveness and robustness of CoReT compared with the state-of-the-art baselines.
Paper Type: Long
Research Area: Dialogue and Interactive Systems
Research Area Keywords: task-oriented dialogue evaluation, knowledge augmented
Contribution Types: NLP engineering experiment
Languages Studied: chinese
Submission Number: 3612
Loading