Towards Automatic Evaluation Of Task-Oriented Dialogue Flows

Anonymous

Abstract: Task-oriented dialogue systems are an important tool in conversational AI, relying on predefined dialogue flows to guide user-agent interactions. Currently, these flows are either manually created by domain experts or generated through AI techniques, leading to variability in their quality. This lack of standardization poses a challenge for conversational designers and automated techniques. To address this issue, we propose a novel framework for evaluating the quality of these dialogue flows. Our framework focuses on the complexity of the flow structure and its representation of historical data. To this end, we introduce FuDGE, a Fuzzy Dialogue-Graph Edit Distance metric that assesses the match between a flow and a set of real-world conversations. Through extensive experiments, we demonstrate the effectiveness of our framework and show that FuDGE can help standardize and improve dialogue flows for task-oriented dialogue systems.

Paper Type: long

Research Area: Dialogue and Interactive Systems