UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems

Jafar Afzali, Aleksander Mark Drzewiecki, Krisztian Balog, Shuo Zhang

Abstract: We present an extensible user simulation toolkit to facilitate automatic evaluation of conversational recommender systems. It builds on an established agenda-based approach and extends it with several novel elements, including user satisfaction prediction, persona and context modeling, and conditional natural language generation. We showcase the toolkit with a pre-existing movie recommender system and demonstrate its ability to simulate dialogues that mimic real conversations, while requiring only a handful of manually annotated dialogues as training data.