Incorporating Multimodal Sentiments into Conversational Bots for Service Requirement Elicitation

Demin Yu, Junrui Tian, Tonghua Su, Zhiying Tu, Xiaofei Xu, Zhongjie Wang

Abstract: In recent years, task-oriented conversational AI Bots have become very popular in many work and life scenarios with the goal of elicit and satisfy user requirements/intentions by natural language based interactions. To capture user intentions accurately, it is important to design efficient dialog strategy for guiding users to express their intentions by limited rounds of dialogue. However, intention acquisition methods based solely on semantics analysis do not work well especially for those complex user requirements/intentions. In this work, we design a conversational AI bot that could capture user intentions based on multimodal information including video, audio and texts. The bot could incorporate sentiments that are extracted from multimodal information into the bot's dialogue strategy. Sentiments are used for dynamically adjusting the questioning tactics during the dialogue. We design controlled experiments to explore the effectiveness of incorporating multimodal sentiments into the process of guiding users to expressing their intentions. Volunteers are asked to talk with the conversational AI Bot with rich emotions in real application scenarios. Experimental results show that our proposed approach could effectively improve the accuracy of user intention recognition and increase user satisfaction during dialogue. This work lays a solid foundation for the future's service solution design which is a key step in Service-Oriented Systems Engineering (SOSE).