A Korean Multimodal Dialogue System Across Multiple User Interfaces

Yeongbeom Lim, Jin Yea Jang, San Kim, Byunggill Joe, Saim Shin, Minyoung Jung

Abstract: Conversational agents, encompassing systems such as chatbots, avatars, and robots, utilize text or speech to repli-cate human conversation. When interacting with conversational agents, individuals may experience distrust or frustration. Nu-merous factors influence the quality of human-agent interactions, including emotions, dialogue acts, appearance, and nonverbal behavior. This paper acknowledges the necessity for a more realistic and extensible Korean-based conversational agent and develops several multimodal user interfaces for evaluation. We measure average mean opinion scores under four conditions: messenger, no visual (black screen), cat face, and avatar. Our experiment demonstrated that the avatar interface with gesture classification received the highest overall score. These findings underscore the critical importance of the appearance and inter-face of conversational agents in facilitating effective interaction.