Development and evaluation of a semi-autonomous parallel attentive listening system

Divesh Lala, Koji Inoue, Haruki Kawai, Zi Haur Pang, Mikey Elmers, Tatsuya Kawahara

Abstract: Open domain spoken dialogue systems are still not at the level of human performance and understanding. In this work we propose that the a remote human operator who can take over the conversation from an autonomous system would improve the quality of an attentive listening conversation. Furthermore this operator could also manage multiple users simultaneously. We describe and implement this as a semi-autonomous parallel system. Features of this system are detection of disengagement to let the operator know when to intervene, summarization of conversations using ChatGPT to allow the operator to manage multiple users, and conversion of the operator’s voice to the agent’s to make intervention less abrupt. We conduct an experiment to compare this system to a fully autonomous system and find that it improves performance for enjoyment and empathy.