The Multimodal Correction Detection Problem

Amos Azaria, Keren Nivasch

Abstract: In order for socially aware agents to be truly useful, they should have abilities associated with human intelligence, such as the ability to detect their own mistakes from user reactions. This is an instance of implicit feedback. In this work we address the problem of detecting an agent's mistakes by identifying when the user tries to correct the agent. We refer to this problem as the Correction Detection task. We use a multimodal approach, using both the voice (acoustics and non-verbal sounds) as well as the transcript of the user's spoken commands.