Learning Multimodal Interaction Manager for Assistive Robots from Human-Human Data

Afagh Mehri Shervedani, Siyu Li, Natawut Monaikul, Bahareh Abbasi, Barbara Di Eugenio, Milos Zefran

Keywords: Human-Robot-Interaction, Learning from Human Demonstrations, Multimodal Data Augmentation, Dialogue Systems, User Simulator, Multimodal HRI

TL;DR: Since the submission is anonymous, we declare here that the User Simulator section of this paper has been accepted for publication previously.

Abstract:

This paper describes a Reinforcement Learning (RL) framework for developing assistive robots capable of multimodal interaction. The framework critically depends on a neural network-based human user simulator trained on the existing ELDERLY-AT-HOME corpus, accommodating multiple modalities such as language, pointing gestures, and haptic-ostensive actions. The simulator provides a multimodal interactive environment for training the Reinforcement Learning (RL) agents in collaborative tasks involving various modes of communication. In contrast to conventional dialog systems, our agent is trained using a simulator developed with human data and capable of handling multiple modalities, including language and physical actions. The paper also presents a novel multimodal data augmentation approach, which addresses the challenge of using a dataset that is small due to the expensive and time-consuming nature of collecting human demonstrations. Overall, the study highlights the potential for using RL and multimodal user simulators in developing and improving assistive robots.

Since the submission is anonymous, we declare that the User Simulator section of this paper has been accepted for publication previously.

Submission Number: 19