Towards Effective Communication with Robotic Assistants for the Elderly: Integrating Speech, Vision and Haptics

Barbara Di Eugenio, Milos Zefran, Jezekiel Ben-Arie, Marquis Foreman, Lin Chen, Simone Franzini, Shankaranand Jagadeesan, Maria Javaid, Kai Ma

Abstract: Our goal is to develop an interface for older people to effectively communicate with a robotic assistant so that they can safely remain living in their home. We are devising a multimodal interface since people communicate with one another using a variety of verbal and non-verbal signals, including haptics, i.e., physical interactions. We view haptics as an integral component of communication, which in some cases drives the interaction between the user and the robot, and we study its relation to speech and gestures. We illustrate features of interactions between an elderly person and an assistant via excerpts from our ongoing data collection. We also describe the architecture of our interface and ongoing research to bring this interface to fruition.