Towards a Smart Wearable Tool to Enable People with SSPI to Communicate by Sentence Fragments

Gyula Vörös, Anita Verö, Balázs Pintér, Brigitta Miksztai-Réthey, Takumi Toyama, András Lörincz, Daniel Sonntag

Abstract: The ability to communicate with others is of paramount importance for mental well-being. In this paper, we describe an interaction system to reduce communication barriers for people with severe speech and physical impairments (SSPI) such as cerebral palsy. The system consists of two main components: (i) the head-mounted human-computer interaction (HCI) part consisting of smart glasses with gaze trackers and text-to-speech functionality (which implement a communication board and the selection tool), and (ii) a natural language processing pipeline in the backend in order to generate complete sentences from the symbols on the board. We developed the components to provide a smooth interaction between the user and the system thereby including gaze tracking, symbol selection, symbol recognition, and sentence generation. Our results suggest that such systems can dramatically increase communication efficiency of people with SSPI.