Towards a DHH Accessible Theater: Real-Time Synchronization of Subtitles and Sign Language Videos with ASR and NLP Solutions

Grigoris Bastas, Maximos A. Kaliakatsos-Papakostas, Georgios Paraskevopoulos, Pantelis I. Kaplanoglou, Konstantinos Christantonis, Charalampos Tsioustas, Dimitris Mastrogiannopoulos, Depy Panga, Evita F. Fotinea, Athanasios Katsamanis, Vassilis Katsouros, Konstantinos I. Diamantaras, Petros Maragos

Abstract: Despite the fact that in some areas of cultural life, as in the case of certain online video platforms or TV programs, notable progress has been made to provide content accessible to Deaf and Hard of Hearing people (DHH), the same cannot be said for live theater performances. In this work, a system called NLP-Theatre is presented, with the emphasis given to its technical aspects relevant to the problem of DHH inclusivity. The leading concept behind this system’s functioning is the automatic real-time extraction of the actors’ speech transcripts and their matching to a pre-arranged set of subtitles that correspond to the theater script at hand. The matching accuracy scores indicate the correct timings for the display of each subtitle to the audience during a live performance and the concurrent activation of several other events that render possible the synchronization of multiple streams of information directed towards the spectators. One such stream is also the video of sign language interpretation that is also synchronized in real-time with the live performance and mainly targets DHH individuals. A subjective evaluation of the subtitles and sign language streams is presented based on questionnaire responses provided by the guests in an experimental theater show. Furthermore, purely technical experiments were run, centered around the performance assessment of the Speech-To-Text (STT) alternatives that are employed in the system, that is both a commercial general-purpose remote solution and a custom ASR service. The latter employs a script-specific TDNN-HMM model that runs locally and is created using a Language Model (LM) adaptation strategy.