NLP-Theatre: Employing Speech Recognition Technologies for Improving Accessibility and Augmenting the Theatrical Experience

Alkiviadis Katsalis, Konstantinos Christantonis, Charalampos Tsioustas, Pantelis I. Kaplanoglou, Maximos A. Kaliakatsos-Papakostas, Athanasios Katsamanis, Konstantinos I. Diamantaras, Vassilis Katsouros, Evita F. Fotinea, Depy Panga, Dimitra Loupi

Abstract: This paper presents a system that automates activation of events that improve the accessibility and enhance the experience in theatrical performances in real time and proposes and evaluates the core method employed therein. This method aligns a given set of subtitles that is created and synchronized by experts for a given “rehearsal” audio stream, to a new “performance” audio stream in real-time. The performance stream may be different from the rehearsal not only in terms of timing, but also in terms of spoken content. The method is built around an Automatic Speech Recognition System (ASR) that captures the performance stream in real-time and generates suggestions for subtitle timing corrections. The system is evaluated in an experimental performance with 35 participants while the core method is analysed independently, using sixteen artificial rehearsal-performance pair scenarios. Objective and subjective evaluation reveals specific directions for improvements for the core method. The investigation of the system shows that this innovative approach is promising for improving accessibility and the enhancement of the theatrical experience of audiences.