Multimodal HALEF: An Open-Source Modular Web-Based Multimodal Dialog Framework

Zhou Yu, Vikram Ramanarayanan, Robert Mundkowsky, Patrick L. Lange, Alexei V. Ivanov, Alan W. Black, David Suendermann-Oeft

Abstract: We presentYu, Zhou an open-source web-based multimodal dialogRamanarayanan, Vikram framework, “Multimodal HALEF”, that integratesMundkowsky, Robert video conferencing and telephonyLange, Patrick abilitiesIvanov, Alexei into the existing HALEF cloud-based dialogBlack, Alan W. framework via the FreeSWITCH video telephonySuendermann-Oeft, David server. Due to its distributed and cloud-based architecture, Multimodal HALEF allows researchers to collect video and speech data from participants interacting with the dialog system outside of traditional lab settings, therefore largely reducing cost and labor incurred during the traditional audio-visual data collection process. The framework is equipped with a set of tools including a web-based user survey template, a speech transcription, an annotation and rating portal, a web visual processing server that performs head tracking, and a database that logs full-call audio and video recordings as well as other call-specific information. We present observations from an initial data collection based on an job interview application. Finally we report on some future plans for development of the framework.