Crowdsourcing Multimodal Dialog Interactions: Lessons Learned from the HALEF Case

Vikram Ramanarayanan, David Suendermann-Oeft, Hillary Molloy, Eugene Tsuprun, Patrick L. Lange, Keelan Evanini

Abstract: We present a retrospective on collecting data of human interactions with multimodal dialog systems ("dialog data") using crowdsourcing techniques. This is largely based on our experience using the HALEF multimodal dialog system to deploy education-domain conversational applications on the Amazon Mechanical Turk crowdsourcing platform. We list the various lessons learned from this endeavor over two years and make recommendations on best practices to be followed by practitioners and researchers looking to crowdsource dialog data for a new domain.