A Wizard or a Fool? Initial Assessment of a Wizard of Oz Agent Supporting Collaborative Virtual Environments

James Simpson, Michael J. Richardson, Deborah Richards

Abstract: A popular technique for evaluating envisioned AI agents without committing significant engineering resources to implement such an agent is the Wizard of Oz (WoZ) methodology. Thus far, many of the WoZ studies concerned with the design of conversational agents have been standalone systems for individual participants to interact with. The aim of this work was to determine whether a WoZ conversational agent could be embedded within an existing team-based collaborative virtual environment. In doing so, we uncovered a number of potential challenges in designing such a system which may be valuable for those implementing such a system in the future.