A Formative Evaluation Methodology for VR Training Simulations

Alec G. Moore, Xinyu Hu, James Coleman Eubanks, Afham Ahmed Aiyaz, Ryan P. McMahan

Abstract: Virtual reality (VR) has a long history of being used to simulate real-world tasks, often for training and educational purposes. Yet, developing a VR system that is usable and effective is not trivial and is still a challenging task. In this paper, we present a methodology for formatively evaluating the usability of VR simulations based on capturing a real-world task, automating data logging, running a rigorous user study, analyzing subtask user performances, and inspecting those subtasks for which users performed worse in terms of usability. Using this methodology, we present the formative evaluation of a VR simulation designed for training troubleshooting skills during robotic surgeries. The results of our formative evaluation clearly demonstrated usability issues for some of our targeted training subtasks and we discuss how those issues will be addressed.