Using Qualitative Data from Targeted Interviews to Inform Rapid AIED Development

Jaclyn Ocumpaugh, Stephen Hutt, Juliana Ma. Alexandra L. Andres, Ryan Baker, Gautam Biswas

Abstract: This paper examines how interviews with students—at critical moments of the learning process—may be leveraged to improve the design of educational software. Specifically, we discuss iterative work to improve the design of a pedagogical agent in the Betty’s Brain learning environment, Mr. Davis. Students interacted with the pedagogical agent in Betty’s Brain during two separate studies, two months apart. During study one, qualitative interviews were prompted by student actions within the system and theoretically aligned sequences of educationally relevant affective states (as detected by previously validated models). Facilitaed by an app called the Quick Red Fox (QRF), these in situ interviews were then used to identify ways to rapidly improve Mr. Davis’ design, investigated in study two. Results indicate that changes designed to make Mr. Davis more empathetic correlate with improved learning outcomes. We also discuss the potential for rapidly collected qualitative data in future developments.