Machine-Generated Questions Attract Instructors When Acquainted with Learning Objectives

Machi Shimmei, Norman L. Bier, Noboru Matsuda

Abstract: Answering questions is an essential learning activity on online courseware. It has been shown that merely answering questions facilitates learning. However, generating pedagogically effective questions is challenging. Although there have been studies on automated question generation, the primary research concern thus far is about if and how those question generation techniques can generate answerable questions and their anticipated effectiveness. We propose Quadl, a pragmatic method for generating questions that are aligned with specific learning objectives. We applied Quadl to an existing online course and conducted an evaluation study with in-service instructors. The results showed that questions generated by Quadl were evaluated as on-par with human-generated questions in terms of their relevance to the learning objectives. The instructors also expressed that they would be equally likely to adapt Quadl-generated questions to their course as they would human-generated questions. The results further showed that Quadl-generated questions were better than those generated by a state-of-the-art question generation model that generates questions without taking learning objectives into account.