Inferring the Overall Difficulty of Isomorphic Questions

Bickio O'Callahan, Xinfeng Ye, Sathiamoorthy Manoharan

Abstract: Isomorphic assessments operate on the principle of using question variants that test the same learning outcome. This approach supplies each student with their own set of question variants, effectively eliminating the chance of being influenced by another student’s answer. Moreover, it presents a broad range of practice questions. This method also offers a non-punitive means to uphold academic integrity, as blindly copying answers from a peer would prove unfruitful. However, using isomorphic questions can result in unfairness. Some question variants may be more difficult than others. A common solution employed by instructors to ensure fairness is to select parametric values within a close range.The research question this paper seeks to address is the following: is it possible to design and implement a formal system that can infer the difficulty levels of question variants and, if so, restrict these difficulty levels to an acceptable range? The paper employs the Z3 solver to build a proof-of-concept system that infers difficulty levels for a class of problems involving integers and Booleans. It demonstrates that this system can determine difficulty levels for this class of problems. We observe that choosing a limited range of values does not necessarily constrain the difficulty levels; minor alterations to the values can lead to significant variations in difficulty.