Can Open Source LLMs Generate Math Questions at Graduate Level Courses? An Empirical Study of Linear Algebra Course

Alka Bhushan, Nirav Bhatt, Anik Bhowmick, Patrick Vincent Ndowo, Sukanya Tukaram Naik, Ramadhan Mwinyi Pembe, Adhil Ahmed P. M. Shums

Abstract: This study investigates the suitability of existing pre-trained LLMs in generating questions for linear algebra courses at the graduate level. A total of ten LLMs are included in the study where eight LLMs are pre-trained on a Math corpora and two models are pre-trained on a general text corpora. The study is done in two phases: (I) In the Phase I, LLMs are tested for their understanding of graduate-level linear algebra text and their ability to generate relevant questions. Based on the analysis, three suitable LLMs among ten LLMs were selected for Phase II study, (II) In Phase II, these selected LLMs were tested for versatility, quality, diversity, and subject knowledge. The evaluation was done by seven experts who studied mathematics including linear algebra at graduate levels in diverse geographical locations. The results show that these LLMs understand linear algebra concepts and can generate diverse, and versatile questions with lower-order thinking skills. However, the questions generated by LLMs are not always on par with graduate-level standards.

External IDs:dblp:conf/aied/BhushanBBNNPS25