Classifying Course Discussion Board Questions using LLMs

Paul Zhang, Brandon Jaipersaud, Jimmy Ba, Andrew Petersen, Lisa Zhang, Michael R. Zhang

Abstract: Large language models (LLMs) can be used to answer student questions on course discussion boards, but there is a risk of LLMs answering questions they are unable to address. We propose and evaluate an LLM-based system that classifies student questions into one of four types: conceptual, homework, logistics, and not answerable. We then prompt an LLM using a type-specific prompt. Using GPT-3, we achieve 81% classification accuracy across the four categories. Furthermore, we achieve 93% accuracy on classifying not answerable questions. This indicates that our system effectively ignores questions that it cannot address.