Scalable Mentoring Support with a Large Language Model Chatbot

Hassan Soliman, Milos Kravcik, Alexander Tobias Neumann, Yue Yin, Norbert Pengel, Maike Haag

Abstract: Education students engage in diverse learning activities requiring appropriate assistance and timely feedback. As their numbers grow, providing them with scalable support is an important challenge. Here, we focus on the development of a didactic chatbot based on a Large Language Model (LLM). The potential of LLMs is enhanced by existing materials and pedagogical course descriptions. Using Retrieval Augmented Generation (RAG), the bot can retrieve and analyse course materials, in order to provide comprehensive answers to specific questions. Preliminary results indicate that it is possible to distinguish between different student contexts and to generate a prompt answer, taking into account the relevant materials. The evaluation results achieved 84.78% accuracy in providing correct answers for seminar materials.