Using Cipherbot: An Exploratory Analysis of Student Interaction with an LLM-Based Educational Chatbot

Joni Salminen, Soon-Gyo Jung, Johanne Medina, Kholoud Khalil Aldous, Jinan Y. Azem, Waleed Akhtar, Bernard J. Jansen

Abstract: Cipherbot, an educational chatbot using large language models to answer student questions concerning learning materials uploaded by the educator, was pilot tested in a classroom setting. Forty-four students used Cipherbot for seven weeks, sending 8077 messages. The average number of messages sent per student was 184 (SD = 80), with an average length of 98 characters (SD = 80). The engagement followed a non-normal distribution, with few power users, implying that most students are still hesitant to adopt tools like Cipherbot. Cipherbot was able to answer 82.5% of the student questions, demonstrating a scalable ability to address students' learning queries, with some room for improvement.