The Trust Divide: Chatbots’ Superior Performance and Skeptical Students
Keywords: chatbots, education, trustworthiness, quality, human computer interaction
TL;DR: Exploring the gap between chatbots' perceived trustworthiness and actual quality in education, this study reveals the undervalued potential of tools like ChatGPT, despite students' skepticism..
Abstract: Chatbots have emerged as fundamental components of innovative teaching approaches to address the significant challenge of catering to students' diverse learning backgrounds with limited resources. However, their usefulness depends on their ability to provide high-quality, reliable outcomes and create trustworthiness. Against this background, the present study investigated the trustworthiness of chatbots in supporting educational tasks and the extent to which this trustworthiness is justified. The study involved 189 students from vocational nursing schools who created medical care plans with and without ChatGPT's (Model GPT-4) support. Additionally, ChatGPT was asked to solve the tasks without students being involved. Experts then evaluated all three sets of plans. This allowed us to compare the quality of the answers solved by the chatbot to the solutions by the students. To examine the trustworthiness of the chatbot, another independent care plan was evaluated by the students following experimentally manipulated feedback that it was created either by a human or a chatbot. Statistical analyses revealed that students' beliefs about the source of the care plan -explicitly informed as either chatbot-generated or human-created- significantly affected their perceptions of trustworthiness. Students who believed a chatbot solved the task perceived it as less trustworthy than those who thought a human created the plan. However, analyses of the expert ratings revealed that the solutions created by the chatbot were actually of higher quality than those created by humans. Therefore, the mistrust in the chatbots was not justified. This finding suggests that students underestimate chatbot performance in educational settings.
Submission Number: 24
Loading