Improving Assessment of Tutoring Practices using Retrieval-Augmented Generation

Zifei FeiFei Han, Jionghao Lin, Ashish Gurung, Danielle Thomas, Eason Chen, Conrad Borchers, Shivang Gupta, Ken Koedinger

Track: Innovations in AI for Education (Day 1)

Paper Length: long-paper (6 pages + references)

Keywords: Large Language Model, Personalized Tutor Training, Automatic Assessment

TL;DR: This study reports on the usage of different prompting strategies to leverage large language models in evaluating the use of social-emotional tutoring strategies by novice tutors during online tutoring sessions.

Abstract:

One-on-one tutoring is an effective instructional method for enhancing learning, yet its
efficacy hinges on tutor competencies. Novice math tutors often prioritize content-specific
guidance, neglecting aspects such as social-emotional learning. Social-emotional learning
promotes equity and inclusion and nurtures relationships with students, which is crucial
for holistic student development. Assessing the competencies of tutors accurately and effi-
ciently can drive the development of tailored tutor training programs. However, evaluating
novice tutor ability during real-time tutoring remains challenging as it typically requires
experts-in-the-loop. To address this challenge, this study harnesses Generative Pre-trained
Transformers (GPT), such as GPT-3.5 and GPT-4, to automatically assess tutors’ abil-
ity of using social-emotional tutoring strategies. Moreover, this study also reports on the
financial dimensions and considerations of employing these models in real-time and at
scale for automated assessment. Four prompting strategies were assessed: two basic Zero-
shot prompt strategies, Tree of Thought prompting, and Retrieval-Augmented Generator
(RAG) prompting. The results indicate that RAG prompting demonstrated the most ac-
curate performance (assessed by the level of hallucination and correctness in the generated
assessment texts) and the lowest financial costs. These findings inform the development
of personalized tutor training interventions to enhance the the educational effectiveness of
tutored learning.

Submission Number: 59