ReWIRED: Instructional Explanations in Teacher-Student Dialogues

Anonymous

Abstract: How to assess the quality of teaching in instructional explanation dialogues is a recurring point of debate in didactics research. For the NLP community, this is a challenging topic thus far, even with the use of LLMs. To address the matter, we create a new annotation scheme of teaching acts aligned with contemporary didactic teaching models. On this basis, we extend an existing dataset of conversational explanations about communicating scientific understanding in teacher-student settings on five levels of the explainee's expertise, with the proposed teaching annotation: explanation and dialogue acts. For better granularity, we reframe the task from a dialogue turn classification to a span labeling task. We then evaluate language models on the labeling of such acts and find that the broad range and structure of the proposed labels is hard to model for LLMs such as GPT-3.5/-4 via prompting, but a fine-tuned BERT can perform both act classification and span labeling well. Finally, we operationalize a series of quality metrics for instructional explanations in the form of a test suite. We find that they match the five expertise levels well and that experts in our data often stick to best practices in teaching.

Paper Type: long

Research Area: Resources and Evaluation

Contribution Types: NLP engineering experiment, Reproduction study, Data resources, Data analysis, Theory

Languages Studied: English