SimPal: Towards a Meta-Conversational Framework to Understand Teacher's Instructional Goals for K-12 Physics

Effat Farhana, Souvika Sarkar, R. Alexander Knipper, Indrani Dey, N Narayanan, Sadhana Puntambekar, Santu Karmaker

Abstract:

Simulations are widely used to teach science in grade schools. These
simulations are often augmented with a conversational artificial
intelligence (AI) agent to provide real-time scaffolding support for
students conducting experiments using the simulations. AI agents
are highly tailored for each simulation, with a predesigned set of Instructional Goals (IGs), making it difficult for teachers to adjust IGs
as the agent may no longer align with the revised IGs. Additionally,
teachers are hesitant to adopt new third-party simulations for the
same reasons. In this research, we introduce SimPal, a Large Language Model (LLM) based meta-conversational agent, to solve this
misalignment issue between a pre-trained conversational AI agent
and the constantly evolving pedagogy of instructors. Through natural conversation with SimPal, teachers first explain their desired IGs,
based on which SimPal identifies a set of relevant physical variables
and their relationships to create symbolic representations of the
desired IGs. The symbolic representations can then be leveraged
to design prompts for the original AI agent to yield better alignment with the desired IGs. We empirically evaluated SimPal using
two LLMs, ChatGPT-3.5 and PaLM 2, on 63 Physics simulations
from PhET and Golabz. Additionally, we examined the impact of
different prompting techniques on LLM’s performance by utilizing
the TELeR taxonomy to identify relevant physical variables for the
IGs. Our findings showed that SimPal can do this task with a high
degree of accuracy when provided with a well-defined prompt.