Combining LLMs and Term Rewriting for Marking Algebraic Expressions in Physics Exams

Peter Baumgartner, Lachlan McGinness

Keywords: Large Language Models, Automated Grading, Term Rewriting, Theorem Proving, Physics

TL;DR: A neuro-symbolic approach for automated marking of physics exams.

Abstract: We present our method for automatically marking Physics exams. The marking problem consists in assessing typed student answers for correctness with respect to a ground truth solution. This is a challenging problem that we seek to tackle using a combination of a computer algebra system, an SMT solver and a term rewriting system. A Large Language Model is used to interpret and remove errors from student responses and rewrite these in a machine readable format. Once formalized and language-aligned, the next step then consists in applying automated reasoning techniques for assessing student solution correctness. We consider two methods of automated theorem proving: off-the-shelf SMT solving and a term rewrite system tailored for physics problems involving trigonometric expressions. We report on experiments with these two systems on a rich pool of real-world student exam responses from the 2023 Australian Physics Olympiad.

Submission Number: 3