From First Draft to Final Insight: A Multi-agent Approach for Feedback Generation

Jie Cao, Chloe Qianhui Zhao, Xian Chen, Shuman Wang, Christian Schunn, Kenneth R. Koedinger, Jionghao Lin

Abstract: Producing large volumes of high-quality, timely feedback poses significant challenges to instructors. To address this issue, automation technologies—particularly Large Language Models (LLMs)—show great potential. However, current LLM-based research still shows room for improvement in terms of feedback quality. Our study proposed a multi-agent approach performing “generation, evaluation, and regeneration” (G-E-RG) to further enhance feedback quality. In the first-generation phase, six methods were adopted, combining three feedback theoretical frameworks and two prompt methods: zero-shot and retrieval-augmented generation with chain-of-thought (RAG_CoT). The results indicated that, compared to first-round feedback, G-E-RG significantly improved final feedback across six methods for most dimensions. Specifically:(1) Evaluation accuracy for six methods increased by \(3.36\%\) to \(12.98\%\) (\(p<0.001\)); (2) The proportion of feedback containing four effective components rose from an average of \(27.72\%\) to an average of \(98.49\%\) among six methods, sub-dimensions of providing critiques, highlighting strengths, encouraging agency, and cultivating dialogue also showed great enhancement (\(p<0.001\)); (3) There was a significant improvement in most of the feature values (\(p<0.001\)), although some sub-dimensions (e.g., strengthening the teacher-student relationship) still require further enhancement; (4) The simplicity of feedback was effectively enhanced (\(p<0.001\)) for three methods.

External IDs:dblp:conf/aied/CaoZCWSKL25