Heterogeneous Treatment Effects of Learning Analytics Dashboards: Do All Learners Benefit Equally?

Aylin Ozturk, Robin Schmucker, Tom Mitchell, Alper Tolga Kumtepe

Abstract: This study investigates the heterogeneity in the effects of a Learning Analytics Dashboard (LAD) intervention, which provides personalized feedback messages, across a diverse population of learners. Specifically, it evaluates the I'mpact of the LAD on learners’ total material usage and final grades, considering variables such as age, sex, prior material access level, prior academic performance, and first-year status. The intervention was implemented through a controlled experimental design with 26,753 learners over 17 weeks in a fully online course. Participants were randomly assigned to either the experimental (n = 13,377) or control (n = 13,376) groups. Results indicate significant heterogeneity in the effects of the LAD intervention within the online learning environment. The LAD improved learning material usage and academic performance, with factors such as sex, age, prior course material access, and prior academic performance moderating these effects. Female learners, older learners, and those with low prior academic performance benefited more from the intervention. This suggests that personalized learning analytics tools can enhance engagement and outcomes for specific learner subgroups, reinforcing the need for targeted interventions in online education.