What factors will affect the effectiveness of using ChatGPT to solve programming problems? A Quasi-Experimental Study Targeting University Learners

Yuhui Jing, Haoming Wang, Xiaojiao Chen, Chengliang Wang

Abstract: The emergence of ChatGPT has ignited new expectations for AI-empowered educational transformation, particularly in higher education, where it has become an emerging tool for teaching and learning, giving rise to a new wave of research interest. However, there is currently limited research attempting to explore how it can effectively be leveraged in the teaching process. To address this gap, this study recruited 53 university learners, using Python graphing in programming education as an example, to investigate the factors influencing the effectiveness of university learners in applying ChatGPT to problem-solving. The findings of the study reveal that AI literacy significantly influences the learners' effectiveness in using ChatGPT to solve problems, with AI awareness and AI utilization being key factors. The systematic knowledge base significantly affects learners' effectiveness in applying ChatGPT to solve programming problems, but this effect is contingent on whether learners possess a systematic knowledge base. Having a systematic knowledge base does not significantly impact the knowledge base level. Furthermore, the cognitive level of university learners regarding ChatGPT significantly influences their effectiveness in applying ChatGPT to problem-solving, while their intention to use it for teaching does not have a significant impact. However, their intention to use ChatGPT significantly increases. Based on these findings, this study proposes that in the process of empowering education with Artificial Intelligence Generated-Content (AIGC) products, the focus on university learners should shift from cultivating their AI usage literacy to fostering AI literacy, thus laying the foundation for empowered learning with AIGC products. The transition should also shift from acquiring specific knowledge to employing graph-based rules as a method for empowering learning with AIGC products. Additionally, the emphasis should shift from enhancing university learners' intention to use the technology to strengthening their technological awareness, thereby creating practical pathways for empowered learning with AIGC products.