Observing How Students Program with ChatGPT: A Quantitative Eye-Tracking Assesment of Visual Expertise Acquisition

Anonymous

Abstract: The proliferation of language models is revolutionizing the process of Human-AI Interaction (HAI), offering users a conversational interface to accomplish various tasks and access information. Understanding how these models affect the way students learn the skill of computer programming remains an unstudied area of research. This paper presents an experiment designed to investigate the interaction dynamics of undergraduate students with varying computer programming abilities when utilizing ChatGPT, as an AI-assisted tool to accomplish coding tasks. Eye-tracking technology is employed to capture participants' gaze patterns and visual attention during their interactions with the language model. The paper presents the analysis of a total of 120 eye tracking cases. Using the Kruskal-Wallis statistical test to assess whether students selectively accord attention to programming tasks based on their perceived importance and complexity, we find that significant differences ($p$ < .001) across the `hit time', `time to the first fixation' and the `areas of interest duration' eye tracking features. The results shed light on differences in visual attention patterns, the utilization of AI-generated suggestions, code comprehension strategies, and preferences for interacting with ChatGPT during coding tasks.

Paper Type: short

Research Area: Computational Social Science and Cultural Analytics

Contribution Types: Model analysis & interpretability, Data analysis, Surveys

Languages Studied: English

Consent To Share Submission Details: On behalf of all authors, we agree to the terms above to share our submission details.