On the Variations of ChatGPT's Response Quality for Generating Source Code Across Programming Languages

Ángela González de Diego, Franz Wotawa

Abstract: The rise of Large Language Models, particularly the ChatGPT model, has transformed the field of natural language information processing and has led to widespread adoption in a diverse range of applications and across a multitude of industries. In this paper, we focus on assessing the quality of the responses generated by Chat-GPT for the code generation tasks using seven different programming languages. We selected the languages considering diversity in terms of the fields of application, philosophies, and popularity. We carried out an experimental evaluation utilizing different introductory coding examples for each of the programming languages using the pass@k metric for evaluation. The results indicate a correlation between the effectiveness of the model and the popularity of programming languages.