On Leakage of Code Generation Evaluation Datasets
Abstract: In this paper we consider contamination by code generation test sets, in particular in their use in modern large language models.
We discuss three possible sources of such contamination and show findings supporting each of them: (i) direct data leakage, (ii) indirect data leakage through the use of synthetic data and (iii) overfitting to evaluation sets during model selection.
Key to our findings is a new dataset of 161 prompts with their associated python solutions, dataset which we plan to release with this paper.
Paper Type: Short
Research Area: Resources and Evaluation
Research Area Keywords: automatic evaluation, corpus creation
Contribution Types: Model analysis & interpretability, Data resources, Data analysis
Languages Studied: English
Submission Number: 1343
Loading