Multilingual Code Retrieval Without Paired Data: New Datasets and Benchmarks
Primary Area: datasets and benchmarks
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Multilingual data, Code retrieval from text, Contrastive learning, Generalization to unseen conditions
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
TL;DR: We introduce a new multi-lingual text to code benchmark and a training procedure that enables generalization to unseen language combinations.
Abstract: We seek to overcome limitations to code retrieval quality posed by the scarcity of data containing pairs of code snippets and natural language queries in languages other than English. To do so, we introduce two new datasets. First, we make a new evaluation benchmark available, dubbed M$^2$CRB, containing pairs of text and code, for multiple natural and programming language pairs - namely: Spanish, Portuguese, German, and French, each paired with code snippets for: Python, Java, and JavaScript. The dataset is curated via an automated filtering pipeline from source files within GitHub followed by human verification to ensure accurate language classification. Additionally, in order be able to train models and evaluate on the proposed task, we pose the following hypothesis: if a model can map from English to code, and from other natural languages to English, then the model can directly map from those non-English languages into code. We thus build a training corpus made of a new paired English/Code dataset we curate, and further combine it with existing translation datasets given by pairs of English and other natural languages. Extensive evaluations on both our new tasks as well as on existing code-to-code search benchmarks confirm our hypothesis: models are able to generalize to unseen language pairs they indirectly observed during training. We examine a broad set of model classes and report the influence of different design choices on the observed generalization capabilities.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 5254
Loading