SCELMo: Source Code Embeddings from Language Models

Rafael - Michael Karampatsis, Charles Sutton

Keywords: Transfer Learning, Pretraining, Program Repair

TL;DR: A new set of deep contextualized word representations for computer programs based on language models.

Abstract:

Continuous embeddings of tokens in computer programs have been used to support a variety of software development tools, including readability, code search, and program repair.
Contextual embeddings are common in natural language processing but have not been previously applied in software engineering.
We introduce a new set of deep contextualized word representations for computer programs based on language models.
We train a set of embeddings using the ELMo (embeddings from language models) framework of Peters et al (2018).
We investigate whether these embeddings are effective when fine-tuned for the downstream task of bug detection.
We show that even a low-dimensional embedding trained on a relatively small corpus of programs can improve a state-of-the-art machine learning system for bug detection.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2004.13214/code)