An Empirical Study on the Usage of BERT Models for Code Completion

An Empirical Study on the Usage of Transformer Models for Code Completion

“…In this work, we extend our MSR 2021 paper [22] by showing that the T5 substantially overcomes the performance of the RoBERTa model, being able to correctly predict even entire code blocks, something that we found to be not achievable with RoBERTa. As in [22], we focus on three code prediction scenarios: (i) token-level predictions, namely classic code completion in which the model is used to guess the last n tokens in a statement the developer started writing; (ii) construct-level predictions, in which the model is used to predict specific code constructs (e.g., the condition of an if statement) that can be particularly useful to developers while writing code; and (iii) block-level predictions, with the masked code spanning one or more entire statements composing a code block (e.g., the iterated block of a for loop).…”

“…Such research has allowed to move from simple alphabetically ranked lists of recommendations for completing what a developer is typing (e.g., a list of possible method calls matching what has been typed by the developer) to "in-telligent" completions considering the context surrounding the code [17], [66], the history of code changes [66], and/or coding patterns mined from software repositories [9], [36], [38], [59], [60], [61], [72]. Last, but not least, Deep Learning (DL) models have been applied to code completion [7], [22], [45], [47], [68], [77], setting new standards in terms of prediction performance. Although the performance of code completion techniques have substantially improved over time, the type of support they provide to developers has not evolved at the same pace.…”

“…Among the many DL models proposed in the literature, we focus on models using the Transformer architecture [75]. In particular, in our recent work published at MSR 2021 [22] we evaluated the performance of a RoBERTa model [53] in the code completion tasks described above. RoBERTa is a BERT (Bidirectional Encoder Representations from Transformers) model [24] using a pre-training task in which random words in the input sentences are masked out using a special <MASK> token, with the model in charge of predicting the masked words.…”

“…This would not be realistic in a real usage scenario, in which the code completion engine must guess the tokens to generate, without the developer suggesting how many tokens must be generated. To overcome this limitation, we had to adapt the RoBERTa pre-training objective to be able to guess, from a single <MASK> token masking one or more code tokens in the given statements, which and how many code tokens must be generated [22]. The adaptation of the RoBERTa pre-training objective was inspired by the recently proposed Text-To-Text Transfer Transformer (T5) architecture [63], suggesting this as a good fit for the task of code completion.…”

Multi-View Graph Representation for Programming Language Processing: An Investigation into Algorithm Detection

“…For examples, Harer et al (2018), Ben-Nun, Jakobovits, and Hoefler (2018), and Zuo et al (2019 apply the word2vec model (Le and Mikolov 2014) to learn the embeddings of program tokens. Feng et al (2020), Wang et al (2020) and Ciniselli et al (2021) use a pre-trained BERT model to encode programs. Such sequence-based methods are easy to use and can benefit largely from the NLP community.…”