An Empirical Study on the Usage of BERT Models for Code Completion

Abstract

Code completion is one of the main features of modern Integrated Development Environments (IDEs). Its objective is to speed up code writing by predicting the next code token(s) the developer is likely to write. Research in this area has substantially bolstered the predictive performance of these techniques. However, the support to developers is still limited to the prediction of the next few tokens to type. In this work, we take a step further in this direction by presenting a large-scale empirical study aimed at exploring the capabilities of state-ofthe-art deep learning (DL) models in supporting code completion at different granularity levels, including single tokens, one or multiple entire statements, up to entire code blocks (e.g., the iterated block of a for loop). To this aim, we train and test several adapted variants of the recently proposed RoBERTa model, and evaluate its predictions from several perspectives, including: (i) metrics usually adopted when assessing DL generative models (i.e., BLEU score and Levenshtein distance); (ii) the percentage of perfect predictions (i.e., the predicted code snippets that match those written by developers); and (iii) the "semantic" equivalence of the generated code as compared to the one written by developers. The achieved results show that BERT models represent a viable solution for code completion, with perfect predictions ranging from ∼7%, obtained when asking the model to guess entire blocks, up to ∼58%, reached in the simpler scenario of few tokens masked from the same code statement.

Evaluating the Usability and Functionality of Intelligent Source Code Completion Assistants: A Comprehensive Review

“…Similar to traditional assistants, intelligent ones allow developers to explore APIs by displaying a list of all available methods and attributes. However, these results are typically ranked by relevance rather than in alphabetical order [32]. With the help of these data, they can determine the developer's intention and generate a proposal that is as relevant as possible to the developer.…”

Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study

“…3) Empirical Studies on Auto-Completion Models: Ciniselli et al [29,30] analyzed the performance of two language models for text namely, T5 [17] and RoBERTa [16], for completing code in three granularity levels; single-token, line, and block. The authors included two datasets, containing Java methods and Android app methods from open-source GitHub repositories.…”

Can ChatGPT Replace a Template-based Code Generator?

“…Both approaches have their flaws. Due to their nature, DSL languages match specific domains, and will never become general-purpose tools; generative LLMs have difficulty extracting complex coding patterns from code corpora, and often generate codes riddled with syntax or semantic errors [12], [13]. The results returned by either model are seldom predictable or replicable.…”