Fast and Memory-Efficient Neural Code Completion

Svyatkovskiy, A.; Lee, Sebastian; Hadjitofi, Anna; Riechert, Maik; Franco, Juliana; Allamanis, Miltiadis

doi:10.1109/msr52588.2021.00045

Cited by 47 publications

(17 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Svyatkovskiy et al [59] proposed a different perspective on neural code completion, shifting from a generative task to a learning-to-rank task. Their model is used to rerank the recommendations provided via static analysis, being cheaper in terms of memory footprint as compared to generative models.…”

Section: Related Workmentioning

confidence: 99%

An Empirical Study on the Usage of BERT Models for Code Completion

Ciniselli¹,

Cooper²,

Pascarella³

et al. 2021

Preprint

View full text Add to dashboard Cite

Code completion is one of the main features of modern Integrated Development Environments (IDEs). Its objective is to speed up code writing by predicting the next code token(s) the developer is likely to write. Research in this area has substantially bolstered the predictive performance of these techniques. However, the support to developers is still limited to the prediction of the next few tokens to type. In this work, we take a step further in this direction by presenting a large-scale empirical study aimed at exploring the capabilities of state-ofthe-art deep learning (DL) models in supporting code completion at different granularity levels, including single tokens, one or multiple entire statements, up to entire code blocks (e.g., the iterated block of a for loop). To this aim, we train and test several adapted variants of the recently proposed RoBERTa model, and evaluate its predictions from several perspectives, including: (i) metrics usually adopted when assessing DL generative models (i.e., BLEU score and Levenshtein distance); (ii) the percentage of perfect predictions (i.e., the predicted code snippets that match those written by developers); and (iii) the "semantic" equivalence of the generated code as compared to the one written by developers. The achieved results show that BERT models represent a viable solution for code completion, with perfect predictions ranging from ∼7%, obtained when asking the model to guess entire blocks, up to ∼58%, reached in the simpler scenario of few tokens masked from the same code statement.

show abstract

Section: Related Workmentioning

confidence: 99%

An Empirical Study on the Usage of BERT Models for Code Completion

Ciniselli¹,

Cooper²,

Pascarella³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Svyatkovskiy et al [69] proposed a different perspective on neural code completion, shifting from a generative task to a learning-to-rank task. Their model is used to rerank the recommendations provided via static analysis, being cheaper in terms of memory footprint than generative models.…”

Section: Code Completion Approachesmentioning

confidence: 99%

An Empirical Study on the Usage of Transformer Models for Code Completion

Ciniselli,

Cooper,

Pascarella

et al. 2021

Preprint

View full text Add to dashboard Cite

Code completion aims at speeding up code writing by predicting the next code token(s) the developer is likely to write. Works in this field focused on improving the accuracy of the generated predictions, with substantial leaps forward made possible by deep learning (DL) models. However, code completion techniques are mostly evaluated in the scenario of predicting the next token to type, with few exceptions pushing the boundaries to the prediction of an entire code statement. Thus, little is known about the performance of state-of-the-art code completion approaches in more challenging scenarios in which, for example, an entire code block must be generated. We present a large-scale study exploring the capabilities of state-of-the-art Transformer-based models in supporting code completion at different granularity levels, including single tokens, one or multiple entire statements, up to entire code blocks (e.g., the iterated block of a for loop). We experimented with several variants of two recently proposed Transformer-based models, namely RoBERTa and the Text-To-Text Transfer Transformer (T5), for the task of code completion. The achieved results show that Transformer-based models, and in particular the T5, represent a viable solution for code completion, with perfect predictions ranging from ∼29%, obtained when asking the model to guess entire blocks, up to ∼69%, reached in the simpler scenario of few tokens masked from the same code statement.

show abstract

“…To evaluate quantitatively our model, we use two commonly used metrics in very similar tasks such as code completion [16,37,39], namely the Recall@k and the Mean Reciprocal Rank (MRR). We consider a recommendation as…”

Section: Effectiveness Metricsmentioning

confidence: 99%

Recommending Metamodel Concepts during Modeling Activities with Pre-Trained Language Models

Weyssow,

Sahraoui,

Syriani

2021

Preprint

View full text Add to dashboard Cite

The design of conceptually sound metamodels that embody proper semantics in relation to the application domain is particularly tedious in Model-Driven Engineering. As metamodels define complex relationships between domain concepts, it is crucial for a modeler to define these concepts thoroughly while being consistent with respect to the application domain.We propose an approach to assist a modeler in the design of a metamodel by recommending relevant domain concepts in several modeling scenarios. Our approach does not require to extract knowledge from the domain or to hand-design completion rules. Instead, we design a fully data-driven approach using a deep learning model that is able to abstract domain concepts by learning from both structural and lexical metamodel properties in a corpus of thousands of independent metamodels. We evaluate our approach on a test set containing 166 metamodels, unseen during the model training, with more than 5000 test samples. Our preliminary results show that the trained model is able to provide accurate top-5 lists of relevant recommendations for concept renaming scenarios. Although promising, the results are less compelling for the scenario of the iterative construction of the metamodel, in part because of the conservative strategy we use to evaluate the recommendations.

show abstract

Fast and Memory-Efficient Neural Code Completion

Cited by 47 publications

References 24 publications

An Empirical Study on the Usage of BERT Models for Code Completion

An Empirical Study on the Usage of BERT Models for Code Completion

An Empirical Study on the Usage of Transformer Models for Code Completion

Recommending Metamodel Concepts during Modeling Activities with Pre-Trained Language Models

Contact Info

Product

Resources

About