An Empirical Study on the Usage of Transformer Models for Code Completion

Ciniselli, Matteo; Cooper, Nathan; Pascarella, Luca; Mastropaolo, Antonio; Aghajani, Emad; Poshyvanyk, Denys; Penta, Massimiliano Di; Bavota, Gabriele

doi:10.1109/tse.2021.3128234

Cited by 44 publications

(39 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The goal of this study is to understand the extent to which DL-based code recommenders are prone to suggest code snippets being clones of instances present in the training set. The context is represented by the four training datasets described in Section 2.2 and by the snippets of code generated by a state-of-the-art code recommender built on top of the Text-To-Text-Transfer-Transformer (T5) [12].…”

Section: Methodsmentioning

confidence: 99%

“…Indeed, while we have access to Copilot, the dataset used to train it is not publicly available, making it impossible to check whether the code it recommends is a clone of the instances in its training set. For this reason, we focused on another DL-based code recommender recently proposed by Ciniselli et al [12].…”

Section: Study Context: Dl-based Code Recommendermentioning

confidence: 99%

“…Most of the proposed tools were limited to the prediction of just a single token, and only recent studies focused on predicting multiple contiguous tokens or even entire statements [3,12,53]. Building on top of this research, OpenAI and GitHub presented GitHub Copilot [10], a code assistant that achieved state-of-the-art performance in code recommendation.…”

Section: Introductionmentioning

confidence: 99%

“…Unfortunately, such an analysis is not possible using GitHub Copilot since the training dataset is not publicly available. For this reason, we adopt the Text-To-Text-Transfer-Transformer (T5), a DL model that has been used to support several code-related tasks [43], including code completion [12].…”

Section: Introductionmentioning

confidence: 99%

“…We start from the pre-trained T5 made available by Ciniselli et al [12], who also made publicly available the pre-training dataset. The model has been pre-trained on ∼2M Java methods.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set?

Ciniselli¹,

Pascarella²,

Bavota³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Deep Learning (DL) models have been widely used to support code completion. These models, once properly trained, can take as input an incomplete code component (e.g., an incomplete function) and predict the missing tokens to finalize it. GitHub Copilot is an example of code recommender built by training a DL model on millions of open source repositories: The source code of these repositories acts as training data, allowing the model to learn "how to program". The usage of such a code is usually regulated by Free and Open Source Software (FOSS) licenses, that establish under which conditions the licensed code can be redistributed or modified. As of Today, it is unclear whether the code generated by DL models trained on open source code should be considered as "new" or as "derivative" work, with possible implications on license infringements. In this work, we run a large-scale study investigating the extent to which DL models tend to clone code from their training set when recommending code completions. Such an exploratory study can help in assessing the magnitude of the potential licensing issues mentioned before: If these models tend to generate new code that is unseen in the training set, then licensing issues are unlikely to occur. Otherwise, a revision of these licenses urges to regulate how the code generated by these models should be treated when used, for example, in a commercial setting. Highlights from our results show that ∼10% to ∼0.1% of the predictions generated by a state-of-the-art DL-based code completion tool are Type-1 clones of instances in the training set, depending on the size of the predicted code. Long predictions are unlikely to be cloned. CCS CONCEPTS• Software and its engineering → Software maintenance tools.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Study Context: Dl-based Code Recommendermentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…We start from the pre-trained T5 made available by Ciniselli et al [12], who also made publicly available the pre-training dataset. The model has been pre-trained on ∼2M Java methods.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set?

Ciniselli¹,

Pascarella²,

Bavota³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Statistical Models and Machine Learning to Advance Code Completion: Are We There Yet?

Nguyen¹

2023

Natural Computing Series

View full text Add to dashboard Cite

A methodology for refined evaluation of neural code completion approaches

Rashidi

Andrzejak

2022

Data Min Knowl Disc

View full text Add to dashboard Cite

Code completion has become an indispensable feature of modern Integrated Development Environments. In recent years, many approaches have been proposed to tackle this task. However, it is hard to compare between the models without explicitly re-evaluating them due to the differences of used benchmarks (e.g. datasets and evaluation metrics). Besides, almost all of these works report the accuracy of the code completion models as aggregated metrics averaged over all types of code tokens. Such evaluations make it difficult to assess the potential improvements for particularly relevant types of tokens (i.e. method or variable names), and blur the differences between the performance of the methods. In this paper, we propose a methodology called Code Token Type Taxonomy (CT3) to address the issue of using aggregated metrics. We identify multiple dimensions relevant for code prediction (e.g. syntax type, context, length), partition the tokens into meaningful types along each dimension, and compute individual accuracies by type. We illustrate the utility of this methodology by comparing the code completion accuracy of a Transformer-based model in two variants: with closed, and with open vocabulary. Our results show that the refined evaluation provides a more detailed view of the differences and indicates where further work is needed. We also survey the state-of-the-art of Machine Learning-based code completion models to illustrate that there is a demand for a set of standardized benchmarks for code completion approaches. Furthermore, we find that the open vocabulary model is significantly more accurate for relevant code token types such as usage of (defined) variables and literals.

show abstract

An Empirical Study on the Usage of Transformer Models for Code Completion

Cited by 44 publications

References 60 publications

To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set?

To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set?

Statistical Models and Machine Learning to Advance Code Completion: Are We There Yet?

A methodology for refined evaluation of neural code completion approaches

Contact Info

Product

Resources

About