Using pre-trained models to boost code review automation

Tufano, Rosalia; Masiero, Simone; Mastropaolo, Antonio; Pascarella, Luca; Poshyvanyk, Denys; Bavota, Gabriele

doi:10.1145/3510003.3510621

Cited by 64 publications

(6 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As representative of transformers [1], we adopt the T5 proposed by Raffel et al [20], that has been already used in SE to automate code-related tasks [9], [13], [14], [58], [59]. Masks X% of tokens (usually 15%) in the instance (e.g., a function) and asks the model to guess the masked tokens based on their bidirectional context.…”

Section: A Transformer Modelmentioning

confidence: 99%

Automating Code-Related Tasks Through Transformers: The Impact of Pre-training

Tufano¹,

Pascarella²,

Bavota³

2023

Preprint

View full text Add to dashboard Cite

Transformers have gained popularity in the software engineering (SE) literature. These deep learning models are usually pre-trained through a self-supervised objective, meant to provide the model with basic knowledge about a language of interest (e.g., Java). A classic pre-training objective is the masked language model (MLM), in which a percentage of tokens from the input (e.g., a Java method) is masked, with the model in charge of predicting them. Once pre-trained, the model is then finetuned to support the specific downstream task of interest (e.g., code summarization). While there is evidence suggesting the boost in performance provided by pre-training, little is known about the impact of the specific pre-training objective(s) used. Indeed, MLM is just one of the possible pre-training objectives and recent work from the natural language processing field suggest that pre-training objectives tailored for the specific downstream task of interest may substantially boost the model's performance. For example, in the case of code summarization, a tailored pretraining objective could be the identification of an appropriate name for a given method, considering the method name to generate as an extreme summary. In this study, we focus on the impact of pre-training objectives on the performance of transformers when automating code-related tasks. We start with a systematic literature review aimed at identifying the pre-training objectives used in SE. Then, we pre-train 32 transformers using both (i) generic pre-training objectives usually adopted in SE; and (ii) pre-training objectives tailored to specific code-related tasks subject of our experimentation, namely bug-fixing, code summarization, and code completion. We also compare the pretrained models with non pre-trained ones and show the advantage brought by pre-training in different scenarios, in which more or less fine-tuning data are available. Our results show that: (i) pre-training helps in boosting performance only if the amount of fine-tuning data available is small; (ii) the MLM objective is usually sufficient to maximize the prediction performance of the model, even when comparing it with pre-training objectives specialized for the downstream task at hand.

show abstract

Section: A Transformer Modelmentioning

confidence: 99%

Automating Code-Related Tasks Through Transformers: The Impact of Pre-training

Tufano¹,

Pascarella²,

Bavota³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Thongtanunam et al [41] further introduces advanced Transformer architecture and a Byte-Pair Encoding (BPE) approach to handle the Out-Of-Vocabulary and long sequence problems. To better learn code properties, pre-training techniques are increasingly adopted in the code review scenario [11,18,24,44,54]. Hong et al [18] proposes a CodeT5-based approach to recommend code review comments automatically.…”

Section: Nmt Models For Code Generationmentioning

confidence: 99%

On the Reliability and Explainability of Automated Code Generation Approaches

Liu¹,

Tantithamthavorn²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Automatic code generation, the task of generating new code snippets from existing code or comments, has long been of interest. Numerous code generation models have been proposed and proven on different benchmark datasets. However, little is known about whether this objective has been achieved and why code generation models effectively transform code sequences automatically. In other words, can we totally trust these automated code generation models? Consequently, there is a pressing need to understand the inner logic of code generation models and to investigate their replicability, reliability, and explainability. To bridge these research gaps, we conduct a thorough empirical study of five code generation models on four representative code generation datasets to assess the limits and capabilities of automatic code generation approaches. We further employ advanced explainable AI approaches to highlight the tokens that significantly contribute to the code generation. Experiments demonstrate that we successfully replicate state-of-the-art code generation approaches. We discover that state-of-the-art approaches suffer from severe data duplication and input insensitivity, which are subtle issues with significant implications. Our explainability analysis reveals that, in various experimental scenarios, code generation models can recognize code grammar and structural information, but can not capture key tokens that need to be updated. Our results draw several lessons and guidelines for future work in this area.CCS Concepts: • Software and its engineering → Software maintenance tools; • General and reference → Reliability; • Computing methodologies → Natural language processing.

show abstract

“…3) Fine-tuning: We fine-tune the best pre-trained model (T5 NL+DF ) with the best learning rate strategy (ST-LR) on D FT-train . We use early stopping to avoid overfitting [29], [38]: We save a checkpoint every 10k steps and compute the BLEU-4 score on the evaluation set every 100k steps. When the 100k steps do not lead to an improvement, we stop the training procedure, and we keep the last model.…”

Section: Training T5 For Generating Dockerfilesmentioning

confidence: 99%

Automatically Generating Dockerfiles via Deep Learning: Challenges and Promises

Rosa¹,

Mastropaolo²,

Scalabrino³

et al. 2023

Preprint

View full text Add to dashboard Cite

Containerization allows developers to define the execution environment in which their software needs to be installed. Docker is the leading platform in this field, and developers that use it are required to write a Dockerfile for their software. Writing Dockerfiles is far from trivial, especially when the system has unusual requirements for its execution environment. Despite several tools exist to support developers in writing Dockerfiles, none of them is able to generate entire Dockerfiles from scratch given a high-level specification of the requirements of the execution environment. In this paper, we present a study in which we aim at understanding to what extent Deep Learning (DL), which has been proven successful for other coding tasks, can be used for this specific coding task. We preliminarily defined a structured natural language specification for Dockerfile requirements and a methodology that we use to automatically infer the requirements from the largest dataset of Dockerfiles currently available. We used the obtained dataset, with 670,982 instances, to train and test a Text-to-Text Transfer Transformer (T5) model, following the current state-of-the-art procedure for coding tasks, to automatically generate Dockerfiles from the structured specifications. The results of our evaluation show that T5 performs similarly to the more trivial IR-based baselines we considered. We also report the open challenges associated with the application of deep learning in the context of Docker file generation.Index Terms-docker, deep learning 1 FROM tomcat:7.0.75-jre8 2 3 RUN echo deb http://archive.ubuntu.com/ubuntu precise universe multiverse >> /etc/apt/sources.list; apt-get update && \ 4 apt-get -y --fix-missing install autoconf automake build-essential \ 5 git mercurial cmake libass-dev libgpac-dev libtheora-dev libtool \ 6 libvdpau-dev libvorbis-dev pkg-config texi2html zlib1g-dev \ 7 libmp3lame-dev wget yasm && \ 8 apt-get clean 9 10 WORKDIR /usr/local/src 11 # Install x265 12 RUN hg clone https:

show abstract

Using pre-trained models to boost code review automation

Cited by 64 publications

References 27 publications

Automating Code-Related Tasks Through Transformers: The Impact of Pre-training

Automating Code-Related Tasks Through Transformers: The Impact of Pre-training

On the Reliability and Explainability of Automated Code Generation Approaches

Automatically Generating Dockerfiles via Deep Learning: Challenges and Promises

Contact Info

Product

Resources

About