Automating Code-Related Tasks Through Transformers: The Impact of Pre-training

Abstract

Transformers have gained popularity in the software engineering (SE) literature. These deep learning models are usually pre-trained through a self-supervised objective, meant to provide the model with basic knowledge about a language of interest (e.g., Java). A classic pre-training objective is the masked language model (MLM), in which a percentage of tokens from the input (e.g., a Java method) is masked, with the model in charge of predicting them. Once pre-trained, the model is then finetuned to support the specific downstream task of interest (e.g., code summarization). While there is evidence suggesting the boost in performance provided by pre-training, little is known about the impact of the specific pre-training objective(s) used. Indeed, MLM is just one of the possible pre-training objectives and recent work from the natural language processing field suggest that pre-training objectives tailored for the specific downstream task of interest may substantially boost the model's performance. For example, in the case of code summarization, a tailored pretraining objective could be the identification of an appropriate name for a given method, considering the method name to generate as an extreme summary. In this study, we focus on the impact of pre-training objectives on the performance of transformers when automating code-related tasks. We start with a systematic literature review aimed at identifying the pre-training objectives used in SE. Then, we pre-train 32 transformers using both (i) generic pre-training objectives usually adopted in SE; and (ii) pre-training objectives tailored to specific code-related tasks subject of our experimentation, namely bug-fixing, code summarization, and code completion. We also compare the pretrained models with non pre-trained ones and show the advantage brought by pre-training in different scenarios, in which more or less fine-tuning data are available. Our results show that: (i) pre-training helps in boosting performance only if the amount of fine-tuning data available is small; (ii) the MLM objective is usually sufficient to maximize the prediction performance of the model, even when comparing it with pre-training objectives specialized for the downstream task at hand.

DeepCNN: A Dual Approach to Fault Localization and Repair in Convolutional Neural Networks

“…Researchers have been investigating the potential of utilizing pre-trained large language models (LLMs) to assist software engineers in various aspects of software development. This includes tasks such as code generation [40], completion [41], summarization [42], repair [43], bug fixing [16], and code review [44]. In fact, LLMs have shown remarkable capabilities, outperforming many existing state-of-the-art methods in various code-related tasks.…”

“…Transformers have revolutionized Natural Language Processing (NLP) and have found applications in various other fields [63], [64]. In software engineering, transformers can be particularly beneficial for tasks like code summarization [42], defect prediction [16], and automated code generation [40]. Their ability to capture long-range dependencies and contextual information makes them highly effective for these complex tasks.…”