Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks

Abstract

Deep learning (DL) techniques are gaining more and more attention in the software engineering community. They have been used to support several code-related tasks, such as automatic bug fixing and code comments generation. Recent studies in the Natural Language Processing (NLP) field have shown that the Text-To-Text Transfer Transformer (T5) architecture can achieve state-of-the-art performance for a variety of NLP tasks. The basic idea behind T5 is to first pre-train a model on a large and generic dataset using a self-supervised task (e.g., filling masked words in sentences). Once the model is pre-trained, it is fine-tuned on smaller and specialized datasets, each one related to a specific task (e.g., language translation, sentence classification). In this paper, we empirically investigate how the T5 model performs when pre-trained and fine-tuned to support code-related tasks. We pre-train a T5 model on a dataset composed of natural language English text and source code. Then, we fine-tune such a model by reusing datasets used in four previous works that used DL techniques to: (i) fix bugs, (ii) inject code mutants, (iii) generate assert statements, and (iv) generate code comments. We compared the performance of this single model with the results reported in the four original papers proposing DL-based solutions for those four tasks. We show that our T5 model, exploiting additional data for the self-supervised pre-training phase, can achieve performance improvements over the four baselines.

CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation

“…CuBERT employs BERT's powerful masked language modeling objective to derive generic code-specific representation, and CodeBERT further adds a replaced token detection (Clark et al, 2020) encoder-decoder models based on T5 for programming language pre-training and support a more comprehensive set of tasks. Some emerging work (Clement et al, 2020;Mastropaolo et al, 2021;Elnaggar et al, 2021) in the recent literature also explore the T5 framework on code, but they only focus on a limited subset of generation tasks and do not support understanding tasks like us. Apart from these, PLBART (Ahmad et al, 2021) based on another encoder-decoder model BART can also support both understanding and generation tasks.…”

Pop Quiz! Can a Large Language Model Help With Reverse Engineering?

“…For instance, there have been numerous investigations into techniques for code summarization, from early approaches using text retrieval [11] through to more recent ML-based approaches that are often framed as a Neural Machine Translation (NMT) problem (e.g., see recent works by Mastropaolo et al [17] and Haque et al [12]). Generally, deep learning approaches for code summarization try to infer information that is not explicitly in the code, being trained on code/comment pairs (such as LeClair and McMillan's dataset [14]) and evaluated on regular (i.e., not decompiled) source code.…”

A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspective

“…Other pretrained transformers used on source code include CodeT5 (Wang et al, 2021b). Code-Trans (Elnaggar et al, 2021), PyMT5 (Clement et al, 2020), CuBERT (Kanade et al, 2020), PLBART , ProphetNet-X (Qi et al, 2021), CoTexT (Phan et al, 2021), T5-Code (Mastropaolo et al, 2021), GraphCode-BERT , and AlphaCode (Li et al, 2022). Pretrained GPT-style Models for source code generation include CodeGPT , and GPT-Codex (Chen et al, 2021a).…”