An Empirical Study on Code Comment Completion

Mastropaolo, Antonio; Aghajani, Emad; Pascarella, Luca; Bavota, Gabriele

doi:10.1109/icsme52107.2021.00021

Cited by 15 publications

(8 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Currently, the efforts regard studying the capabilities of Copilot for generating (and reproducing) correct and efficient solutions for fundamental algorithmic problems and comparing Copilot's proposed solutions with those of human programmers on a set of programming tasks [19]. Having the same goal of evaluating GitHub Copilot, an empirical study aims to understand whether different but semantically equivalent natural language descriptions obtain in the same recommended function [20].…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Voting for Detecting the Algorithmic Solving Strategy in Competitive Programming Solutions

Stoica

Băbiceanu

Rebedea

et al. 2023

Preprint

View full text Add to dashboard Cite

The problem of source code analysis using machine learning techniques has been gaining much attention recently as several powerful code embeddings methods have been created. Having available different embedding methods for source code opened the way for tackling many practical problems related to source code analysis. This paper addresses the issue of determining the number of distinct algorithmic strategies that may be found in a set of correct solutions for a competitive programming problem. We have investigated using five embedding methods with three algorithmic strategies in a data analysis pipeline that tackle the previously described issues on a newly created dataset consisting of 15 algorithmic problems. We propose an unsupervised algorithm that considers each embedding as a different dataset view. On each view, a hard clustering algorithm can be employed to split the correct solutions into K distinct algorithmic approaches, where K is a hyperparameter. In the ideal case, if all the algorithmic approaches in each view are determined correctly, a mapping could be done between each consecutive view such that each algorithmic approach from one view maps to a single corresponding algorithmic approach in the other. We are interested in determining the maximum subset of solutions that satisfy the assumption from above and use this subset as a base for a co-training procedure where an estimator employed on each view votes for the algorithmic approach of each solution which is not part of the subset. According to the results, the proposed unsupervised voting algorithm highly improves the baseline clustering approach. This improvement was observed across all problems in the dataset except one. Scale-up of the data analysis pipeline to datasets of thousands of problems may yield the ability to profoundly understand and learn about the innovative process of correctly designing and writing code in the context of competitive programming or even industry code.

show abstract

Section: Related Workmentioning

confidence: 99%

Unsupervised Voting for Detecting the Algorithmic Solving Strategy in Competitive Programming Solutions

Stoica

Băbiceanu

Rebedea

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…A T5 model is trained in two phases: (i) pre-training, in which the model is trained with a self-supervised objective that allows defining a shared knowledge-base useful for a large class of tasks, and fine-tuning, which specializes the model on a downstream task (e.g., language translation). T5 already showed its effectiveness in code-related tasks [13], [18], [28]- [32]. However, its application to the generation of Dockerfiles is novel and still unexplored.…”

Section: Training T5 For Generating Dockerfilesmentioning

confidence: 99%

“…However, its application to the generation of Dockerfiles is novel and still unexplored. As done in previous work [13], [18], we use the smallest T5 version available (T5 small), which is composed of 60M parameters.…”

Section: Training T5 For Generating Dockerfilesmentioning

confidence: 99%

“…This might be due to two main issues. First, a larger training dataset might be needed for this task: Despite we consider the largest collection of Dockerfiles in the literature [16], our results suggest that the T5 learning could benefit from more training data, considering the number of used instances to previous studies working on source code [18]. Second, a different T5 training stop criterion needs to be defined:…”

Section: Introductionmentioning

confidence: 98%

See 1 more Smart Citation

Automatically Generating Dockerfiles via Deep Learning: Challenges and Promises

Rosa¹,

Mastropaolo²,

Scalabrino³

et al. 2023

Preprint

View full text Add to dashboard Cite

Containerization allows developers to define the execution environment in which their software needs to be installed. Docker is the leading platform in this field, and developers that use it are required to write a Dockerfile for their software. Writing Dockerfiles is far from trivial, especially when the system has unusual requirements for its execution environment. Despite several tools exist to support developers in writing Dockerfiles, none of them is able to generate entire Dockerfiles from scratch given a high-level specification of the requirements of the execution environment. In this paper, we present a study in which we aim at understanding to what extent Deep Learning (DL), which has been proven successful for other coding tasks, can be used for this specific coding task. We preliminarily defined a structured natural language specification for Dockerfile requirements and a methodology that we use to automatically infer the requirements from the largest dataset of Dockerfiles currently available. We used the obtained dataset, with 670,982 instances, to train and test a Text-to-Text Transfer Transformer (T5) model, following the current state-of-the-art procedure for coding tasks, to automatically generate Dockerfiles from the structured specifications. The results of our evaluation show that T5 performs similarly to the more trivial IR-based baselines we considered. We also report the open challenges associated with the application of deep learning in the context of Docker file generation.Index Terms-docker, deep learning 1 FROM tomcat:7.0.75-jre8 2 3 RUN echo deb http://archive.ubuntu.com/ubuntu precise universe multiverse >> /etc/apt/sources.list; apt-get update && \ 4 apt-get -y --fix-missing install autoconf automake build-essential \ 5 git mercurial cmake libass-dev libgpac-dev libtheora-dev libtool \ 6 libvdpau-dev libvorbis-dev pkg-config texi2html zlib1g-dev \ 7 libmp3lame-dev wget yasm && \ 8 apt-get clean 9 10 WORKDIR /usr/local/src 11 # Install x265 12 RUN hg clone https:

show abstract

“…In our recent work [44] we empirically investigated the potential of a T5 model when pre-trained and fine-tuned to support four code-related tasks also characterized by text-totext transformations. In particular, we started by pre-training a T5 model using a large dataset consisting of 499,618 English sentences and 1,569,889 source code components (i.e., Java methods).…”

Section: Introductionmentioning

confidence: 99%

Using Transfer Learning for Code-Related Tasks

Mastropaolo¹,

Cooper²,

Palacio³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Deep learning (DL) techniques have been used to support several code-related tasks such as code summarization and bug-fixing. In particular, pre-trained transformer models are on the rise, also thanks to the excellent results they achieved in Natural Language Processing (NLP) tasks. The basic idea behind these models is to first pre-train them on a generic dataset using a self-supervised task (e.g., filling masked words in sentences). Then, these models are fine-tuned to support specific tasks of interest (e.g., language translation). A single model can be fine-tuned to support multiple tasks, possibly exploiting the benefits of transfer learning. This means that knowledge acquired to solve a specific task (e.g., language translation) can be useful to boost performance on another task (e.g., sentiment classification). While the benefits of transfer learning have been widely studied in NLP, limited empirical evidence is available when it comes to code-related tasks. In this paper, we assess the performance of the Text-To-Text Transfer Transformer (T5) model in supporting four different code-related tasks: (i) automatic bug-fixing, (ii) injection of code mutants, (iii) generation of assert statements, and (iv) code summarization. We pay particular attention in studying the role played by pre-training and multi-task fine-tuning on the model's performance. We show that (i) the T5 can achieve better performance as compared to state-of-the-art baselines; and (ii) while pre-training helps the model, not all tasks benefit from a multi-task fine-tuning.

show abstract

An Empirical Study on Code Comment Completion

Cited by 15 publications

References 66 publications

Unsupervised Voting for Detecting the Algorithmic Solving Strategy in Competitive Programming Solutions

Unsupervised Voting for Detecting the Algorithmic Solving Strategy in Competitive Programming Solutions

Automatically Generating Dockerfiles via Deep Learning: Challenges and Promises

Using Transfer Learning for Code-Related Tasks

Contact Info

Product

Resources

About