ComFormer: Code Comment Generation via Transformer and Fusion Method-based Hybrid Code Representation

Recently, machine learning (ML) methods have been used to create powerful language models for a broad range of natural language processing tasks. An important subset of this field is that of generating code of programming languages for automatic software development. This review provides a broad and detailed overview of studies for code generation using ML. We selected 37 publications indexed in arXiv and IEEE Xplore databases that train ML models on programming language data to generate code. The three paradigms of code generation we identified in these studies are description-to-code, code-todescription, and code-to-code. The most popular applications that work in these paradigms were found to be code generation from natural language descriptions, documentation generation, and automatic program repair, respectively. The most frequently used ML models in these studies include recurrent neural networks, transformers, and convolutional neural networks. Other neural network architectures, as well as non-neural techniques, were also observed. In this review, we have summarized the applications, models, datasets, results, limitations, and future work of 37 publications. Additionally, we include discussions on topics general to the literature reviewed. This includes comparing different model types, comparing tokenizers, the volume and quality of data used, and methods for evaluating synthesized code. Furthermore, we provide three suggestions for future work for code generation using ML.

show abstract

“…Code-todescription Documentation generation [21], [34], [44], [66], [69], [83], [4], [89], [95] Code-to-code APR [31], [38], [43], [71], [85],…”

Section: Description-tocodementioning

confidence: 99%

“…RNN LSTM [21], [24], [34], [43], [44], [47], [50], [66], [67], [73], [97], [111] GRU [31], [69] Other [18], [114] Transformer [2] [47], [75], [76], [79], [83], [85], [4], [89], [91], [95], [102], [114] CNN [24], [29], [38], [66], [69], [89], [114] Other neural MLP [59], [63] Graph NN [69] Neural trees…”

Section: Model Type Sub-type Studiesmentioning

confidence: 99%

Code Generation Using Machine Learning: A Systematic Review

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Finally, they used beam search to improve comment quality. Yang et al [27] proposed a novel method ComFormer based on Transformer and fusion methodbased hybrid code presentation LeClair et al [28] considered the graph neural network, based on the graph2seq model [29]. Ahmad et al [30] considered the Vanilla-Transformer architecture, and used a relative position representation and copy mechanism.…”

Section: B Deep Learning-based Text Summarization and Source Code Sum...mentioning

confidence: 99%

SOTitle: A Transformer-based Post Title Generation Approach for Stack Overflow

Liu¹,

Yang²,

Chen³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

On Stack Overflow, developers can not only browse question posts to solve their programming problems but also gain expertise from the question posts to help improve their programming skills. Therefore, improving the quality of question posts in Stack Overflow has attracted the wide attention of researchers. A concise and precise title can play an important role in helping developers understand the key information of the question post, which can improve the post quality. However, the quality of the generated title is not high due to the lack of professional knowledge related to their questions or the poor presentation ability of developers. A previous study aimed to automatically generate the title by analyzing the code snippets in the question post. However, this study ignored the useful information in the corresponding problem description. Therefore, we propose an approach SOTitle for automatic post title generation by leveraging the code snippets and the problem description in the question post (i.e., the multi-modal input). SOTitle follows the Transformer structure, which can effectively capture long-term dependencies through a multi-head attention mechanism. To verify the effectiveness of SOTitle, we construct a large-scale high-quality corpus from Stack Overflow, which includes 1,168,257 high-quality question posts for four popular programming languages. Experimental results show that SOTitle can significantly outperform six state-of-the-art baselines in both automatic evaluation and human evaluation. To encourage follow-up studies, we make our corpus and approach publicly available.

show abstract

“…For example, the identifiers satisfying the Camel casing naming convention are split into multiple words to alleviate the OOV problem. Yang et al [39] proposed a novel method ComFormer based on Transformer and fusion method-based hybrid code presentation. Recently, Kang et al [40] investigated whether using the pre-trained word embedding could improve the model performance.…”

Section: B Code Generation and Code Summarizationmentioning

confidence: 99%

“…Hu et al [58] added the pointer network [59] to the Transformer to solve this problem. Yang et al [39] alleviated the OOV problem with the help of subword tokenize method (i.e., Byte BPE [60]). However, due to the nature of shellcode statements, it is more reliable to fix this type of error based on rules.…”

Section: Rule-based Repair Componentmentioning

confidence: 99%