Fret: Functional Reinforced Transformer With BERT for Code Summarization

Wang, Ruyun; Zhang, Hanwen; Lu, Guoliang; Lyu, Lei; Lyu, Chen

doi:10.1109/access.2020.3011744

Cited by 40 publications

(34 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For each API review aspect, we evaluate the performance in terms of five evaluation metrics (i.e., P, R, F 1, M CC, and AU C) as introduced in Section III-E. RQ1 Can pre-trained transformer-based models achieve better performance than the state-of-the-art approach which is based on traditional machine learning models? Motivation Previous studies have shown the great potential of pre-trained transformer-based models on many software engineering tasks, e.g., sentiment analysis for software data [9] and code summarization [12]. However, the efficacy of the pre-trained transformer-based models for various types of For the summative result in Table IV, we calculate the arithmetic average of the used evaluation metrics of each approach across all the aspects as avg.…”

Section: Discussionmentioning

confidence: 99%

“…In recent years, pre-trained transformer-based models have achieved exceptional performance in many tasks and areas including the software engineering domain [9]- [12]. For example, Zhang et al conduct an empirical study on benchmarking four pre-trained transformer-based models (BERT [13], RoBERTa [14], ALBERT [15], and XLNet [16]) for sentiment analysis on six software repositories (e.g., code reviews) [9].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Aspect-Based API Review Classification: How Far Can Pre-Trained Transformer Model Go?

Yang¹,

Xu²,

Khan³

et al. 2022

Preprint

View full text Add to dashboard Cite

APIs (Application Programming Interfaces) are reusable software libraries and are building blocks for modern rapid software development. Previous research shows that programmers frequently share and search for reviews of APIs on the mainstream software question and answer (Q&A) platforms like Stack Overflow, which motivates researchers to design tasks and approaches related to process API reviews automatically. Among these tasks, classifying API reviews into different aspects (e.g., performance or security), which is called the aspect-based API review classification, is of great importance. The current state-of-the-art (SOTA) solution to this task is based on the traditional machine learning algorithm. Inspired by the great success achieved by pre-trained models on many software engineering tasks, this study fine-tunes six pre-trained models for the aspect-based API review classification task and compares them with the current SOTA solution on an API review benchmark collected by Uddin et al. The investigated models include four models (BERT, RoBERTa, ALBERT and XLNet) that are pretrained on natural languages, BERTOverflow that is pre-trained on text corpus extracted from posts on Stack Overflow, and CosSensBERT that is designed for handling imbalanced data. The results show that all the six fine-tuned models outperform the traditional machine learning-based tool. More specifically, the improvement on the F1-score ranges from 21.0% to 30.2%. We also find that BERTOverflow, a model pre-trained on the corpus from Stack Overflow, does not show better performance than BERT. The result also suggests that CosSensBERT also does not exhibit better performance than BERT in terms of F1, but it is still worthy of being considered as it achieves better performance on MCC and AUC.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Aspect-Based API Review Classification: How Far Can Pre-Trained Transformer Model Go?

Yang¹,

Xu²,

Khan³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The machine translation was significantly enhanced because they used the transformer architecture. Ahmad et al [32] and Wang et al [33] also used transformer-based deep learning models to complete translation tasks, and relative to the findings of previous studies, they improved the accuracy of the generated results. By contrast, Dai et al [34] discovered that when a transformer-based deep learning model processes long text, the text is truncated into multiple fixed-length fields.…”

Section: Quantum Machine Learningmentioning

confidence: 94%

Android-SEM: Generative Adversarial Network for Android Malware Semantic Enhancement Model Based on Transfer Learning

Huang¹,

Li²,

Qiao³

et al. 2022

Electronics

View full text Add to dashboard Cite

Currently, among the millions of Android applications, there exist numerous malicious programs that pose significant threats to people’s security and privacy. Therefore, it is imperative to develop approaches for detecting Android malware. Recently developed malware detection methods usually rely on various features, such as application programming interface (API) sequences, images, and permissions, thereby ignoring the importance of source code and the associated comments, which are not usually included in malware. Therefore, we propose Android-SEM, which is an Android source code semantic enhancement model based on transfer learning. Our proposed model is built upon the Transformer architecture to achieve a pretraining framework for generating code comments from malware source code. The performance of the pretraining framework is optimized using a generative adversarial network. Our proposed model relies on a novel regression model-based filter to retain high-quality comments and source code for feature fusion pertinent to semantic enhancement. Creatively, and contrary to conventional methods, we incorporated a quantum support vector machine (QSVM) for classifying malicious Android code by combining quantum machine learning and classical deep learning models. The results proved that Android-SEM achieves accuracy levels of 99.55% and 99.01% for malware detection and malware categorization, respectively.

show abstract

“…Feng et al [31] proposed the bi-modal pre-training model CodeBERT based on Transformer neural architecture for programming language and natural language. Wang et al [32] proposed a BERT-based functional enhanced transformer model, they proposed a new enhancer to generate higher-quality code summarization.…”

Section: B Deep Learning-based Text Summarization and Source Code Sum...mentioning

confidence: 99%

SOTitle: A Transformer-based Post Title Generation Approach for Stack Overflow

Liu¹,

Yang²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

On Stack Overflow, developers can not only browse question posts to solve their programming problems but also gain expertise from the question posts to help improve their programming skills. Therefore, improving the quality of question posts in Stack Overflow has attracted the wide attention of researchers. A concise and precise title can play an important role in helping developers understand the key information of the question post, which can improve the post quality. However, the quality of the generated title is not high due to the lack of professional knowledge related to their questions or the poor presentation ability of developers. A previous study aimed to automatically generate the title by analyzing the code snippets in the question post. However, this study ignored the useful information in the corresponding problem description. Therefore, we propose an approach SOTitle for automatic post title generation by leveraging the code snippets and the problem description in the question post (i.e., the multi-modal input). SOTitle follows the Transformer structure, which can effectively capture long-term dependencies through a multi-head attention mechanism. To verify the effectiveness of SOTitle, we construct a large-scale high-quality corpus from Stack Overflow, which includes 1,168,257 high-quality question posts for four popular programming languages. Experimental results show that SOTitle can significantly outperform six state-of-the-art baselines in both automatic evaluation and human evaluation. To encourage follow-up studies, we make our corpus and approach publicly available.

show abstract

Fret: Functional Reinforced Transformer With BERT for Code Summarization

Cited by 40 publications

References 38 publications

Aspect-Based API Review Classification: How Far Can Pre-Trained Transformer Model Go?

Aspect-Based API Review Classification: How Far Can Pre-Trained Transformer Model Go?

Android-SEM: Generative Adversarial Network for Android Malware Semantic Enhancement Model Based on Transfer Learning

SOTitle: A Transformer-based Post Title Generation Approach for Stack Overflow

Contact Info

Product

Resources

About