CodeReviewer: Pre-Training for Automating Code Review Activities

Abstract

Code review is an essential part to software development lifecycle since it aims at guaranteeing the quality of codes. Modern code review activities necessitate developers viewing, understanding and even running the programs to assess logic, functionality, latency, style and other factors. It turns out that developers have to spend far too much time reviewing the code of their peers. Accordingly, it is in significant demand to automate the code review process. In this research, we focus on utilizing pre-training techniques for the tasks in the code review scenario. We collect a large-scale dataset of real world code changes and code reviews from open-source projects in nine of the most popular programming languages. To better understand code diffs and reviews, we propose CodeReviewer, a pre-trained model that utilizes four pre-training tasks tailored specifically for the code review senario. To evaluate our model, we focus on three key tasks related to code review activities, including code change quality estimation, review comment generation and code refinement. Furthermore, we establish a high-quality benchmark dataset based on our collected data for these three tasks and conduct comprehensive experiments on it. The experimental results demonstrate that our model outperforms the previous state-of-theart pre-training approaches in all tasks. Further analysis show that our proposed pre-training tasks and the multilingual pre-training dataset benefit the model on the understanding of code changes and reviews.

On the Reliability and Explainability of Automated Code Generation Approaches

“…Numerous NMT techniques (especially Transformer-based models) have been increasingly adopted for automated code-based tasks. In addition, NMT models like T5 [37] have demonstrated the ability to effectively learn code representation from unlabeled data to conduct a wide range of downstream tasks given supervised discriminative fine-tuning on specific tasks (e.g., code completion [25,46], code search [17,40], code summarization [46], code review [24,41,42], and API recommendation [9]).…”

“…These existing approaches can be categorized into three types: encoder-based models such as CodeBERT [15], decoder-based models such as CodeGPT [26], and encoder-decoder-based models like CodeT5 [50]. Since prior works [7,24,53] have proven that encoder-based models and decoder-based models are not good at generation tasks, we only consider encoder-decoder-based code generation models in this study. There are many encoder-decoder-based models proposed for code generation tasks.…”

“…There are many encoder-decoder-based models proposed for code generation tasks. While there can be many potential state-of-the-art models to be studied, we chose five representative pre-trained code generation models (i.e., T5 [37], CodeT5 [50], CoTexT [35], CodeTrans [14], and CodeReviewer [24]) that are publicly available.…”

“…In response to the observations and concerns raised above, we conduct the first comprehensive empirical study on the existing code generation models for automated code generation tasks. Specifically, we adopt five mainstream encoder-decoder-based NMT models, i.e., T5 [37], CodeT5 [50], CoTexT [35], CodeTrans [14], and CodeReviewer [24] as our studied objects. We then evaluated the studied code generation models on four state-of-the-art benchmark datasets (i.e., ICSE'19 [42], TOSEM'19 [43], FSE'22 [24], NIPS'21 [26]).…”

“…Specifically, we adopt five mainstream encoder-decoder-based NMT models, i.e., T5 [37], CodeT5 [50], CoTexT [35], CodeTrans [14], and CodeReviewer [24] as our studied objects. We then evaluated the studied code generation models on four state-of-the-art benchmark datasets (i.e., ICSE'19 [42], TOSEM'19 [43], FSE'22 [24], NIPS'21 [26]).…”