Using Pre-Trained Models to Boost Code Review Automation

Abstract

Code review is a practice widely adopted in open source and industrial projects. Given the non-negligible cost of such a process, researchers started investigating the possibility of automating specific code review tasks. We recently proposed Deep Learning (DL) models targeting the automation of two tasks: the first model takes as input a code submitted for review and implements in it changes likely to be recommended by a reviewer; the second takes as input the submitted code and a reviewer comment posted in natural language and automatically implements the change required by the reviewer. While the preliminary results we achieved are encouraging, both models had been tested in rather simple code review scenarios, substantially simplifying the targeted problem. This was also due to the choices we made when designing both the technique and the experiments. In this paper, we build on top of that work by demonstrating that a pre-trained Text-To-Text Transfer Transformer (T5) model can outperform previous DL models for automating code review tasks. Also, we conducted our experiments on a larger and more realistic (and challenging) dataset of code review activities.

CodeReviewer: Pre-Training for Automating Code Review Activities

“…To tackle the problems, we pre-train CodeReviewer, an encoderdecoder transformer model. Different from Tufano et al [40]'s work, CodeReviewer is pre-trained on a large dataset in code review scenario, consisting of code diff hunks and code review comments. We propose four pre-training tasks, including diff tag prediction, denoising code diff, denoising review comment, and review comment generation to make CodeReviewer better understand code diffs and generate review comments.…”

“…The input is still a code change, i.e., X = {𝐷 (𝐶 0 , 𝐶 1 )}, with its context. In some previous works [18,40,41], researchers use the changed code as input but not the code diff, without taking into account that review comments have to focus on the changed part. It's not recommended for reviewers to give suggestions to the code context which has not been revised.…”

“…To demonstrate the superiority of our multilingual code review related pre-training dataset and carefully designed pre-training tasks, we compare our CodeReviewer model with three baselines, including a state-of-the-art (SOTA) model architecture Transformer [42] trained from scratch and two pre-trained models: T5 for code review [40] and CodeT5 [43].…”

“…Many researchers have explored ways to assist reviewers and committers (i.e., code authors) to reduce their workload in the code review process, such as recommending the best reviewer [9,37], recommending or generating the possible review comments [18,40,41] and even revising the code before submitting it for review [40]. This paper shares the same goal to automate some specific tasks related to code review.…”

“…We demonstrate that it cannot generate any meaningful comment in the review generation task based on our evaluation. Tufano et al [40] attempt to use a pre-trained model for code review automation. However, their pre-training dataset is collected from Stack Overflow and Code-SearchNet [21], which is not directly related to code review process.…”