Assessing the Students' Understanding and their Mistakes in Code Review Checklists: An Experience Report of 1,791 Code Review Checklist Questions from 394 Students

Abstract

Code review is a widely-used practice in software development companies to identify defects. Hence, code review has been included in many software engineering curricula at universities worldwide. However, teaching code review is still a challenging task because the code review effectiveness depends on the code reading and analytical skills of a reviewer. While several studies have investigated the code reading techniques that students should use to find defects during code review, little has focused on a learning activity that involves analytical skills. Indeed, developing a code review checklist should stimulate students to develop their analytical skills to anticipate potential issues (i.e., software defects). Yet, it is unclear whether students can anticipate potential issues given their limited experience in software development (programming, testing, etc.). We perform a qualitative analysis to investigate whether students are capable of creating code review checklists, and if the checklists can be used to guide reviewers to find defects. In addition, we identify common mistakes that students make when developing a code review checklist. Our results show that while there are some misconceptions among students about the purpose of code review, students are able to anticipate potential defects and create a relatively good code review checklist. Hence, our results lead us to conclude that developing a code review checklist can be a part of the learning activities for code review in order to scaffold students' skills.

Data and Material for 'Less is More: Supporting Developers in Vulnerability Detection during Code Review'

“…Rong et al [68] conducted a semi-controlled experiment with students and found evidence that checklists can help in guiding them during code reviews. In the context of education, Chong et al [20] found that students are able to anticipate potential defects and create a relatively good code review checklist, which can be used to find defects. Finally, Gonçalves et al [38] registered an experiment report with the goal of investigating whether review checklists and guidance improve code review performance.…”

SQAPlanner: Generating Data-Informed Software Quality Improvement Plans

“…Respondents described that some of the SQA planning activities in their organisations involve human heuristics in decision-making. For example, they used documentation and review checklists [7] (e.g., R34: "Lessons learnt from projects are documented and common mistakes are included in review checklists to ensure that they are not repeated. "), and team meetings (e.g., R10: "team meetings, brainstorm, and in house system", and R48: "... through step by step manual processes working together in a core team").…”

“…Then, we integrated our SQAPlanner approach and replaced their visualization (bar plots) with our visualization generated by SQAPlanner using the implementation of bullet plots as provided by the d3.js Javascript library. 7 To investigate the practitioners' perceptions of our SQAPlanner visualization, we used a qualitative survey as a research method. We also used the visualization of Microsoft's Code Defect AI (see Figure 2) as a baseline comparison.…”

Assessing the Students' Understanding and their Mistakes in Code Review Checklists -- An Experience Report of 1,791 Code Review Checklist Questions from 394 Students

“…Noted that at this stage, the students only have access to the requirements specification, without the actual artifact that needs to be reviewed. The requirements specification and relevant software artifacts are available online [5]. We do not set a limit on the number of checklist questions that the students can prepare.…”