Programmer-based fault prediction

Abstract

Background: Previous research has provided evidence that a combination of static code metrics and software history metrics can be used to predict with surprising success which files in the next release of a large system will have the largest numbers of defects. In contrast, very little research exists to indicate whether information about individual developers can profitably be used to improve predictions.Aims: We investigate whether files in a large system that are modified by an individual developer consistently contain either more or fewer faults than the average of all files in the system. The goal of the investigation is to determine whether information about which particular developer modified a file is able to improve defect predictions. We also continue an earlier study to evaluate the use of counts of the number of developers who modified a file as predictors of the file's future faultiness.Method: We analyzed change reports filed by 107 programmers for 16 releases of a system with 1,400,000 LOC and 3100 files. A "bug ratio" was defined for programmers, measuring the proportion of faulty files in release R out of all files modified by the programmer in release R-1. The study compares the bug ratios of individual programmers to the average bug ratio, and also assesses the consistency of the bug ratio across releases for individual programmers.Results: Bug ratios varied widely among all the programmers, as well as for many individual programmers across all the releases that they participated in. We found a statistically significant correlation between the bug ratios for programmers for the first half of changed files versus the ratios for the second half, indicating a measurable degree of persistence in the bug ratio. However, when the computation was repeated with the bug ratio controlled not only by release, but also by file size, the correlation disappeared. In addition to the bug ratios, we confirmed that counts of the cumulative number of different developers changing a file over its lifetime can help to improve predictions, while other developer counts are not helpful. Conclusions:The results from this preliminary study indicate that adding information to a model about which particular developer modified a file is not likely to improve defect predictions. The study is limited to a single large system, and its results may not hold more widely. The bug ratio is only one way of measuring the "fault-proneness" of an individual programmer's coding, and we intend to investigate other ways of evaluating bug introduction by individuals.

Personalized defect prediction

“…Different developers have different experience levels, different coding styles, and different commit patterns, resulting in different buggy change patterns [11]. Figure 2 shows differences in buggy change patterns of four prolific Linux kernel developers in the mainline repository from 2005 to 2010.…”

“…Recently, Shivaji et al [48] improve the work above by applying feature selection algorithms. Ostrand et al [11] use negative binomial regression as the classifier, and various metadata and developer-specific metrics as features to predict defects. Rahman et al [16] compare the effect of code metrics and process metrics.…”

“…They combine all developers' changes to build a single prediction model. Different developers have different coding styles, commit frequencies, and experience levels, all of which cause different defect patterns [11]. For example, based on our inspection of the Linux kernel's mainline repository from 2005 to 2010, 48.0% of one developer's changes related to for loops are buggy while the percentage is only 13.3% for another developer.…”

Cost Effectiveness of Software Defect Prediction in an Industrial Project

“…Two other publications of Khoshgoftaar and Seliya from 2004 [15] and 2005 [16] continued with the previous concept and focused on commercial data analysis, but were not applied to a real-world environment. A similar approach can be found in publications by Ostrand and Weyuker [29], Ostrand et al [31], Tosun et al [45], Turhan et al [47,48]. Examples of industrial applications of information gathered by using defect prediction can be found in publications by Wong et al [50], Succi et al [41] and Kläs et al [17].…”

Influence of confirmation biases of developers on software quality: an empirical study

“…detected no significant contribution of this kind of developer information to the defect prediction performance. Following this research, the authors later analyzed the effectiveness of individual developer performance on the defect prediction performance and found no evidence of a significant improvement in the defect prediction performance either [39].…”