Comparing static bug finders and statistical prediction

Rahman, Foyzur; Khatri, Sameer; Barr, Earl T.; Dévanbu, Prémkumar

doi:10.1145/2568225.2568269

Cited by 103 publications

(94 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prior research has assumed that 5% (sometimes 20%) of the code could realistically be inspected under deadline [44]. Additionally, Rahman et al compare SBF with DP (a file level statistical defect predictor) by allowing the number warnings from SBF to set the inspection budget (denoted AUCECL) [43]. They assign the DP the same budget and compare the resulting AUCEC scores.…”

Section: Evaluating Defect Predictionsmentioning

confidence: 99%

“…A prediction model is awarded credit, ranging from 0 to 1, for each (ipso facto eventually) buggy line flagged as suspicious. Previous work by Rahman et al has compared SBF and DP models using two types of credit: full (or optimistic) and partial (or scaled) credit [43], which we adapt to line level defect prediction. The former metric awards a model one credit point for each bug iff at least one line of the bug was marked buggy by the model.…”

Section: Evaluating Defect Predictionsmentioning

confidence: 99%

See 1 more Smart Citation

On the "naturalness" of buggy code

Ray

Hellendoorn

Godhane

et al. 2016

Proceedings of the 38th International Conference on Software Engineering

Self Cite

185

138

View full text Add to dashboard Cite

Real software, the kind working programmers produce by the kLOC to solve real-world problems, tends to be "natural", like speech or natural language; it tends to be highly repetitive and predictable. Researchers have captured this naturalness of software through statistical models and used them to good effect in suggestion engines, porting tools, coding standards checkers, and idiom miners. This suggests that code that appears improbable, or surprising, to a good statistical language model is "unnatural" in some sense, and thus possibly suspicious. In this paper, we investigate this hypothesis. We consider a large corpus of bug fix commits (ca. 7,139), from 10 different Java projects, and focus on its language statistics, evaluating the naturalness of buggy code and the corresponding fixes. We find that code with bugs tends to be more entropic (i.e. unnatural), becoming less so as bugs are fixed. Ordering files for inspection by their average entropy yields cost-effectiveness scores comparable to popular defect prediction methods. At a finer granularity, focusing on highly entropic lines is similar in cost-effectiveness to some well-known static bug finders (PMD, FindBugs) and ordering warnings from these bug finders using an entropy measure improves the cost-effectiveness of inspecting code implicated in warnings. This suggests that entropy may be a valid, simple way to complement the effectiveness of PMD or FindBugs, and that search-based bug-fixing methods may benefit from using entropy both for fault-localization and searching for fixes.

show abstract

Section: Evaluating Defect Predictionsmentioning

confidence: 99%

Section: Evaluating Defect Predictionsmentioning

confidence: 99%

On the "naturalness" of buggy code

Ray

Hellendoorn

Godhane

et al. 2016

Proceedings of the 38th International Conference on Software Engineering

Self Cite

185

138

View full text Add to dashboard Cite

show abstract

“…For example, Kocaguneli et al combined several single software effort estimation models to create more powerful multi-model ensembles [21]. Also, Rahman et al used static bug-finding to improve the performance of statistical defect prediction and vice versa [38].…”

Section: Related Workmentioning

confidence: 99%

Information retrieval and spectrum based bug localization: better together

Oentaryo

2015

Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

107

View full text Add to dashboard Cite

Debugging often takes much effort and resources. To help developers debug, numerous information retrieval (IR)-based and spectrum-based bug localization techniques have been proposed. IR-based techniques process textual information in bug reports, while spectrum-based techniques process program spectra (i.e., a record of which program elements are executed for each test case). Both eventually generate a ranked list of program elements that are likely to contain the bug. However, these techniques only consider one source of information, either bug reports or program spectra, which is not optimal. To deal with the limitation of existing techniques, in this work, we propose a new multi-modal technique that considers both bug reports and program spectra to localize bugs. Our approach adaptively creates a bug-specific model to map a particular bug to its possible location, and introduces a novel idea of suspicious words that are highly associated to a bug. We evaluate our approach on 157 real bugs from four software systems, and compare it with a state-of-the-art IR-based bug localization method, a state-of-the-art spectrum-based bug localization method, and three state-of-the-art multi-modal feature location methods that are adapted for bug localization. Experiments show that our approach can outperform the baselines by at least 47.62%, 31.48%, 27.78%, and 28.80% in terms of number of bugs successfully localized when a developer inspects 1, 5, and 10 program elements (i.e., Top 1, Top 5, and Top 10), and Mean Average Precision (MAP) respectively.

show abstract

“…Rahman et.al. [34] argue, that a more efficient way of comparing quality assurance efforts, when the defect prediction models are involved, is by a comparison of AUCEC (Area Under Cost Efficiency Curve) values [1]. The approach followed in our paper was motivated by the fact that the comparison of cost values (actual and simulated) is considered to be more readable by our business…”

Section: Threats To Validitymentioning

confidence: 99%

Cost Effectiveness of Software Defect Prediction in an Industrial Project

Hryszko

Madeyski

2018

Foundations of Computing and Decision Sciences

View full text Add to dashboard Cite

Abstract. Software defect prediction is a promising approach aiming to increase software quality and, as a result, development pace. Unfortunately, the cost effectiveness of software defect prediction in industrial settings is not eagerly shared by the pioneering companies. In particular, this is the first attempt to investigate the cost effectiveness of using the DePress open source software measurement framework (jointly developed by Wroclaw University of Science and Technology, and Capgemini software development company) for defect prediction in commercial software projects. We explore whether defect prediction can positively impact an industrial software development project by generating profits. To meet this goal, we conducted a defect prediction and simulated potential quality assurance costs based on the best possible prediction results when using a default, non-tweaked DePress configuration, as well as the proposed Quality Assurance (QA) strategy. Results of our investigation are optimistic: we estimated that quality assurance costs can be reduced by almost 30% when the proposed approach will be used, while estimated DePress usage Return on Investment (ROI) is fully 73 (7300%), and Benefits Cost Ratio (BCR) is 74. Such promising results, being the outcome of the presented research, have caused the acceptance of continued usage of the DePress-based software defect prediction for actual industrial projects run by Volvo Group.

show abstract

Comparing static bug finders and statistical prediction

Cited by 103 publications

References 25 publications

On the "naturalness" of buggy code

On the "naturalness" of buggy code

Information retrieval and spectrum based bug localization: better together

Cost Effectiveness of Software Defect Prediction in an Industrial Project

Contact Info

Product

Resources

About