Predicting Defective Lines Using a Model-Agnostic Technique

Abstract

Defect prediction models are proposed to help a team prioritize source code areas files that need Software Quality Assurance (SQA) based on the likelihood of having defects. However, developers may waste their unnecessary effort on the whole file while only a small fraction of its source code lines are defective. Indeed, we find that as little as 1%-3% of lines of a file are defective. Hence, in this work, we propose a novel framework (called LINE-DP) to identify defective lines using a model-agnostic technique, i.e., an Explainable AI technique that provides information why the model makes such a prediction. Broadly speaking, our LINE-DP first builds a file-level defect model using code token features. Then, our LINE-DP uses a state-of-the-art model-agnostic technique (i.e., LIME) to identify risky tokens, i.e., code tokens that lead the file-level defect model to predict that the file will be defective. Then, the lines that contain risky tokens are predicted as defective lines. Through a case study of 32 releases of nine Java open source systems, our evaluation results show that our LINE-DP achieves an average recall of 0.61, a false alarm rate of 0.47, a top 20%LOC recall of 0.27, and an initial false alarm of 16, which are statistically better than six baseline approaches. Our evaluation shows that our LINE-DP requires an average computation time of 10 seconds including model construction and defective identification time. In addition, we find that 63% of defective lines that can be identified by our LINE-DP are related to common defects (e.g., argument change, condition change). These results suggest that our LINE-DP can effectively identify defective lines that contain common defects while requiring a smaller amount of inspection effort and a manageable computation cost. The contribution of this paper builds an important step towards line-level defect prediction by leveraging a model-agnostic technique.

JITLine: A Simpler, Better, Faster, Finer-grained Just-In-Time Defect Prediction

“…Yan et al [43] proposed a two-phase approach-i.e., the ML model trained on software metrics (e.g., #added lines) is first used to identify which commits are the most risky, then the N-gram model trained on textual features is finally used to localise the riskiest lines. On the other hand, a recent work by Wattanakriengkrai et al [42] pointed out that a machine learning approach outperforms the n-gram approach. However, their experiment focused solely on file-level defect prediction-not Just-In-Time defect prediction.…”

“…In our studied projects, we found that the average size of the commit varies from 73 to 140 changed lines, but the average ratio of actual defective lines is as low as 51%-53%. Thus, developers still spend unnecessarily effort on locating actual defective lines of that commit [42]. To address bors.…”

Code Prediction by Feeding Trees to Transformers

“…Besides of helping understand the behaviour of deep learning models, interpretability methods have also been used to directly improve the process of software engineering. For example, [43], [44] used LIME [45], a model-agnostic explainability method, to pinpoint the defective lines of code from defect detection models.…”

Deep Learning for Android Malware Defenses: a Systematic Literature Review

“…Interpretable or explainable deep learning based Android malware analysis is also a future interesting topic [60,160]. Recently, researchers have focused on conducting empirical studies to highlight the need of explainable AI/ML models for software engineering [75] and developing novel approaches for explainable AI/ML models for software engineering [75,83,125,125,129,135,176]. Although existing studies have attempted to employ local explainable approaches to provide explanations based on the Android characteristic-based features for each unknown sample [179], there are still several issues requiring further exploration.…”