Applications of psychological science for actionable analytics

Abstract

Actionable analytics are those that humans can understand, and operationalize. What kind of data mining models generate such actionable analytics? According to psychological scientists, humans understand models that most match their own internal models, which they characterize as lists of "heuristic" (i.e., lists of very succinct rules). One such heuristic rule generator is the Fast-and-Frugal Trees (FFT) preferred by psychological scientists. Despite their successful use in many applied domains, FFTs have not been applied in software analytics. Accordingly, this paper assesses FFTs for software analytics.We find that FFTs are remarkably effective. Their models are very succinct (5 lines or less describing a binary decision tree). These succinct models outperform state-of-the-art defect prediction algorithms defined by Ghortra et al. at ICSE'15. Also, when we restrict training data to operational attributes (i.e., those attributes that are frequently changed by developers), FFTs perform much better than standard learners.Our conclusions are two-fold. Firstly, there is much that software analytics community could learn from psychological science. Secondly, proponents of complex methods should always baseline those methods against simpler alternatives. For example, FFTs could be used as a standard baseline learner against which other software analytics tools are compared.

Better Data Labelling With EMBLEM (and how that Impacts Defect Prediction)

“…The best tree (as discovered on the training data) is then applied to the test data. Despite the apparent simplicity of FFTs, Chen et al [18] reported that this method performs dramatically better than many prior results seen at recent ICSE conferences [3], [36]. (including those that used hyperparameter optimization and data pre-processing with SMOTE).…”

“…They say a "good" defect predictor selects the 20% of files containing 80% of the defects In the literature, this 20/80 rule is often called P opt 20 (the percent of the bugs found after reading 20%). P opt 20 is widely used in the literature and, for details on that measure, we refer the reader to those publications [18], [42], [48], [62], [64], [69], [69], [111]. For this paper, all we need to say about P opt 20 is the conclusions reached from this metric are nearly the same as the conclusions reached via G-score.…”

“…learning on static code attributes such as C.K. and McGabe metrics) [1], [3], [18], [20], [33], [36], [44], [57], [64], [65], [72], [74], [80], [96] that are more granulated and high-dimensional.…”

How to “DODGE” Complex Software Analytics

“…• Software defect prediction (classifying modules into "buggy" or otherwise [3], [15], [22], [24], [36], [56], [57]); • Software bug report text mining (to find severity [3], [44]). Table 1 is a partial list of some of the tunings that might be explored.…”

“…Recently it was discovered how to (a) save most of that CPU cost while at the same time (b) find better tunings. As discussed later, a method called "FFtrees" [51] (which just selects a best model within a small forest of shallow decision trees) generates much better predictions than supposed state-of-the-art results obtained after CPU intensive tuning [15]. This is strange since standard tuning tries thousands of options, but FFtrees tries just a dozen.…”