With a plethora of available classification performance measures, choosing the right metric for the right task requires careful thought. To make this decision in an informed manner, one should study and compare general properties of candidate measures. However, analysing measures with respect to complete ranges of their domain values is a difficult and challenging task. In this study, we attempt to support such analyses with a specialized visualization technique, which operates in a barycentric coordinate system using a 3D tetrahedron. Additionally, we adapt this technique to the context of imbalanced data and put forward a set of properties which should be taken into account when selecting a classification performance measure. As a result, we compare 22 popular measures and show important differences in their behaviour. Moreover, for parametric measures such as the F β and IBA α (G-mean), we analytically derive parameter thresholds that change measure properties. Finally, we provide an online visualization tool that can aid the analysis of complete domain ranges of performance measures. NOTICE: This is a preliminary version of an article submitted to Information Sciences all situations. However, which measure is used in a given problem seems to be, to a large extent, dictated simply by the measure's popularity rather than a thorough discussion of its properties.Although there are a few systematic studies on different properties of classifier performance measures [19,16,11,30], we still postulate the need for thorough analysis of the measures' behaviour. In particular, methods for: interpreting and comparing measures with respect to whole domain ranges, analysing their nature for different class and prediction distributions, and detecting the presence of unusual values are much needed. Theoretical investigations of these aspects are often very laborious and time consuming, especially when multi-dimensional aspects, provided by the confusion matrices, need to be taken into account. Due to these difficulties, such an analysis could be alternatively carried out with visual techniques to aid the understanding and interpretability of various measure properties.In this paper, we put forward a new visualization technique for analysing entire domains of classification performance measures, which depicts all possible configurations of predictions in a confusion matrix, regardless of the used classifier. For this purpose, we adapt an approach originally created for rule interestingness measures to the context of classification [31]. Contrary to existing performance measure visualizations, such as ROC space [11], the proposed approach presents measures in a space which is defined directly on elements of the confusion matrix, is easily interpretable in 3D, and remains defined for all elements of the domain. Moreover, based on the devised visualization, we propose ten properties which should be taken into account while selecting evaluation measures, particularly for class imbalanced data. Consequently, we compare 22 popular c...
