Preference-Driven Classification Measure

Abstract

Classification is one of the main problems of machine learning, and assessing the quality of classification is one of the most topical tasks, all the more difficult as it depends on many factors. Many different measures have been proposed to assess the quality of the classification, often depending on the application of a specific classifier. However, in most cases, these measures are focused on binary classification, and for the problem of many decision classes, they are significantly simplified. Due to the increasing scope of classification applications, there is a growing need to select a classifier appropriate to the situation, including more complex data sets with multiple decision classes. This paper aims to propose a new measure of classifier quality assessment (called the preference-driven measure, abbreviated p-d), regardless of the number of classes, with the possibility of establishing the relative importance of each class. Furthermore, we propose a solution in which the classifier’s assessment can be adapted to the analyzed problem using a vector of preferences. To visualize the operation of the proposed measure, we present it first on an example involving two decision classes and then test its operation on real, multi-class data sets. Additionally, in this case, we demonstrate how to adjust the assessment to the user’s preferences. The results obtained allow us to confirm that the use of a preference-driven measure indicates that other classifiers are better to use according to preferences, particularly as opposed to the classical measures of classification quality assessment.

Assessing Sensor Integrity for Nuclear Waste Monitoring Using Graph Neural Networks

“…Eventually, the same could be achieved using the F1-scores [62,[64][65][66], and the results are presented in Figure 20. The F1-score is the harmonic mean of the recall and precision, as showcased by Equation ( 24):…”

“…This section shows the results of each of the 1620 GNNs tested on the unique testing dataset. Using the confusion matrix presented in Figure 16 , it was possible to use recall metrics [ 62 , 64 , 65 , 66 ], such as the true positive rate (TPR) and true negative rate (TNR), to plot the efficiency of each of the GNNs. Figure 17 presents the recall metrics on the testing dataset for each GNN trained.…”

“…Then, by using the accuracy metrics [ 62 , 64 , 65 , 66 ], it was possible to plot the proportion of GNNs that attained certain accuracy thresholds. This is shown in Figure 18 .…”

“…Figure 17 presents the recall metrics on the testing dataset for each GNN trained. Equations ( 19) and ( 20) describe the recall metrics: Then, by using the accuracy metrics [62,[64][65][66], it was possible to plot the proportion of GNNs that attained certain accuracy thresholds. This is shown in Figure 18.…”

“…This is shown in Figure 18. Equation ( 21) presents the accuracy metric: The same analysis could be performed using precision metrics [62,[64][65][66], such as positive predictive value (PPV) and negative predictive value (NPV). Figure 19 shows the distribution of the precision metrics on the testing dataset for each GNN tested.…”