Efficacious End User Measures—Part 1: Relative Class Size and End User Problem Domains

Abstract

Biological and medical endeavors are beginning to realize the benefits of artificial intelligence and machine learning. However, classification, prediction, and diagnostic (CPD) errors can cause significant losses, even loss of life. Hence, end users are best served when they have performance information relevant to their needs, this paper’s focus. Relative class size (rCS) is commonly recognized as a confounding factor in CPD evaluation. Unfortunately, rCS-invariant measures are not easily mapped to end user conditions. We determine a cause of rCS invariance, joint probability table (JPT) normalization. JPT normalization means that more end user efficacious measures can be used without sacrificing invariance. An important revelation is that without data normalization, the Matthews correlation coefficient (MCC) and information coefficient (IC) are not relative class size invariants; this is a potential source of confusion, as we found not all reports using MCC or IC normalize their data. We derive MCC rCS-invariant expression. JPT normalization can be extended to allow JPT rCS to be set to any desired value (JPT tuning). This makes sensitivity analysis feasible, a benefit to both applied researchers and practitioners (end users). We apply our findings to two published CPD studies to illustrate how end users benefit.

Efficacious Discriminant Analysis (Classifier) Measures for End Users

“…The ratio + invariant measures, ROC-AUC and the Youden index, have as their basis the measure suite { + /| |, + /| |} [6]. Both of these measures are conditioned by ground truth ( and ), not the DA tool outputs visible to the end user ( and ); they do not satisfy Criterion 3.…”

“…There are occasions when the impact is not cumulative, but each output is important individually, for example, a medical diagnosis. In these situations, ratio + is confounding, so normalized JPTs are used; normalization mathematically balances the relative class sizes, and thus it mitigates any skew resulting from ratio + (JPT normalization's ratio + mitigation is limited by the strong law of large numbers; if the minor class sample is too small, then ratio + skew exerts a significant influence [6]…”

“…Thus, can be calculated from data available to end users and can be deconstructed into and relevant components. , as written in (6), is the sum of the estimated impact of and ; can be deconstructed into and relevant components. Thus both satisfy Criterion 3.…”

“…(Such a sensitivity analysis can test results at the identified boundary; however, ratio + causes the optimum boundary to shift. So, without the actual data, JPT tuning cannot be used to estimate the peak impact [6].) JPT tuning is used for two other relative class sizes: ratio + = 1.5 and ratio + = 2.5.…”

“…As an example of the problem end users face, two commonly seen summary statistics [4,5], total accuracy rate (TAR) [5] and -score [5], have opposite responses to relative class size (ratio + ) (also known as class imbalance and prevalence) [6]. (In the current context, summary statistics are formulae that take multiple joint probability table (JPT) based values as input and output a single value that represents the target DA tool's composite utility.…”