Visualization of the significance of Receiver Operating Characteristics based on confidence ellipses

Abstract

The Receiver Operating Characteristics (ROC) is used for the evaluation of prediction methods in various disciplines like meteorology, geophysics, complex system physics, medicine etc. The estimation of the significance of a binary prediction method, however, remains a cumbersome task and is usually done by repeating the calculations by Monte Carlo. The FORTRAN code provided here simplifies this problem by evaluating the significance of binary predictions for a family of ellipses which are based on confidence ellipses and cover the whole ROC space.Keywords: Receiver Operating Characteristics (ROC), complex systems, systems obeying power laws, significance level, p-value Solution method: Using the statistics of random binary predictions for a given value of the predictor threshold ϵ t , one can construct the corresponding confidence ellipses. The envelope of these corresponding confidence ellipses is estimated when ϵ t varies from 0 to 1. This way a new family of ellipses is obtained, named kellipses, which covers the whole ROC plane and leads to a well defined Area Under the Curve (AUC). For the latter quantity, Mason and Graham [1] has shown that it follows the Mann-Whitney U-statistics[2] which can be applied [3] for the estimation of the statistical significance of each k-ellipse. As the transformation is invertible, any point on the ROC plane corresponds to a unique value of k, thus to a unique p-value to obtain this point by chance. The present FORTRAN code provides this p-value field on the ROC plane as well as the k-ellipses corresponding to the (p=)10%, 5% and 1% significance levels using as input the number of the positive (P) and negative (Q) cases to be predicted. PROGRAM SUMMARY Restrictions:Unusual features: In some machines, the compiler directive -O2 or -O3 should be used to avoid NaN's in some points of the p-field along the diagonal Additional comments:Running time: Depending on the application, e.g., 4s for an Intel(R) Core(TM)2 CPU E7600 at 3.06GHz with 2GB RAM for the examples presented here[1] S.

Statistical Significance of Minimum of the Order Parameter Fluctuations of Seismicity Before Major Earthquakes in Japan

“…Very recently, a visualization scheme for the statistical significance of ROC curves has been proposed (Sarlis and Christopoulos, 2014). It is based on k-ellipses which are the envelopes of the confidence ellipses -cf.…”

“…a point lies outside a confidence ellipse with probability exp(−k/2)-obtained when using a random predictor and vary the prediction 8 N. V. Sarlis et al threshold. These k-ellipses cover the whole ROC plane and upon using their A we can have a measure (Sarlis and Christopoulos, 2014) of the probability p to obtain by chance (i.e., using a random predictor) an ROC curve passing through each point of the ROC plane.…”

Change ΔS of the entropy in natural time under time reversal: Complexity measures upon change of scale

“…In each case, the colored contours depict the probability p to obtain by chance an ROC point in the ROC diagram as it results [53] from the study of confidence ellipses. This visualization scheme for the statistical significance of ROC curves is based [53] on the k-ellipses which are the envelop es of the confidence ellipses -cf. a point lies outside a confidence ellipse with probability exp(−k/2)-obtained when using a random predictor and vary the prediction threshold.…”

“…A random predictor yields ROC curves around the diagonal and the statistical significance of a predictor can be estimated [52] by evaluating the area under the curve (AUC) in the ROC diagram (see also ref. [53]). When the number of positive and negative individuals remains the same (as holds here), the larger AUC signifies a smaller probability to obtain the results by chance, and hence a higher statistical significance.…”

“…a point lies outside a confidence ellipse with probability exp(−k/2)-obtained when using a random predictor and vary the prediction threshold. The k-ellipses cover the whole ROC plane and using their AUC we can have a measure [53] of the probability p to obtain by chance (i.e., using a random predictor) an ROC curve passing through each point on the ROC plane. We observe that in both cases highly statistically significant results are obtained that lie well above the p = 1% k-ellipses.…”