Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation

Mason, Simon J.; Graham, Nicholas E.

doi:10.1256/003590002320603584

Cited by 734 publications

(531 citation statements)

References 88 publications

Supporting

Mentioning

527

Contrasting

Unclassified

Order By: Relevance

“…In fact, the AUC metric has been shown to be equal to the probability of, given a pair of items with different labels (one "success" case and one "failure" case, in our situation), correctly considering the item that was truly a "success" as more likely to be a "success" case than the other item. It has also been shown to be related to other important statistical metrics, such as the Wilcoxon test of ranks, the Mann-Whitney U and the Gini coefficient [32,33,59].…”

Section: Multiple Threshold Analysismentioning

confidence: 99%

Every team deserves a second chance: an extended study on predicting team performance

Marcolino

Lakshminarayanan

Nagarajan

et al. 2016

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

Voting among different agents is a powerful tool in problem solving, and it has been widely applied to improve the performance in finding the correct answer to complex problems. We present a novel benefit of voting, that has not been observed before: we can use the voting patterns to assess the performance of a team and predict their final outcome. This prediction can be executed at any moment during problem-solving and it is completely domain independent. Hence, it can be used to identify when a team is failing, allowing an operator to take remedial procedures (such as changing team members, the voting rule, or increasing the allocation of resources). We present three main theoretical results: (1) we show a theoretical explanation of why our prediction method works; (2) contrary to what would be expected based on a simpler explanation using classical voting models, we show that we can make accurate predictions irrespective of the strength (i.e., performance) of the teams, and that in fact, the prediction can work better for diverse teams composed of different agents than uniform teams made of copies of the best agent; (3) we show that the quality of our prediction increases with the size of the action space. We perform extensive experimentation in two different domains: Computer Go and Ensemble Learning. In Computer Go, we obtain high Multi-Agent Syst (2017) 31:1003-1054 quality predictions about the final outcome of games. We analyze the prediction accuracy for three different teams with different levels of diversity and strength, and show that the prediction works significantly better for a diverse team. Additionally, we show that our method still works well when trained with games against one adversary, but tested with games against another, showing the generality of the learned functions. Moreover, we evaluate four different board sizes, and experimentally confirm better predictions in larger board sizes. We analyze in detail the learned prediction functions, and how they change according to each team and action space size. In order to show that our method is domain independent, we also present results in Ensemble Learning, where we make online predictions about the performance of a team of classifiers, while they are voting to classify sets of items. We study a set of classical classification algorithms from machine learning, in a data-set of hand-written digits, and we are able to make high-quality predictions about the final performance of two different teams. Since our approach is domain independent, it can be easily applied to a variety of other domains.

show abstract

Section: Multiple Threshold Analysismentioning

confidence: 99%

Every team deserves a second chance: an extended study on predicting team performance

Marcolino

Lakshminarayanan

Nagarajan

et al. 2016

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

show abstract

“…In order to estimate the statistical significance of each k-ellipse, we have to resort to the statistics of the Area Under the Curve (AUC), labelled by A, in the ROC plane which has been the subject of the study by Mason and Graham [10]. Mason and Graham [10] have shown that…”

Section: The K-ellipses Familymentioning

confidence: 99%

“…For large values of P and Q, i.e., when P or Q are greater or equal to 30 and P + Q ≥ 40, the U-statistics can be [10] approximated by a Gaussian distribution having a mean…”

Section: The K-ellipses Familymentioning

confidence: 99%

See 1 more Smart Citation

Visualization of the significance of Receiver Operating Characteristics based on confidence ellipses

Sarlis

Christopoulos

2014

Computer Physics Communications

View full text Add to dashboard Cite

The Receiver Operating Characteristics (ROC) is used for the evaluation of prediction methods in various disciplines like meteorology, geophysics, complex system physics, medicine etc. The estimation of the significance of a binary prediction method, however, remains a cumbersome task and is usually done by repeating the calculations by Monte Carlo. The FORTRAN code provided here simplifies this problem by evaluating the significance of binary predictions for a family of ellipses which are based on confidence ellipses and cover the whole ROC space.Keywords: Receiver Operating Characteristics (ROC), complex systems, systems obeying power laws, significance level, p-value Solution method: Using the statistics of random binary predictions for a given value of the predictor threshold ϵ t , one can construct the corresponding confidence ellipses. The envelope of these corresponding confidence ellipses is estimated when ϵ t varies from 0 to 1. This way a new family of ellipses is obtained, named kellipses, which covers the whole ROC plane and leads to a well defined Area Under the Curve (AUC). For the latter quantity, Mason and Graham [1] has shown that it follows the Mann-Whitney U-statistics[2] which can be applied [3] for the estimation of the statistical significance of each k-ellipse. As the transformation is invertible, any point on the ROC plane corresponds to a unique value of k, thus to a unique p-value to obtain this point by chance. The present FORTRAN code provides this p-value field on the ROC plane as well as the k-ellipses corresponding to the (p=)10%, 5% and 1% significance levels using as input the number of the positive (P) and negative (Q) cases to be predicted. PROGRAM SUMMARY Restrictions:Unusual features: In some machines, the compiler directive -O2 or -O3 should be used to avoid NaN's in some points of the p-field along the diagonal Additional comments:Running time: Depending on the application, e.g., 4s for an Intel(R) Core(TM)2 CPU E7600 at 3.06GHz with 2GB RAM for the examples presented here[1] S.

show abstract

“…MOS applied to 42 years of DEMETER (Palmer et al, 2004) data, and in particular DJF forecast fields produced by the UKMO fully coupled system, show correlation skill patterns over South Africa (not shown) similar to the ECHAM4.5-MOS simulations presented here. In addition to the similarities in the spatial distribution of the correlation fields, additional verification parameters such as linear error in probability space (LEPS) (Potts et al, 1996) and relative operating characteristic (ROC) scores (Mason and Graham, 2002) show patterns similar to the correlation fields (not shown). Therefore, as stated earlier, the correlation maps presented here should be considered good approximations of model performance spatially.…”

Section: Simulation Skill Levels For the 10-year Periodmentioning

confidence: 99%

Performance comparison of some dynamical and empirical downscaling methods for South Africa from a seasonal climate modelling perspective

Landman

Kgatuke

Mbedzi

et al. 2008

Intl Journal of Climatology

View full text Add to dashboard Cite

ABSTRACT:The ability of advanced state-of-the-art methods of downscaling large-scale climate predictions to regional and local scale as seasonal rainfall forecasting tools for South Africa is assessed. Various downscaling techniques and raw general circulation model (GCM) output are compared to one another over 10 December-January-February (DJF) seasons from 1991/1992 to 2000/2001 and also to a baseline prediction technique that uses only global sea-surface temperature (SST) anomalies as predictors. The various downscaling techniques described in this study include both an empirical technique called model output statistics (MOS) and a dynamical technique where a finer resolution regional climate model (RCM) is nested into the large-scale fields of a coarser GCM. The study addresses the performance of a number of simulation systems (no forecast lead-time) of varying complexity. These systems' performance is tested for both homogeneous regions and for 963 stations over South Africa, and compared with each other over the 10-year test period. For the most part, the simulations method outscores the baseline method that uses SST anomalies to simulate rainfall, therefore providing evidence that current approaches in seasonal forecasting are outscoring earlier ones. Current operational forecasting approaches involve the use of GCMs, which are considered to be the main tool whereby seasonal forecasting efforts will improve in the future. Advantages in statistically post-processing output from GCMs as well as output from RCMs are demonstrated. Evidence is provided that skill should further improve with an increased number of ensemble members. The demonstrated importance of statistical models in operation capacities is a major contribution to the science of seasonal forecasting. Although RCMs are preferable due to physical consistency, statistical models are still providing similar or even better skill and should still be applied.

show abstract

Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation

Cited by 734 publications

References 88 publications

Every team deserves a second chance: an extended study on predicting team performance

Every team deserves a second chance: an extended study on predicting team performance

Visualization of the significance of Receiver Operating Characteristics based on confidence ellipses

Performance comparison of some dynamical and empirical downscaling methods for South Africa from a seasonal climate modelling perspective

Contact Info

Product

Resources

About