How to evaluate performance of prediction methods? Measures and their interpretation in variation effect analysis

Abstract

BackgroundPrediction methods are increasingly used in biosciences to forecast diverse features and characteristics. Binary two-state classifiers are the most common applications. They are usually based on machine learning approaches. For the end user it is often problematic to evaluate the true performance and applicability of computational tools as some knowledge about computer science and statistics would be needed.ResultsInstructions are given on how to interpret and compare method evaluation results. For systematic method performance analysis is needed established benchmark datasets which contain cases with known outcome, and suitable evaluation measures. The criteria for benchmark datasets are discussed along with their implementation in VariBench, benchmark database for variations. There is no single measure that alone could describe all the aspects of method performance. Predictions of genetic variation effects on DNA, RNA and protein level are important as information about variants can be produced much faster than their disease relevance can be experimentally verified. Therefore numerous prediction tools have been developed, however, systematic analyses of their performance and comparison have just started to emerge.ConclusionsThe end users of prediction tools should be able to understand how evaluation is done and how to interpret the results. Six main performance evaluation measures are introduced. These include sensitivity, specificity, positive predictive value, negative predictive value, accuracy and Matthews correlation coefficient. Together with receiver operating characteristics (ROC) analysis they provide a good picture about the performance of methods and allow their objective and quantitative comparison. A checklist of items to look at is provided. Comparisons of methods for missense variant tolerance, protein stability changes due to amino acid substitutions, and effects of variations on mRNA splicing are presented.

Detecting and Analyzing Corrosion Spots on the Hull of Large Marine Vessels Using Colored 3d Lidar Point Clouds

“…For quantitative results, some of these zones were manually segmented to obtain ground truth. The detection results are presented in Table 1 using different standard evaluation metrics as described in (Vihinen, 2012). The analysis was conducted with 3D points.…”

Detecting and Updating Changes in Lidar Point Clouds for Automatic 3D Urban Cartography

“…Using these values of Similarity threshold and u threshold along with L = 2 m, e tol = (0.000125 0.000125 0.000125) T (in m 3 ) and nreset = 3 (maximum number of passages possible in our case is 4), we then, evaluated the performance of our method, for all three neighborhoods, using different standard evaluation metrics as described in (Vihinen, 2012) (see Table 2). Although, all these metrics are commonly used to evaluate such algorithms, MCC (Matthews Correlation Coefficient) is regarded as most balanced measure as it is insensitive to different class sizes (like in our application the number of changed cells (changes) is generally quite less as compared to unchanged cells in the urban environment).…”

Explanation and prediction of nsSNP-induced pathology using association mining, transduction, and active learning

“…These studies explicitly reported values for %Acc, BER, and MCC, thereby allowing for comparison with our approach. While we utilize an additional performance metric (PPV) to evaluate our approach, %Acc, BER, and MCC is the largest set of metrics shared by all studies [27]. A comparative assessment of all these methods is shown in Table 3, where we also report the size of the data used (Size of DB) as well as the percentage of the data used for training (%Use) so that an easy comparison of these factors can be made.…”

“…Overall performance is derived by averaging performance metrics computed over the k splits. [27] The learning fold is used to train the base classifier / learner (A priori or Ripper) in order to learn association rules while tuning takes place over the validation folds. The final association rules learned are run on the (sequestered) test fold partition and results are tabulated.…”