The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets

Saito, Takaya; Rehmsmeier, Marc

doi:10.1371/journal.pone.0118432

Cited by 2,873 publications

(2,154 citation statements)

References 47 publications

Supporting

Mentioning

1,943

Contrasting

Unclassified

Order By: Relevance

“…Namely, such setting adopts the k-fold cross-validation (CV) procedure (k = 5) to assess the generalization abilities of the compared methods, and the Area Under the ROC Curve (AUC) and the Precision at different Recall levels (PXR) to measure the corresponding performance. Furthermore, as done by authors of NWGP on the same data, we also computed for our method the Area Under the Precision-Recall Curve (AUPRC), being AUPRC more informative than AUC on unbalanced settings [68].…”

Section: Resultsmentioning

confidence: 99%

Gene2DisCo: Gene to disease using disease commonalities

Frasca

2017

Artificial Intelligence in Medicine

View full text Add to dashboard Cite

show abstract

Section: Resultsmentioning

confidence: 99%

Gene2DisCo: Gene to disease using disease commonalities

Frasca

2017

Artificial Intelligence in Medicine

View full text Add to dashboard Cite

show abstract

“…These results are also confirmed by the AUROC 50 , AUROC 100 , and AUROC 1000 results, where hyperSMURF with tuned parameters largely and significantly outperforms hyperSMURF with default parameters ( Table 1). Note that we do not report AUROC results, since in this highly imbalanced context pure AUROC results are not as significant as AUPRC or AUROC limited to the top ranked SNVs [Saito and Rehmsmeier, 2015]. Table 1: Comparison of hyperSMURF results obtained respectively with default parameters (n = 100, f = 2, m = 3) and with the best parameters obtained by internal cross-validation on the training data (n = 300, f = 1, m = 10).…”

Section: Resultsmentioning

confidence: 99%

Parameters tuning boosts hyperSMURF predictions of rare deleterious non-coding genetic variants

Petrini

Schubach

Ré

et al. 2017

Preprint

View full text Add to dashboard Cite

The regulatory code that determines whether and how a given genetic variant affects the function of a regulatory element remains poorly understood for most classes of regulatory variation. Indeed the large majority of bioinformatics tools have been developed to predict the pathogenicity of genetic variants in coding sequences or conserved splice sites.Computational algorithms for the prediction of non-coding deleterious variants associated with rare genetic diseases are faced with special challenges owing to the rarity of confirmed pathogenic mutations. Indeed in this context classical machine learning methods are biased toward neutral variants that constitute the large majority of genetic variation, and are not able to detect the potential deleterious variants that constitute only a tiny minority of all known genetic variation. We recently proposed hyperSMURF, hyperensemble of SMOTE Undersampled Random Forests, an ensemble approach explicitly designed to deal with the huge imbalance between deleterious and neutral variants, and able to significantly outperform state-of-the-art methods for the prediction of non-coding variants associated with Mendelian diseases. Despite its successful application to the detection of deleterious single nucleotide variants (SNV) as well as to small insertions or deletions (indels), hyperSMURF is a method that depends on several learning parameters, that strongly influence its overall performances. In this work we experimentally show that by tuning hyperSMURF parameters we can significantly boost the performance of the method, thus predicting with significantly better precision and recall rare SNVs associated with Mendelian diseases.PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.3185v1 | CC BY 4.0 Open Access |

show abstract

“…The sample genotypes were analyzed using the ΔΔCt [22] method in accordance with the description [8], the data was processed with the SPSS statistics package [23] using the ROCanalysis [24]. The use of other binary classifiers [25] and statistical methods [18] to increase the reliability of results can be promising in such studies.…”

Section: Introductionmentioning

confidence: 99%

MOLECULAR GENOTYPING OF CHICKEN (Gallus gallus L.) FEATHERING GENES IN CONNECTION WITH SEPARATION BY SEX

Alekseev¹,

Borodin²,

Nikulin³

et al. 2017

S-h. biol.

View full text Add to dashboard Cite

A b s t r a c tTraditional breeding is time and material consuming. Modern laboratory techniques significantly speed up and reduce the costs for breeding animals with the desired properties. A test based on the quantitative real-time PCR (qPCR) technology was developed to distinguish between homozygous and heterozygous state of the genes from alleles K and k which are responsible for the rate of wing feather growth in day-old chicks. The use of quantitative real-time PCR for the analysis of genotypes is aimed at the discrimination between one and two copies of the target gene in a genome. To obtain reliable results, certain rules must be followed when conducting the assay: the efficiency of the PCR should be close to the maximum; it is possible to obtain a significant number of false results without the appropriate statistical analysis. A new assay algorithm was proposed to overcome the limitations of qPCR: all samples are subjected to two successive independent analyses in parallel with the reference samples of both genotypes; if the two runs produce divergent results then the assay is repeated and the previous results are discarded. This approach allows to reduce assay error probability down to zero. The new system consists of three (instead of four) primers for amplification of two genes and two probes, allowing efficient analysis of various allele K genotypes. Quantitative realtime PCR data analysis was performed by ΔΔCt method using the statistics software package SPSS for ROC analysis. Using the method developed, the percentage of KK, Kk and kk genotypes was determined in 145 cocks of original lines B5, B6, B7 and B9 of domestic meat chicken of cross Smena 8. It was shown that 19 cocks of line B5 and 15 cocks of line B6 had kk genotype. From the 46 cocks of line B7, none had kk genotype, 17 cocks (37 %) had Kk genotype, and 29 cocks (63 %) had KK genotype. From the 65 cocks of line B9, none had kk genotype, 17 cocks (26 %) had Kk genotype, and 48 cocks (74 %) had KK genotype. Analyzed fragments were sequenced to exclude the effects of possible nucleotide sequence variability on the assay. The sequences did not contain any nucleotide substitutions in the sites of the primers and probes annealing. The data obtained will accelerate selection of new domestic meat chicken breeds with possibility of sexing based on feather length in day-old chicken. Further breeding work involves the assessment of the offspring using traditional and molecular genetic methods.

show abstract

The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets

Cited by 2,873 publications

References 47 publications

Gene2DisCo: Gene to disease using disease commonalities

Gene2DisCo: Gene to disease using disease commonalities

Parameters tuning boosts hyperSMURF predictions of rare deleterious non-coding genetic variants

MOLECULAR GENOTYPING OF CHICKEN (Gallus gallus L.) FEATHERING GENES IN CONNECTION WITH SEPARATION BY SEX

Contact Info

Product

Resources

About