Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes

Abstract

Purpose Where multiple in silico tools are concordant, the American College of Medical Genetics and Genomics/Association for Molecular Pathology (ACMG/AMP) framework affords supporting evidence toward pathogenicity or benignity, equivalent to a likelihood ratio of ~2. However, limited availability of “clinical truth sets” and prior use in tool training limits their utility for evaluation of tool performance. Methods We created a truth set of 9,436 missense variants classified as deleterious or tolerated in clinically validated high-throughput functional assays for BRCA1, BRCA2, MSH2, PTEN, and TP53 to evaluate predictive performance for 44 recommended/commonly used in silico tools. Results Over two-thirds of the tool–threshold combinations examined had specificity of <50%, thus substantially overcalling deleteriousness. REVEL scores of 0.8–1.0 had a Positive Likelihood Ratio (PLR) of 6.74 (5.24–8.82) compared to scores <0.7 and scores of 0–0.4 had a Negative Likelihood Ratio (NLR) of 34.3 (31.5–37.3) compared to scores of >0.7. For Meta-SNP, the equivalent PLR = 42.9 (14.4–406) and NLR = 19.4 (15.6–24.9). Conclusion Against these clinically validated “functional truth sets," there was wide variation in the predictive performance of commonly used in silico tools. Overall, REVEL and Meta-SNP had best balanced accuracy and might potentially be used at stronger evidence weighting than current ACMG/AMP prescription, in particular for predictions of benignity.

Quantifying prediction of pathogenicity for within-codon concordance (PM5) using 7541 functional classifications of BRCA1 and MSH2 missense variants

“…Notably, in this context the tool is used to compare relative deleteriousness between colocated variants, rather than being used with a prespecified binary threshold of pathogenicity, which is more typical in other tool evaluations. 22,27 The FP rate is generally low for all PM5-definitions: 3.2% (244/7541) for the most lenient definition of PM5 (PM5-definition_a) and <1% for more stringent PM5definitions. The low FP rate drives high specificity, positive predictive value, and pLRs for calling of pathogenicity.…”

“…Notably, in this context the tool is used to compare relative deleteriousness between colocated variants, rather than being used with a prespecified binary threshold of pathogenicity, which is more typical in other tool evaluations. 22 , 27 …”

“…Second, we would advocate use of different in silico tools for PM5 and PP3; measures of protein distance such as BLOSUM and The Grantham Score are most apposite for PM5 evaluation, whereas best performance for PP3 is attained by meta tools such as REVEL and Meta-SNP optimized on multiple datasources. 22 Third, once high quality MAVE data become available for (nearly) all variants in a gene (or region), we would deem that PM5 has been superseded and has become redundant.…”

“…[11][12][13]18,19 REVEL, Meta-SNP, and CADD were selected because these tools are widely-used clinically and/or assessed as high-performing. 14,[20][21][22] In silico scores were retrieved using Annovar (dbnsfp33a database), Alamut-HT, the Meta-SNP, and REVEL webservers. 14,20,23,24 Generation and evaluation of PM5 predictions…”

Parental Segregation Study Reveals Rare Benign and Likely Benign Variants in a Brazilian Cohort of Rare Diseases

“…In this study, we have used REVEL, which is an ensemble method for predicting the pathogenicity of rare missense variants, as our sole in-silico predictor. REVEL has shown better performance when compared to other methods especially for rare neutral variants [2,3,15]. These new pieces of evidence demonstrate adequate strength of REVEL as a benignity predictor and make this tool t adequately in our study, which aims to unravel rare neutral variants.…”