Assessment of in silico protein sequence analysis in the clinical classification of variants in cancer risk genes

Abstract

Missense variants represent a significant proportion of variants identified in clinical genetic testing. In the absence of strong clinical or functional evidence, the American College of Medical Genetics recommends that these findings be classified as variants of uncertain significance (VUS). VUSs may be reclassified to better inform patient care when new evidence is available. It is critical that the methods used for reclassification are robust in order to prevent inappropriate medical management strategies and unnecessary, life-altering surgeries. In an effort to provide evidence for classification, several in silico algorithms have been developed that attempt to predict the functional impact of missense variants through amino acid sequence conservation analysis. We report an analysis comparing internally derived, evidence-based classifications with the results obtained from six commonly used algorithms. We compiled a dataset of 1118 variants in BRCA1, BRCA2, MLH1, and MSH2 previously classified by our laboratory's evidence-based variant classification program. We compared internally derived classifications with those obtained from the following in silico tools: Align-GVGD, CONDEL, Grantham Analysis, MAPP-MMR, PolyPhen-2, and SIFT. Despite being based on similar underlying principles, all algorithms displayed marked divergence in accuracy, specificity, and sensitivity. Overall, accuracy ranged from 58.7 to 90.8% while the Matthews Correlation Coefficient ranged from 0.26-0.65. CONDEL, a weighted average of multiple algorithms, did not perform significantly better than its individual components evaluated here. These results suggest that the in silico algorithms evaluated here do not provide reliable evidence regarding the clinical significance of missense variants in genes associated with hereditary cancer.

Performance of in silico prediction tools for the classification of rare BRCA1/2 missense variants in clinical diagnostics

“…This means, that nsSNPs classified as benign by SIFT or MutationTaster2 are actual benign variants in the overwhelming majority of cases, i.e., false negative predictions are rare. In agreement with our findings, Kerr et al [ 34 ] observed no false negative calls of SIFT in a set of 69 pathogenic BRCA1/2 nsSNPs. However, confirmation of our hypothesis and its application in clinical diagnostics requires further investigation on larger data sets.…”

“…We refer to Additional file 3 : Table S2 for a summarizing review. The poor results of PolyPhen-2 when compared with Align-GVGD, SIFT and MutationTaster2 in our study are in agreement with a previous study from Rodrigues et al [ 15 ], as well as with the specificities obtained by Hicks et al, Kerr et al, and Miosge et al [ 12 , 34 , 35 ]. However, we were not able to confirm the results of Kerr et al [ 34 ] that indicated a poor performance of SIFT on BRCA1/2 missense variants, especially concerning its specificity.…”

“…The MCC is particularly suitable for the evaluation of predictions on imbalanced data [ 32 , 33 ]. As variant sets for the purpose of evaluation of in silico prediction approaches typically show a strong bias towards neutral variants (pathogenic variants are expected to seldom occur), the MCC has been used as a performance measure in a variety of studies on in silico prediction approaches [ 13 , 15 , 31 , 34 ]. MCC values are defined in a range from -1 (always falsely predicted) to 1 (perfectly predicted) with a value of 0 corresponding to a completely random prediction.…”

Benchmarking analysis of deleterious SNP prediction tools on CYP2D6 enzyme

“…In predicting the effect of missense SNPs on UGT1A1, Rodrigues et al (2015) demonstrated that 13 of the 20 tools had an MCC close to zero, and only the predictor of MutPred reached a value of MCC ≥ 0.7 (Rodrigues et al, 2015). Kerr et al (2017) used 7 tools to predict the functional impact of variants of the BRCA1, BRCA2, MLH1 and MSH2 genes associated with hereditary cancer, and demonstrated that within the evaluated set, 5 algorithms had coefficients below 0.35 and only two tools (Align -GVGD (0.65) and MAPP-MMR (0.59)) had MCCs greater than 0.59 (Kerr et al, 2017).…”

DNA repair-related functional assays for the classification of BRCA1 and BRCA2 variants: a critical review and needs assessment

“…Many of these algorithms are not highly sensitive in isolation and tend to ‘over predict’ missense changes as pathogenic 27. Thus, there is a need to develop better in silico models that show higher specificity and sensitivity; it may be that models will need to be calibrated on a gene-by-gene basis.…”