Performance of in silico prediction tools for the classification of rare BRCA1/2 missense variants in clinical diagnostics

Abstract

BackgroundThe use of next-generation sequencing approaches in clinical diagnostics has led to a tremendous increase in data and a vast number of variants of uncertain significance that require interpretation. Therefore, prediction of the effects of missense mutations using in silico tools has become a frequently used approach. Aim of this study was to assess the reliability of in silico prediction as a basis for clinical decision making in the context of hereditary breast and/or ovarian cancer.MethodsWe tested the performance of four prediction tools (Align-GVGD, SIFT, PolyPhen-2, MutationTaster2) using a set of 236 BRCA1/2 missense variants that had previously been classified by expert committees. However, a major pitfall in the creation of a reliable evaluation set for our purpose is the generally accepted classification of BRCA1/2 missense variants using the multifactorial likelihood model, which is partially based on Align-GVGD results. To overcome this drawback we identified 161 variants whose classification is independent of any previous in silico prediction. In addition to the performance as stand-alone tools we examined the sensitivity, specificity, accuracy and Matthews correlation coefficient (MCC) of combined approaches.ResultsPolyPhen-2 achieved the lowest sensitivity (0.67), specificity (0.67), accuracy (0.67) and MCC (0.39). Align-GVGD achieved the highest values of specificity (0.92), accuracy (0.92) and MCC (0.73), but was outperformed regarding its sensitivity (0.90) by SIFT (1.00) and MutationTaster2 (1.00). All tools suffered from poor specificities, resulting in an unacceptable proportion of false positive results in a clinical setting. This shortcoming could not be bypassed by combination of these tools. In the best case scenario, 138 families would be affected by the misclassification of neutral variants within the cohort of patients of the German Consortium for Hereditary Breast and Ovarian Cancer.ConclusionWe show that due to low specificities state-of-the-art in silico prediction tools are not suitable to predict pathogenicity of variants of uncertain significance in BRCA1/2. Thus, clinical consequences should never be based solely on in silico forecasts. However, our data suggests that SIFT and MutationTaster2 could be suitable to predict benignity, as both tools did not result in false negative predictions in our analysis.Electronic supplementary materialThe online version of this article (10.1186/s12920-018-0353-y) contains supplementary material, which is available to authorized users.

BRCA1 ‐ and BRCA2 ‐specific in silico tools for variant interpretation in the CAGI 5 ENIGMA challenge

“…In a previous study, using a small dataset of BRCA2 variants, Karchin, Agarwal, Sali, Couch, and Beattie () find that general tools display good sensitivities but low specificities. A similar trend has been recently reported by Ernst et al (), after testing PolyPhen‐2, SIFT, Align‐GVGD, and MutationTaster2 in a set of 236 BRCA1/2 variants. These authors express concern about the moderate performance observed, particularly about the low specificities observed relative to HBOC diagnosis requirements (e.g., PolyPhen‐2: 0.67 and 0.72 for BRCA1 and BRCA2, respectively).…”

“…In a previous study, using a small dataset of BRCA2 variants, Karchin, Agarwal, Sali, Couch, and Beattie (2008) find that general tools display good sensitivities but low specificities. A similar trend has been recently reported by Ernst et al (2018), after testing PolyPhen-2, SIFT, Align-GVGD, and…”

“…Obtaining good estimates of the functional impact and cancer risk of BRCA1 and BRCA2 sequence variants plays a vital role in the diagnosis and management of inherited breast and ovarian cancers (Eccles et al, 2015;Findlay et al, 2018;Guidugli et al, 2018;Moreno et al, 2016;Paluch-Shimon et al, 2016). A priori, in silico tools, can be used to obtain these estimates; however, their moderate success rate restricts their applicability (Ernst et al, 2018). In this work, we have addressed this issue focusing on the problem of predicting the pathogenicity of BRCA1/2 missense variants using protein-specific information (Riera et al, 2014).…”

“…When tested in isolation, we find that our two methods (MLR and NN) are competitive when compared with general methods (Section 3.5; Table 3; Figure 4), for both BRCA1 and BRCA2. In particular, their specificities are among the best, a property desirable from the point of view of HBOC diagnosis requirements (Ernst et al, 2018); they also have the best balances between specificity and sensitivity, with the only exception of PMut in BRCA1, which has slightly better (PROVEAN) and 0.19 (Mutation Assessor). In a previous study, using a small dataset of BRCA2 variants, Karchin, Agarwal, Sali, Couch, and Beattie (2008) find that general tools display good sensitivities but low specificities.…”

“…However, the moderate success rate of these tools is an obstacle for their extended use in a clinical environment (Riera, Lois, & de la Cruz, ). In the specific case of BRCA1/2 , Ernst et al () suggest, after testing the performance of Align‐GVGD, SIFT, PolyPhen‐2, and MutationTaster2 on a set of 236 BRCA1/2 variants of known effect, that in silico results cannot be used as stand‐alone evidence for diagnosis. In terms of molecular effect, two independent, massive functional assays of BRCA1 variants (Findlay et al, ; Starita et al, ) show that in silico predictors provide only a limited view of the functional impact of these variants.…”