Using deep mutational scanning to benchmark variant effect predictors and identify disease mutations

Livesey, Benjamin J; Marsh, Joseph A.

doi:10.1101/855957

Cited by 51 publications

(119 citation statements)

References 100 publications

Supporting

Mentioning

106

Contrasting

Unclassified

Order By: Relevance

“…1c, gray distributions). Another recent analysis also found a method heavily relying on evolutionary information as one of the best performers on DMS data, although more sophisticated than our naïve approach [48,76].…”

Section: Family Conservation Carries Most Important Signalmentioning

confidence: 78%

“…DMS datasets constitute a uniquely valuable resource for the evaluation of current SAV effect prediction methods [17,47,48], not the least, because most have not used those data. The Fowler lab has, recently, published an excellent analysis of prediction methods on DMS datasets and developed a new regression-based prediction method, Envision, trained only on DMS data [49].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Variant effect predictions capture some aspects of deep mutational scanning experiments

2020

View full text Add to dashboard Cite

Background: Deep mutational scanning (DMS) studies exploit the mutational landscape of sequence variation by systematically and comprehensively assaying the effect of single amino acid variants (SAVs; also referred to as missense mutations, or non-synonymous Single Nucleotide Variantsmissense SNVs or nsSNVs) for particular proteins. We assembled SAV annotations from 22 different DMS experiments and normalized the effect scores to evaluate variant effect prediction methods. Three trained on traditional variant effect data (PolyPhen-2, SIFT, SNAP2), a regression method optimized on DMS data (Envision), and a naïve prediction using conservation information from homologs. Results: On a set of 32,981 SAVs, all methods captured some aspects of the experimental effect scores, albeit not the same. Traditional methods such as SNAP2 correlated slightly more with measurements and better classified binary states (effect or neutral). Envision appeared to better estimate the precise degree of effect. Most surprising was that the simple naïve conservation approach using PSI-BLAST in many cases outperformed other methods. All methods captured beneficial effects (gain-of-function) significantly worse than deleterious (loss-of-function). For the few proteins with multiple independent experimental measurements, experiments differed substantially, but agreed more with each other than with predictions. Conclusions: DMS provides a new powerful experimental means of understanding the dynamics of the protein sequence space. As always, promising new beginnings have to overcome challenges. While our results demonstrated that DMS will be crucial to improve variant effect prediction methods, data diversity hindered simplification and generalization.

show abstract

Section: Family Conservation Carries Most Important Signalmentioning

confidence: 78%

Section: Introductionmentioning

confidence: 99%

Variant effect predictions capture some aspects of deep mutational scanning experiments

2020

View full text Add to dashboard Cite

show abstract

“…Although the bioinformatic predictions were modestly correlated with our experimental measurements for variants in the functional validation set, they were markedly less concordant for other variants at those same residues, or throughout MSH2 at large ( Figure 4B). Similarly weak overall agreement has also been observed when benchmarking bioinformatic classifiers with deep mutational scans of other genes [61][62][63] . As variant effect predictors are often trained on the limited number of known variants with available classifications, their divergence with our experimental measurements may reflect overfitting, further suggesting that the comparison to a small set of functionally characterized alleles overestimates bioinformatic predictors' performance.…”

Section: Loss Of Function Scores Outperform Bioinformatic Predictorsmentioning

confidence: 82%

Massively parallel functional testing ofMSH2missense variants conferring Lynch Syndrome risk

Jia

Burugula

Chen

et al. 2020

Preprint

View full text Add to dashboard Cite

Introduction 3The lack of functional evidence for the majority of missense variants limits their clinical interpretability, 4 and poses a key barrier to the broad utility of carrier screening. In Lynch Syndrome (LS), one of the most 5 highly prevalent cancer syndromes, nearly 90% of clinically observed missense variants are deemed 6 'variants of uncertain significance' (VUS). To systematically resolve their functional status, we performed 7 a massively parallel screen in human cells to identify loss-of-function missense variants in the key DNA 8 mismatch repair factor MSH2. The resulting sequence-function map is substantially complete, covering 9 94% of the 17,746 possible variants, and is highly concordant (≥96%) with existing functional data and 10 expert clinicians' interpretations. The large majority (89%) of missense variants were functionally neutral, 11 perhaps unexpectedly in view of its evolutionary conservation. These data provide ready-to-use 12 functional evidence to resolve the ~1,300 extant missense VUSs in MSH2, and may facilitate the 13 prospective classification of newly discovered variants in the clinic. 14 15 Main text 16 17 Lynch Syndrome (LS, OMIM:120435), the first discovered familial cancer syndrome 1 , confers 18 predisposition to colorectal and endometrial cancers due to the loss of DNA mismatch repair (MMR) and 19 resulting mutagenic burden. Most affected individuals inherit a single loss-of-function variant in one of 20 four MMR factors (MSH2, MLH1, 21 MSH6, PMS2), followed by a somatic 22 'second hit' inactivating the remaining 23 functional copy. Due to high 24 population prevalence (≥1:300) 2,3 , 25 clear genetic etiology, and potential 26 for prevention through intensified 27 surveillance, MMR gene variants are 28 considered clinically actionable 4 . 29 30 Missense changes together comprise 31 20-30% of LS variants 5 , and are 32 particularly challenging to interpret: 33 their functional impacts may range 34 from minimal to profound, most are 35 individually rare, and they exhibit 36 incomplete penetrance 6 . 37 Consequently, the overwhelming 38 majority of LS gene missense variants 39 listed in ClinVar (4,762/5,473, 87.0%) 40 are deemed 'variants of uncertain 41 significance', or VUS 42( Supplementary Fig.1), and cannot 43 be used to guide diagnosis. 45To address these challenges, we 46 performed a deep mutational scan 7 to 47 systematically measure the functional 48 impact of missense variants in the 49 major Lynch Syndrome gene MSH2. 50We established a human cell system 51 to model MSH2 variant function using 52 the mismatch repair proficient 8 cell 53 line HAP1 (Fig. 1a,c). First, to disrupt MMR, we derived clonal MSH2 knockout cells bearing a 19-bp frameshift deletion in MSH2 exon 6, and 55 verified that it lacked detectable MSH2 protein expression ( Supplementary Fig. 2). To restore MMR, we 56 stably reintroduced into cells either wild-type MSH2 cDNA (KO+WT) or a pathogenic founder allele 57 (KO+A636P) on inducible lentiviral constructs. After inducing expression with doxycycl...

show abstract

“…Importantly, we excluded any predictors trained using supervised learning techniques, as well as meta-predictors that utilise the outputs of other predictors, thus including only predictors we labelled as unsupervised and empirical in our recent study 10 . This is due to the fact that predictors based upon supervised learning are likely to have been directly trained on some of the same mutations used in our evaluation dataset, making a fair comparison impossible 10,50 . A few predictors perform substantially better than FoldX, with the best performance seen for SIFT4G 51 , a modified version of the SIFT algorithm 52 .…”

Section: Resultsmentioning

confidence: 99%

“…Although different approaches vary in their implementation, a few types of information are most commonly used, including evolutionary conservation, changes in physiochemical properties of amino acids, biological function, known disease association and protein structure 7 . While these predictors are clearly useful for variant prioritisation, and show a statistically significant ability to distinguish known pathogenic from benign variants, they still make many incorrect predictions [8][9][10] , and the extent to which we can rely on them for diagnosis remains limited 11 .…”

Section: Introductionmentioning

confidence: 99%

Identification of pathogenic missense mutations using protein stability predictors

Gerasimavicius

Liu

Marsh

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Attempts at using protein structures to identify disease-causing mutations have been dominated by the idea that most pathogenic mutations are disruptive at a structural level. Therefore, computational stability predictors, which assess whether a mutation is likely to be stabilising or destabilising to protein structure, have been commonly used when evaluating new candidate disease variants, despite not having been developed specifically for this purpose. We therefore tested 12 different stability predictors for their ability to discriminate between pathogenic and putatively benign missense variants. We find that one method, FoldX, considerably outperforms all others in the identification of disease variants. Moreover, we demonstrate that employing absolute energy change scores improves performance of nearly all predictors. Importantly, however, we observe that the utility of computational stability predictors is highly heterogeneous across different proteins, and that they are all are inferior to the best performing variant effect predictors for identifying pathogenic mutations. We suggest that this is largely due to alternate molecular mechanisms other than protein destabilisation underlying many pathogenic mutations. Thus, better ways of incorporating protein structural information and molecular mechanisms into computational variant effect predictors will be required for improved disease variant prioritisation.

show abstract

Using deep mutational scanning to benchmark variant effect predictors and identify disease mutations

Cited by 51 publications

References 100 publications

Variant effect predictions capture some aspects of deep mutational scanning experiments

Variant effect predictions capture some aspects of deep mutational scanning experiments

Massively parallel functional testing ofMSH2missense variants conferring Lynch Syndrome risk

Identification of pathogenic missense mutations using protein stability predictors

Contact Info

Product

Resources

About