Assessing performance of pathogenicity predictors using clinically-relevant variant datasets

Abstract

15Purpose: Pathogenicity predictors are an integral part of genomic variant interpretation but, despite 16 their widespread usage, an independent validation of performance using a clinically-relevant dataset 17has not been undertaken. 18 19Methods: We derive two validation datasets: an "open" dataset containing variants extracted from 20 publicly-available databases, similar to those commonly applied in previous benchmarking exercises, 21 and a "clinically-representative" dataset containing variants identified through research/diagnostic 22 48 consistency to variant classification and have been followed by a number of regional and disorder-49 specific publications 2-4 . Common to all guidelines is the recommendation of the use of in silico 50 prediction tools to aid in the classification of missense variants. In silico prediction tools are 51 algorithms designed to predict the functional impact of variation, usually missense changes caused 52 by single nucleotide variants (SNVs). Though originally designed for the prioritisation of research 53 variants 5 , the tools are used routinely in clinical diagnostics during variant classification. The tools 54 integrate a number of features in order assess the impact of a variant on protein function 6 . Initially, 55inter-species conservation formed the bulk of the predictions, with some additional functional 56 information, such as substitution matrices of physicochemical distances of amino acids (such as 57Grantham 7 or PAM 8 ), and data derived from a limited number of available X-ray crystallographic 58 structures 9 . Since the development of the first in silico prediction tools over a decade ago 5,9 , large-59 scale experiments such as the ENCODE project 10 have generated huge amounts of functional data, 60and we now also have access to large-scale databases of clinical and neutral variation [11][12][13] . These 61 additional sources of data have led to an explosion of new in silico prediction algorithms 14-16 that 62 purport to increase accuracy. 63 64However, the large increase in the number of predictors integrated into classification algorithms has 65 raised concerns about overfitting 17,18 . Overfitting occurs when the prediction algorithm is trained on 66 superfluous data or features that are irrelevant to the prediction outcome 18 . While it may appear 67that an increasingly large feature list leads to improvements in prediction, random variability within 68 the training dataset may actually result in decreased accuracy when applied to a novel dataset. 69Overfitting can be mitigated through the use of increasingly large training datasets, and the usage of 70 online variant databases, such as the genome aggregation database (gnomAD) 19 and ClinVar 12 , 71 allows for sufficiently large training datasets. Additionally, reliance on additional information -such 72 as protein functional data and allele frequency data such as from gnomAD 19 -may be contrary to the 73 standard assumptions of variant classification methodology, namely that each dataset is 74 inde...

Refining Genotypes and Phenotypes in KCNA2-Related Neurological Disorders

“…To evaluate the likelihood of pathogenic variant effects, we combined the distribution of known pathogenic and benign variants with computational pathogenicity scores for all biologically possible missense variants of KCNA2 . The REVEL meta-score combines variant annotations of 13 individual pathogenicity scores and showed high overall performance in the discrimination of pathogenic and benign variants in large clinical datasets [ 42 , 43 ]. High REVEL scores overlap with areas of pathogenic variant enrichment in KCNA2 ( Figure 3 ).…”

Association analysis of risk genes identified by SCHEMA with schizophrenia in the Chinese Han population

“…The output of the analysis was a numerical score between 0 and 1, pathogenicity threshold for SIFT is ≤0.05, PloyPhen-2 ≥0.909 is probably damaging, and 0.447–0.908 is possibly damaging. Variants with SIFT prediction scores more than 0.05 (Guryev et al , 2004; Livingston et al , 2004; Gunning et al , 2021) and PolyPhen-2 prediction scores less than 0.447 (Guryev et al , 2004; Livingston et al , 2004; Gunning et al , 2021) were removed. Association analysis was performed on the SHEsisPlus online software platform (http://shesisplus.bio-x.cn/) (Shi and He, 2005).…”

Rare variant association testing in the non-coding genome

“…Differences were also observed depending on the type of functional elements: DeepSEA, GWAVA and LINSIGHT performed better with variants in promoters and CADD with intronic variants. In another study (Gunning et al 2020), all scores were found to perform worse when, in the testing set, pathogenic variants were selected from diagnostic panels rather than among variants annotated as pathogenic in public databases. This could probably be explained by the fact that it is from these latter databases that most of the methods choose their training set.…”

“…This is the method recommended in the ACMG guidelines (Richards et al 2015) but without any precision on which scores should be used and how many of these scores should be concordant. However, this simple method was shown to perform much worse than methods that integrate different scores into a single model (Gunning et al 2020). Several integrating methods have been proposed that use different scores and different models to combine them.…”