Assessing performance of pathogenicity predictors using clinically relevant variant datasets

Abstract

BackgroundPathogenicity predictors are integral to genomic variant interpretation but, despite their widespread usage, an independent validation of performance using a clinically relevant dataset has not been undertaken.MethodsWe derive two validation datasets: an ‘open’ dataset containing variants extracted from publicly available databases, similar to those commonly applied in previous benchmarking exercises, and a ‘clinically representative’ dataset containing variants identified through research/diagnostic exome and panel sequencing. Using these datasets, we evaluate the performance of three recent meta-predictors, REVEL, GAVIN and ClinPred, and compare their performance against two commonly used in silico tools, SIFT and PolyPhen-2.ResultsAlthough the newer meta-predictors outperform the older tools, the performance of all pathogenicity predictors is substantially lower in the clinically representative dataset. Using our clinically relevant dataset, REVEL performed best with an area under the receiver operating characteristic curve of 0.82. Using a concordance-based approach based on a consensus of multiple tools reduces the performance due to both discordance between tools and false concordance where tools make common misclassification. Analysis of tool feature usage may give an insight into the tool performance and misclassification.ConclusionOur results support the adoption of meta-predictors over traditional in silico tools, but do not support a consensus-based approach as in current practice.

Quantifying prediction of pathogenicity for within-codon concordance (PM5) using 7541 functional classifications of BRCA1 and MSH2 missense variants

“…Notably, in this context the tool is used to compare relative deleteriousness between colocated variants, rather than being used with a prespecified binary threshold of pathogenicity, which is more typical in other tool evaluations. 22,27 The FP rate is generally low for all PM5-definitions: 3.2% (244/7541) for the most lenient definition of PM5 (PM5-definition_a) and <1% for more stringent PM5definitions. The low FP rate drives high specificity, positive predictive value, and pLRs for calling of pathogenicity.…”

A Study on the Genetics of Primary Ciliary Dyskinesia

“…No individual or meta predictor is able to clearly predict the pathogenicity of all clinically relevant variants [ 34 ]. MDS is a statistical approach adopted in the interpretation of datasets involving several variables (ttp:// ; accessed on 25 October 2021).…”

Artificial intelligence challenges for predicting the impact of mutations on protein stability

“…In rational protein engineering applications, for example, the targeted redesign of proteins makes it possible to optimize the biotechnological and biopharmaceutical processes in which they are involved [1,2]. Stability prediction also plays a key role in interpreting the impact of human genetic variants and may provide a better understanding of how these variants lead to disease conditions [3,4]. Note that stability is all the more important as it is the dominant factor in protein fitness [5].…”