Lessons from the CAGI‐4 Hopkins clinical panel challenge

Abstract

The CAGI-4 Hopkins clinical panel challenge was an attempt to assess state-of-the-art methods for clinical phenotype prediction from DNA sequence. Participants were provided with exonic sequences of 83 genes for 106 patients from the Johns Hopkins DNA Diagnostic Laboratory. Five groups participated in the challenge, predicting both the probability that each patient had each of the 14 possible classes of disease, as well as one or more causal variants. In cases where the Hopkins laboratory reported a variant, at least one predictor correctly identified the disease class in 36 of the 43 patients (84%). Even in cases where the Hopkins laboratory did not find a variant, at least one predictor correctly identified the class in 39 of the 63 patients (62%). Each prediction group correctly diagnosed at least one patient that was not successfully diagnosed by any other group. We discuss the causal variant predictions by different groups and their implications for further development of methods to assess variants of unknown significance. Our results suggest that clinically relevant variants may be missed when physicians order small panels targeted on a specific phenotype. We also quantify the false-positive rate of DNA-guided analysis in the absence of prior phenotypic indication.

Assessment of patient clinical descriptions and pathogenic variants from gene panel sequences in the CAGI‐5 intellectual disability challenge

“…In addition, Figure and Table S3 show the ROC curves and performance measures, respectively, for all submissions in each phenotype. Since the overall predictions are far from a perfect performance, the prediction assessment for each phenotypic trait was performed also in the group of patients where the Padua NDD lab noted a potentially causative variants like previous CAGI challenge assessments (data not shown; Chandonia et al, ). However, this did not show any improvement of predictor performance.…”

“…We have described the assessment of the CAGI‐5 ID challenge. This challenge is based on the phenotype evaluation of patients using gene panel sequences, in analogy to the CAGI‐4 Hopkins panel challenge (Chandonia et al, ). Where the Hopkins panel was testing for different monogenic diseases with Mendelian inheritance, the ID challenge focuses on complex disorders.…”

“…This background fits well into the framework of the Critical Assessment of Genome Interpretation (CAGI) experiment, which has a declared goal of assessing methods to help interpret the effects of variants of unknown significance. A similar challenge was present in the CAGI‐4 experiment with the Hopkins gene panel, where predictors were asked to predict phenotypes based on the results of a genetic screening performed on a set of 83 genes associated to 14 different conditions (Chandonia et al, ).…”

Analysis and Interpretation of the Impact of Missense Variants in Cancer

“…In 2019, we proposed a new computing experiment, focusing on the prediction of the variation of the free energy change induced by hFXN missense variants [ 85 ]. The assessment of the predictions submitted for the hFXN, and other challenges focusing on clinical applications, confirmed that the state-of-the-art methods for predicting the variation of protein stability change upon mutation are achieving a good level of performance, while methods for predicting pathogenic variants reached a good level of performance on challenges focusing on possible clinical applications [ 134 , 135 , 136 ]. The global effects of missense mutations on proteins can be highlighted from the analysis of the thermodynamic parameters of stability, distinguishing neutral mutations from destabilizing or stabilizing ones.…”

Reports from CAGI: The Critical Assessment of Genome Interpretation

“…At the level of full exome and genome sequence, there are challenges that assess methods for assigning complex traits phenotypes and that evaluate the ability to associate genome sequence and an extensive profile of phenotypic traits (including Cai et al., 2017; Daneshjou et al., ; Daneshjou et al., ; Giollo et al., ; Laksshman, Bhat, Viswanath, & Li, ; Pal, Kundu, Yin, & Moult, ; Wang et al., ). CAGI has also included challenges in which participants were asked to identify causative variants for rare diseases in gene panel, exome, and whole‐genome sequence data (including Chandonia et al., ; Kundu, Pal, Yin, & Moult, ; Pal, Kundu, Yin, & Moult, ). Many challenges have focused on cancer, given its prevalence and the impact of genetics.…”