Performance of Protein Disorder Prediction Programs on Amino Acid Substitutions

Abstract

Many proteins contain intrinsically disordered regions, which may be crucial for function, but on the other hand be related to the pathogenicity of variants. Prediction programs have been developed to detect disordered regions from sequences and used to predict the consequences of variants, although, their performance for this task has not been assessed. We tested the performance of protein disorder prediction programs in detecting changes to disorder caused by amino acid substitutions. We assessed the quality of 29 protein disorder predictors and versions with 101 amino acid substitutions, whose effects have been experimentally validated. Disorder predictors detected the true positives at most with 6% success rate and true negatives with 34% rate for variants. The corresponding rates for the wild type forms are 7 and 90%. The analysis revealed that disorder programs cannot reliably predict the effects of substitutions, consequently the tested methods, and possibly similar programs, cannot be recommended for variant analysis without other information indicating to the relevance of disorder. These results inspired us to develop a new method, PON-Diso (http://structure.bmc.lu.se/PON-Diso), for disorder related amino acid substitutions. With 50 % success rate for independent test set and 70.5% rate in cross validation it outperforms the evaluated methods.

Variation Benchmark Datasets: Update, Criteria, Quality and Applications

“…Benchmark studies in relation to variation predictions have been made for variants affecting protein stability [7,8], protein substitution tolerance/pathogenicity [9][10][11][12][13][14], protein localization [15], protein disorder [16], protein solubility [17], benign variants [18], transmembrane proteins [19], alternative splicing [20,21] and phenotypes of amino acid substitutions [22].…”

PON‐P and PON‐P2 predictor performance in CAGI challenges: Lessons learned

“…We and others have assessed the performance of methods for predicting impact of variants on tolerance/pathogenicity (Bendl et al., ; Grimm et al., ; König, Rainer, & Domingues, ; Niroula et al., ; Thusberg et al., ), protein localization (Laurila & Vihinen, ), protein stability (Khan & Vihinen, ; Potapov, Cohen, & Schreiber, ), RNA splicing (Jian, Boerwinkle, & Liu, ), protein disorder (Ali, Urolagin, Gurarslan, & Vihinen, ), and solubility (Yang et al., ). The performance assessments showed vast differences between methods and indicated a need for improved tools.…”

“…When applied outside the primary application area, problems may occur. We have shown that generic protein disorder predictors are not suitable for analysis of the effects of AASs on protein disorder (Ali et al., ) or generic protein solubility methods for AASs affecting protein solubility (Yang et al., ). The CAGI challenges allowed us to test our methods even outside the intended application area to learn about possible applications of methods and to find the best approaches for optimal application.…”

Majority Vote and Other Problems when using Computational Tools

“…A number of studies utilizing benchmarks to assess the performance of, for example, tolerance predictors to identify harmful variations [Thusberg et al., ; Bendl et al., ], protein stability [Potapov et al., ; Khan and Vihinen, ], protein disorder [Ali et al., ], and mRNA alternative splicing [Vořechovský, ; Buratti et al., ; Desmet et al., ] have been published. Method users should consult these and related studies when choosing methods instead of relying to the one used last time.…”