Comparative Assessment of Intrinsic Disorder Predictions with a Focus on Protein and Nucleic Acid-Binding Proteins

Abstract

With over 60 disorder predictors, users need help navigating the predictor selection task. We review 28 surveys of disorder predictors, showing that only 11 include assessment of predictive performance. We identify and address a few drawbacks of these past surveys. To this end, we release a novel benchmark dataset with reduced similarity to the training sets of the considered predictors. We use this dataset to perform a first-of-its-kind comparative analysis that targets two large functional families of disordered proteins that interact with proteins and with nucleic acids. We show that limiting sequence similarity between the benchmark and the training datasets has a substantial impact on predictive performance. We also demonstrate that predictive quality is sensitive to the use of the well-annotated order and inclusion of the fully structured proteins in the benchmark datasets, both of which should be considered in future assessments. We identify three predictors that provide favorable results using the new benchmark set. While we find that VSL2B offers the most accurate and robust results overall, ESpritz-DisProt and SPOT-Disorder perform particularly well for disordered proteins. Moreover, we find that predictions for the disordered protein-binding proteins suffer low predictive quality compared to generic disordered proteins and the disordered nucleic acids-binding proteins. This can be explained by the high disorder content of the disordered protein-binding proteins, which makes it difficult for the current methods to accurately identify ordered regions in these proteins. This finding motivates the development of a new generation of methods that would target these difficult-to-predict disordered proteins. We also discuss resources that support users in collecting and identifying high-quality disorder predictions.

Deep learning in prediction of intrinsic disorder in proteins

“…This list contextualizes the efforts to develop deep learning predictors in a broader setting of the entire disorder prediction field. We identify the 36 predictors using a wide-ranging list of sources including databases of disorder predictions: MobiDB [122] , D 2 P 2 [123] and DescribePROT [124] ; community assessments and surveys that were published on or after 2013 [33] , [41] , [42] , [43] , [46] , [47] , [49] , [50] , [52] , [58] , [59] , and a manual search of relevant articles from PubMed that we collect using the “(disorder[Title]) AND (prediction[Title]) AND protein” query. Table 1 reveals that 13 out of the 36 recent disorder predictors use deep learning models.…”

“…Current surveys point to the long history of the disorder prediction area, providing invaluable insights concerning architectures of these methods, their availability, trends in their development efforts and approaches to comparatively evaluate their predictive performance [40] , [41] , [42] , [43] , [44] , [45] , [46] , [47] , [48] . Moreover, users and developers benefit from empirical studies that comparatively assess predictive quality of disorder predictors [33] , [49] , [50] , [51] , [52] , [53] , [54] , [55] , [56] , [57] , [58] , [59] . These comparative studies include several community assessments, such as Critical Assessment of Structure Prediction (CASP) between CASP5 to CASP10 [53] , [54] , [55] , [56] , [57] , [58] and Critical Assessment of Intrinsic Protein Disorder (CAID) [52] .…”

Capturing a Crucial ‘Disorder-to-Order Transition’ at the Heart of the Coronavirus Molecular Pathology—Triggered by Highly Persistent, Interchangeable Salt-Bridges

“…There are several AI (artificial intelligence)-trained sequence-based disorder predictors [105][106][107] that return residue-wise disorder probability scores, which are trained primarily on evolutionary sequence data (e.g., mutational co-variance matrices). These sequence-based disorder predictors have their known limits in accuracy [108,109], for not explicitly accounting for the actual three-dimensional structural dynamics of the protein(s)/peptide(s), but, can serve as a good first test of the comparative FLCS Spike loopdisorder among its close evolutionary homologs. A representative set of Spike structures (CoV/CoV-2) were culled (resolution ≤ 3 Å), accumulated (see Section 2.1, Materials and Methods), and their UNIPROT sequences (in FASTA format) derived from proteomics data [67], were extracted from corresponding entries in the Protein Data Bank [43].…”

Capturing a crucial ‘disorder-to-order transition’ at the heart of the coronavirus molecular pathology – triggered by highly persistent, interchangeable salt-bridges

“…There are several AI 6 -trained sequence based disorder predictors [101][102][103] that return residue-wise disorder probability scores, which are trained primarily on evolutionary sequence data (e.g., mutational co-variance matrices). These sequence-based disorder predictors have their known limits in accuracy [104,105], for not explicitly accounting for the actual three-dimensional structural dynamics of the protein(s) / peptide(s), but, can serve as a good first test of the comparative FLCS Spike loop-disorder among its close evolutionary homologs. A representative set of Spike structures (CoV/CoV-2) were culled (resolution ≤ 3 Å), accumulated (see subsection 2.1, Materials and Methods), and their UNIPROT sequences (in FASTA format) derived from proteomics data [63], were extracted from corresponding entries in the Protein Data Bank [39].…”