Quantification of biases in predictions of protein stability changes upon mutations

Abstract

The article 10.1093/bioinformatics/bty340/, published alongside this paper, also addresses the problem of biases in protein stability change predictions.

Limitations and challenges in protein stability prediction upon genome variations: towards future applications in precision medicine

“…CV = cross-validation, LOO = leave-one-out, TS = test-set, MCC = Matthews Correlation Coefficient, ρ = correlation, ρ_dir-inv = correlation between direct and inverse variation. *Evaluated on Ssym database [11] .…”

“…Another important feature described above, which has a great impact on the algorithm performance, is represented by the anti-symmetric properties of the free energy changes. Recently, different studies addressed this problem as bias in most predictors [11] , [12] , [48] , [61] , [65] , and specifically designed datasets including both the variations and the corresponding inverses (such as A->B and B->A on the same protein) have been introduced: Ssym [11] , Usmanova-DB [48] and Fang-DB [12] . The results show that only INPS and PopMusic [11] were sufficiently robust to be defined compliant with the anti-symmetric properties [61] , [65] .…”

“…In this work, we focus on the methods and issues related to the robust prediction of ΔΔG values, providing a comprehensive evaluation of both the limits of current approaches and some possible solutions to achieve better predictions by minimizing the bias in the performance assessment. Recent critical reviews were already provided by Pucci et al [11] and Fang [12] . Here we will extend those reviews by introducing some further crucial points, discussing: 1) the unbalance and intrinsic variability of the thermodynamic data; 2) the failure of taking into account the anti-symmetric property of the folding process (ΔΔG u = −ΔΔG f ); 3) the best practices for providing a fair assessment of prediction performance.…”

Predicting protein stability and solubility changes upon mutations: data perspective

“…[58,59] Augmenting the datasets with reverse mutations with opposite signs of ΔΔG or ΔT m has also gained attention recently to promote the so-called anti-symmetry of predictors: reverse mutations should produce the same predictions but with opposite signs, which turns out not to be the case for many predictors. [31,32] To provide the community with additional high quality data, we have manually processed the data from ProTherm as well as new data from literature and are depositing them to our database FireProt DB , where they can be accessed via a user-friendly graphical interface ( Figure 2). We expect to release the databased in the next few months, and its landing page can be found at loschmidt.chemi.muni.cz/fireprotdb.…”

“…However, they all seem to be testing a similar limit to the prediction accuracy, e. g. the root mean square error of around 1 kcal/mol for stability predictions [1] . Moreover, independent experimental validation in subsequent studies often reveals modest performance [28,31–33] . Several explanations can be provided, one of which is the limited data size and quality available for training: data quality and abundance are critical for ML algorithms as they ultimately aim to identify and generalize patterns in the training data.…”