Reply to “Missed opportunities in large scale comparison of QSAR and conformal prediction methods and their applications in drug discovery”

Abstract

In response to Krstajic’s letter to the editor concerning our published paper, we here take the opportunity to reply, to re-iterate that no errors in our work were identified, to provide further details, and to re-emphasise the outputs of our study. Moreover, we highlight that all of the data are freely available for the wider scientific community (including the aforementioned correspondent) to undertake follow-on studies and comparisons.

Critical Assessment of Conformal Prediction Methods Applied in Binary Classification Settings

“…Damjan Krstajic [16] has published a criticism of the approach presented by Bosc et al [15]. Here we will only repeat the main point of his criticism which Bosc et al [21] omitted to comment in their reply. Half of the comparisons between CP and QSAR presented in Bosc et al [15] examine situations when predictions assigned to 'both' are considered correctly classified.…”

Diversifying cheminformatics

“…One way to do so is to invite submissions that report on benchmarking studies. Recently, there has been some discussion around defining standards to enable rigorous comparisons of this kind [ 7 , 8 ] , which of course also includes the discussion about the appropriate statistical tests for the use cases at hand. As we believe those are important discussions to have community-wide in order to bring our field forward and make it fit for the next generation of ML data scientists, we will foster initiatives in these directions in the near future.…”

Comparison of Scaling Methods to Obtain Calibrated Probabilities of Activity for Protein–Ligand Predictions

“… The probability estimates outputted by the approach required generation of local (per-target) likelihood score thresholds, to convert biased probability estimates into predicted labels, and optimal thresholds were shown to behave differently for particular target families because of the aforementioned biases . Other work showed that the probability estimates for negative predictions (i.e., the “non-binding” or “functionally inactive” compound–target predictions) are also influenced by the origin of the negative training set (putative or experimental annotations , ), the proportion of compound–target annotations labeled at protein complexes (∼6% of ChEMBL data) and the degree of imbalance toward the negative (majority) class − [i.e., studies often train on ∼100× more negatively labeled compounds than actives (pXC 50 values larger than 5) . The chemical space of training data is also biased, with often 10–30 compound exemplars per scaffold, but also in some cases very highly populated scaffolds and high numbers of singleton scaffolds being present .…”