Reproducible Drug Repurposing: When Similarity Does Not Suffice

Abstract

Repurposing existing drugs for new uses has attracted considerable attention over the past years. To identify potential candidates that could be repositioned for a new indication, many studies make use of chemical, target, and side effect similarity between drugs to train classifiers. Despite promising prediction accuracies of these supervised computational models, their use in practice, such as for rare diseases, is hindered by the assumption that there are already known and similar drugs for a given condition of interest. In this study, using publicly available data sets, we question the prediction accuracies of supervised approaches based on drug similarity when the drugs in the training and the test set are completely disjoint. We first build a Python platform to generate reproducible similaritybased drug repurposing models. Next, we show that, while a simple chemical, target, and side effect similarity based machine learning method can achieve good performance on the benchmark data set, the prediction performance drops sharply when the drugs in the folds of the cross validation are not overlapping and the similarity information within the training and test sets are used independently. These intriguing results suggest revisiting the assumptions underlying the validation scenarios of similarity-based methods and underline the need for unsupervised approaches to identify novel drug uses inside the unexplored pharmacological space. We make the digital notebook containing the Python code to replicate our analysis that involves the drug repurposing platform based on machine learning models and the proposed disjoint cross fold generation method freely available at github. com/emreg00/repurpose.

Revisiting Cross-Validation of Drug Similarity Based Classifiers Using Paired Data

“…In light of previous works highlighting the perils of cross-validation using paired data [6,7], we recently investigated the effect of using drug-wise disjoint cross-validation in predicting drug-disease pairs, where none of the drugs in the training set appeared in the test set [8]. We showed that the prediction accuracy of the classifier drops dramatically under such cross-validation setting, suggesting that the existing approaches are prone to over-fitting due to the inherent relationships in the data.…”

“…Owing to the larger number of known drug-drug interactions, compared to the number of known drug-disease associations used in our previous study, we explore the effect of sample size in the data set. We use the code and data provided within Repurpose framework [8] and train a logistic regression classifier to predict DDIs using drug chemical, target and side effect similarity calculated via a k-nearest-neighbor approach (k = 20, see [8] for details).…”

“…The proposed models are typically benchmarked using cross-validation, in which the known drug-disease or drug-drug associations are split into training and test sets. Though these methods report areas under receiver operating characteristic (ROC) curves around 90% under cross-validation, their applicability in translational medicine and, thus, ability to reduce drug development costs has been controversial [2,4,5].In light of previous works highlighting the perils of cross-validation using paired data [6,7], we recently investigated the effect of using drug-wise disjoint cross-validation in predicting drug-disease pairs, where none of the drugs in the training set appeared in the test set [8]. We showed that the prediction accuracy of the classifier drops dramatically under such cross-validation setting, suggesting that the existing approaches are prone to over-fitting due to the inherent relationships in the data.…”

“…Owing to the larger number of known drug-drug interactions, compared to the number of known drug-disease associations used in our previous study, we explore the effect of sample size in the data set. We use the code and data provided within Repurpose framework [8] and train a logistic regression classifier to predict DDIs using drug chemical, target and side effect similarity calculated via a k-nearest-neighbor approach (k = 20, see [8] for details).In addition to the drug chemical, target and side effect information available at Repurpose, we retrieve drug-drug interactions from DrugBank [9] (v4.5.0), corresponding to 26,484 known drug-drug interactions for 536 drugs in the data set. Next, we test the prediction accuracy of the classifier on truly novel cases, that is, drugs it has never seen before under a ten fold cross-validation scheme.…”

Methods to Ensure the Reproducibility of Biomedical Research

“…Hundreds of studies have used publicly available data to predict adverse drug reactions and drug indications and have reported seemingly exceptional predictive accuracy: Guney 11 investigates the issue of performance overestimation for drug side effect and indication, and finds that major assumption of these methods (independence) is violated, which overestimates their performance. Haynes et al 12 present a pipeline for expression meta-analysis, which fills an unmet need for systematic processing and visualization of results from such analyses.…”