Revisiting Cross-Validation of Drug Similarity Based Classifiers Using Paired Data

Abstract

SUMMARYFollowing the recent availability of high-throughput data for drug discovery, computational methods, especially machine learning based approaches, have gained remarkable attention.A number of studies use chemical, target and side effect similarity between drugs to build knowledge-based models that predict drug indications and drug-drug interactions. In light of previous works demonstrating the perils of cross-validation using paired data, in this study, we employ a disjoint cross validation approach for similarity-based drug-drug interaction (DDI) prediction and we investigate the prediction accuracy of classifier under various settings.Our results point to the dependence on the cross validation strategy used to evaluate prediction accuracy of drug similarity-based classifiers operating on paired data such as pharmacokinetic interactions between drugs. KEYWORDSRational drug design; Drug-drug interaction; Paired data; Disjoint cross-validation; K-nearest-neighbor; Logistic regression. AVAILABILITY AND REQUIREMENTSThe Jupyter Notebook, named interaction.ipynb containing the code used in this analysis is available in Repurpose framework (github.com/emreg00/ repurpose). BODYFollowing the recent availability of high-throughput data for drug discovery, computational methods, especially machine learning based approaches, have gained remarkable attention. A number of studies use chemical, target and side effect similarity between drugs to build knowledge-based models that predict drug indications and drug-drug interactions [1][2][3]. The proposed models are typically benchmarked using cross-validation, in which the known drug-disease or drug-drug associations are split into training and test sets. Though these methods report areas under receiver operating characteristic (ROC) curves around 90% under cross-validation, their applicability in translational medicine and, thus, ability to reduce drug development costs has been controversial [2,4,5].In light of previous works highlighting the perils of cross-validation using paired data [6,7], we recently investigated the effect of using drug-wise disjoint cross-validation in predicting drug-disease pairs, where none of the drugs in the training set appeared in the test set [8]. We showed that the prediction accuracy of the classifier drops dramatically under such cross-validation setting, suggesting that the existing approaches are prone to over-fitting due to the inherent relationships in the data.Here, we turn our attention to disjoint cross-validation of similarity-based drug-drug interaction (DDI) prediction (Figure 1). Owing to the larger number of known drug-drug interactions, compared to the number of known drug-disease associations used in our previous study, we explore the effect of sample size in the data set. We use the code and data provided within Repurpose framework [8] and train a logistic regression classifier to predict DDIs using drug chemical, target and side effect similarity calculated via a k-nearest-neighbor approach (k = 20, see [8] for details...

Evaluation of knowledge graph embedding approaches for drug-drug interaction prediction in realistic settings

“…However, when samples are in the form of a pair of objects, the traditional CV leads to optimistic results due to the presence of the same objects in both the training set and the test set [27]. To make realistic evaluation of DDI prediction task, we propose two scenarios similar to what Park [27] and Guney [28] suggested for the paired-input methods: (i) drug-wise disjoint CV and (ii) pairwise disjoint CV. To create these scenarios, the drugs that form the drug pairs are split into 2 clusters: cold-start and existing drugs.…”

“…The traditional CV where test pairs might share components with training pairs is prone to over-fitting due to systematic biases in networked/paired data [26–29]. There have been a few studies which have addressed this issue [27, 28, 30, 31]. Some studies [30, 31] demonstrated how well their methods perform to make predictions for new drugs which lacked interaction data.…”

“…However, the setting is rather complex and the failure to provide an algorithm or code make it challenging to reproduce the setting. Guney also suggested similar cross-validation settings for DDI prediction: non-disjoint, disjoint and pairwise disjoint CV [28, 32]. Non-disjoint CV is the same as the traditional CV, while disjoint and pairwise disjoint CV are similar to C2 and C3 scenarios, respectively.…”

A Knowledge-Graph-Based Multimodal Deep Learning Framework for Identifying Drug–Drug Interactions

“…In spite of the fact that the 10-fold CV can avoid the model suffering from overfitting in some cases, it may also produce overly optimistic results [ 58 ]. To make a realistic evaluation of the DDI prediction task, KGCN_NFM, and other baseline models were further evaluated on realistic scenarios proposed by pairwise disjoint CV (PW-CV) [ 59 ] and compared the performance with traditional CV. The PW-CV is a cold-start scenario designed to verify the ability to handle unseen nodes, where a drug in a drug pair contained in the test set is inaccessible in the training set.…”

Protein Interaction Networks and Disease: Highlights of the 3rd Challenges in Computational Biology Meeting

“…He also discussed proper ways to evaluate drug-drug interaction predictions (see his extended abstract in this special issue [6]). …”