Integrated Random Negative Sampling and Uncertainty Sampling in Active Learning Improve Clinical Drug Safety Drug–Drug Interaction Information Retrieval

Abstract

Clinical drug–drug interactions (DDIs) have been a major cause for not only medical error but also adverse drug events (ADEs). The published literature on DDI clinical toxicity continues to grow significantly, and high-performance DDI information retrieval (IR) text mining methods are in high demand. The effectiveness of IR and its machine learning (ML) algorithm depends on the availability of a large amount of training and validation data that have been manually reviewed and annotated. In this study, we investigated how active learning (AL) might improve ML performance in clinical safety DDI IR analysis. We recognized that a direct application of AL would not address several primary challenges in DDI IR from the literature. For instance, the vast majority of abstracts in PubMed will be negative, existing positive and negative labeled samples do not represent the general sample distributions, and potentially biased samples may arise during uncertainty sampling in an AL algorithm. Therefore, we developed several novel sampling and ML schemes to improve AL performance in DDI IR analysis. In particular, random negative sampling was added as a part of AL since it has no expanse in the manual data label. We also used two ML algorithms in an AL process to differentiate random negative samples from manually labeled negative samples, and updated both the training and validation samples during the AL process to avoid or reduce biased sampling. Two supervised ML algorithms, support vector machine (SVM) and logistic regression (LR), were used to investigate the consistency of our proposed AL algorithm. Because the ultimate goal of clinical safety DDI IR is to retrieve all DDI toxicity–relevant abstracts, a recall rate of 0.99 was set in developing the AL methods. When we used our newly proposed AL method with SVM, the precision in differentiating the positive samples from manually labeled negative samples improved from 0.45 in the first round to 0.83 in the second round, and the precision in differentiating the positive samples from random negative samples improved from 0.70 to 0.82 in the first and second rounds, respectively. When our proposed AL method was used with LR, the improvements in precision followed a similar trend. However, the other AL algorithms tested did not show improved precision largely because of biased samples caused by the uncertainty sampling or differences between training and validation data sets.

Novel Method for Early Prediction of Clinically Significant Drug–Drug Interactions with a Machine Learning Algorithm Based on Risk Matrix Analysis in the NICU

“…Most of the alerts generated by the legacy CDSS were related to DDIs and dosages [ 26 ]. Although there are theoretical and review ML studies on DDI extraction from the biomedical literature [ 27 ], DrugBank and other databases [ 28 , 29 ], bioinformatics algorithms to predict DDI [ 30 ], and clinical safety DDI information retrieval [ 31 ], there are no real-life studies that reflect clinical practice in neonates.…”

Multiple sampling schemes and deep learning improve active learning performance in drug-drug interaction information retrieval analysis from the literature

“… Random negative sampling Because more than 99% of unscreened pool abstracts are not DDl related, a random subset of unscreened pool is chosen as negative samples. These random negative samples may contain very small fraction of positive samples [ 33 ]. Similarity sampling aims to quick screen out samples that more like samples in corpus , the cosine similarity (cosSIM) based on TF-IDF (Term Frequency-Inverse Document Frequency) [ 34 ] of each unlabeled sample and all the samples in corpus is used to evaluated.…”

“…Random negative sampling Because more than 99% of unscreened pool abstracts are not DDl related, a random subset of unscreened pool is chosen as negative samples. These random negative samples may contain very small fraction of positive samples [ 33 ].…”

On the road to explainable AI in drug-drug interactions prediction: A systematic review

“…Moreover, the imbalanced class distribution problem has also been considered in many articles since this issue can diminish the power of classification [39] , [40] . Liu et al used several rules to filter negative instances [41] ; others added random negative sampling as part of the active learning algorithm to deal with the imbalanced issue [42] or use focal loss function to mitigate against this problem [43] .…”

“…In the meantime, logistic regression (LR) algorithm has been less used to establish DDIs prediction model. Xie et al [91] integrated active learning, random negative sampling, and uncertainty sampling in clinical safety DDI information retrieval (DDI-IR) analysis using SVM and LR. In addition, Drug-Entity-Topic (DET) model following Bayes-rules was an example in leveraging augmented text-mining features to improve prediction performance in terms of discrimination and calibration [73] .…”