Filter Drug-induced Liver Injury Literature with Natural Language Processing and Ensemble Learning

Abstract

Drug-induced liver injury (DILI) describes the adverse effects of drugs that damage liver. Life-threatening results including liver failure or death were also reported in severe DILI cases. Therefore, DILI-related events are strictly monitored for all approved drugs and the liver toxicity became important assessments for new drug candidates. These DILI-related reports are documented in hospital records, in clinical trial results, and also in research papers that contain preliminary in vitro and in vivo experiments. Conventionally, data extraction from previous publications relies heavily on resource-demanding manual labelling, which considerably decreased the efficiency of the information extraction process. The recent development of artificial intelligence, particularly, the rise of natural language processing (NLP) techniques, enabled the automatic processing of biomedical texts. In this study, based on around 28,000 papers (titles and abstracts) provided by the Critical Assessment of Massive Data Analysis (CAMDA) challenge, we benchmarked model performances on filtering out DILI literature. Among four word vectorization techniques, the model using term frequencyinverse document frequency (TF-IDF) and logistic regression outperformed others with an accuracy of 0.957 with our inhouse test set. Furthermore, an ensemble model with similar overall performances was implemented and was fine-tuned to lower the false-negative cases to avoid neglecting potential DILI reports. The ensemble model achieved a high accuracy of 0.954 and an F1 score of 0.955 in the hold-out validation data provided by the CAMDA committee. Moreover, important words in positive/negative predictions were identified via model interpretation. Overall, the ensemble model reached satisfactory classification results, which can be further used by researchers to rapidly filter DILI-related literature.

A Tri-light Warning System for Hospitalized COVID-19 Patients: Credibility-based Risk Stratification under Data Shift

“…Tri-light Warning System for Hospitalized COVID-19 Patients 11 1) Convert the predicted probability to a nonconformity measurement: a metric to quantify how well a particular feature-label combination conforms to the training data. Here, we leveraged a design of the nonconformity measurement α i that has been validated in multiple machine learning applications [34,44]:…”

“…4) The credibility of the prediction can be computed as the larger P-value [34], which reflects how well the most likely label conform to the distribution of the training data nonconformity measurement. If the credibility is low, the credibility we have for the prediction is low, which may provide us with a flexible tool to tune the policies in medical resource management.…”

“…However, in AI-based applications, the reliability of predictions is significant for assisted decision and risk control in real-world applications: unreliable prediction can interfere with the clinicians' decision and may lead to misdiagnosis put on huge pressure on the patient families. Therefore, knowing how much confidence is associated with a prediction made by the model is important for the decision making process for the clinicians [34]. PROBAST analysis indicates that the majority of proposed models above are at a high risk of bias, and their reported performance is probably optimistic [26] [35].…”

Comparative analysis of classification techniques for topic-based biomedical literature categorisation

“…The results showed that TF-IDF dominates other approaches with respect to the AUROC and AURPC values, suggesting that simple and interpretable approaches could and do, in fact, outperform more sophisticated solutions. In a similar research (Zhan et al, 2022) dedicated specifically to DILI, authors leverage the same text processing techniques like BOW, W2V, and TF-IDF, while instead of D2V, Sentence2Vec is used. In addition, the paper includes the use of Random Forests for particular classification in addition to logistic regression.…”