Artificial Intelligence Clinical Evidence Engine for Automatic Identification, Prioritization, and Extraction of Relevant Clinical Oncology Research

Abstract

PURPOSE We developed a system to automate analysis of the clinical oncology scientific literature from bibliographic databases and match articles to specific patient cohorts to answer specific questions regarding the efficacy of a treatment. The approach attempts to replicate a clinician’s mental processes when reviewing published literature in the context of a patient case. We describe the system and evaluate its performance. METHODS We developed separate ground truth data sets for each of the tasks described in the paper. The first ground truth was used to measure the natural language processing (NLP) accuracy from approximately 1,300 papers covering approximately 3,100 statements and approximately 25 concepts; performance was evaluated using a standard F1 score. The ground truth for the expert classifier model was generated by dividing papers cited in clinical guidelines into a training set and a test set in an 80:20 ratio, and performance was evaluated for accuracy, sensitivity, and specificity. RESULTS The NLP models were able to identify individual attributes with a 0.7-0.9 F1 score, depending on the attribute of interest. The expert classifier machine learning model was able to classify the individual records with a 0.93 accuracy (95% CI, 0.9 to 0.96, P < .0001), and sensitivity and specificity of 0.95 and 0.91, respectively. Using a decision boundary of 0.5 for the positive (expert) label, the classifier demonstrated an F1 score of 0.92. CONCLUSION The system identified and extracted evidence from the oncology literature with a high degree of accuracy, sensitivity, and specificity. This tool enables timely access to the most relevant biomedical literature, providing critical support to evidence-based practice in areas of rapidly evolving science.

Natural Language Processing–Assisted Literature Retrieval and Analysis for Combination Therapy in Cancer

“…[6][7][8] Other subscription-based resources like Embase and Cochrane provide support that cross-links biomedical literature with some specific topics, but do not yet include combination therapy. 9,10 Saiz et al 11 from the IBM Watson Health group developed an automated clinical evidence engine leveraging artificial intelligence to mine clinical oncology-related research. Their application was not yet customized to identify preclinical evidence or the combination therapy context.…”

Physicians' Perceptions and Expectations of an Artificial Intelligence-Based Clinical Decision Support System in Cancer Care in an Underserved Setting

“…IBM's WfO is an AI-based CDSS used for oncology treatment selection that provides ranked, evidence-based therapeutic options to oncologists for consideration. 22 The tool is trained by the Memorial Sloan Kettering Cancer Center (MSKCC) 23 through learnings from test cases and experts utilizing recommendations that are consistent with established guidelines and published evidence. All the information input is verified by the oncologists at MSKCC, and WfO data are updated to the latest information every 1 to 2 months.…”

“…IBM's WfO is an AI-based CDSS used for oncology treatment selection that provides ranked, evidence-based therapeutic options to oncologists for consideration. 22 The tool is Results Eleven oncologists participated in the study. The following overarching themes and subthemes emerged from the analysis of interview transcripts: theme-1, "general context" including (1) current setting, workload, and patient population and (2) existing challenges in cancer treatment, and theme-2, "perceptions around the potential use of an AI-based tool," including (1) perceived benefits and (2) perceived challenges.…”

BIR: Biomedical Information Retrieval System for Cancer Treatment in Electronic Health Record Using Transformers

“…Furthermore, NLP/NLU and ML have focused on features of our system, comprising ML to categorize abstracts, medication-attribute connection in clinical narratives, identification of clinical research evidence, or PubMed-wide annotations [19][20][21]. Saiz et al [22] designed Watson Oncology Literature Insights (WOLI) to support clinicians in the training of evidence-based medicine (EBM) by classifying related and appropriate research information in clinical oncology and peer-reviewed literature. Moreover, clinical information can be contextualized using WOLI using a particular patient situation or cohort to provide clinicians with directed information.…”