Background Rapid advances in science challenge the timely adoption of evidence‐based care in community settings. To bridge the gap between what is possible and what is practiced, we researched approaches to developing an artificial intelligence (AI) application that can provide real‐time patient‐specific decision support. Materials and Methods The Oncology Expert Advisor (OEA) was designed to simulate peer‐to‐peer consultation with three core functions: patient history summarization, treatment options recommendation, and management advisory. Machine‐learning algorithms were trained to construct a dynamic summary of patients cancer history and to suggest approved therapy or investigative trial options. All patient data used were retrospectively accrued. Ground truth was established for approximately 1,000 unique patients. The full Medline database of more than 23 million published abstracts was used as the literature corpus. Results OEA's accuracies of searching disparate sources within electronic medical records to extract complex clinical concepts from unstructured text documents varied, with F1 scores of 90%–96% for non‐time‐dependent concepts (e.g., diagnosis) and F1 scores of 63%–65% for time‐dependent concepts (e.g., therapy history timeline). Based on constructed patient profiles, OEA suggests approved therapy options linked to supporting evidence (99.9% recall; 88% precision), and screens for eligible clinical trials on http://ClinicalTrials.gov (97.9% recall; 96.9% precision). Conclusion Our results demonstrated technical feasibility of an AI‐powered application to construct longitudinal patient profiles in context and to suggest evidence‐based treatment and trial options. Our experience highlighted the necessity of collaboration across clinical and AI domains, and the requirement of clinical expertise throughout the process, from design to training to testing. Implications for Practice Artificial intelligence (AI)‐powered digital advisors such as the Oncology Expert Advisor have the potential to augment the capacity and update the knowledge base of practicing oncologists. By constructing dynamic patient profiles from disparate data sources and organizing and vetting vast literature for relevance to a specific patient, such AI applications could empower oncologists to consider all therapy options based on the latest scientific evidence for their patients, and help them spend less time on information “hunting and gathering” and more time with the patients. However, realization of this will require not only AI technology maturation but also active participation and leadership by clincial experts.
6508 Background: Electronic decision support is increasingly prevalent in clinical practice. Traditional tools map guidelines into an interactive platform. An alternative method builds on experience-based learning. Methods: Memorial Sloan-Kettering (MSK), IBM and WellPoint teamed to develop IBM Watson – a cognitive computing system leveraging natural language processing (NLP), machine learning (ML) and massive parallel processing – to help inform clinical decision making. We made a prototype for lung cancers using manufactured and anonymized patient cases. We configured this tool to read medical language and extract specific attributes from each case to identify appropriate treatment options benchmarked against MSK expertise, anonymized patient cases and published evidence. Treatment options reflect consensus guidelines and MSK best practices where guidelines are not granular enough to match treatments to unique patients. Analysis and building accuracy is ongoing and iterative. Results: 420 manufactured and 525 anonymized patient cases trained the initial models. Early results show accuracy improvement in NLP and ML in identifying treatment options (Table). All treatment plans were guideline adherent. A proportion of cases showed the need to incorporate tailored treatment plans reflecting MSK’s practice beyond guidelines – e.g. 11% of cases required addressing a site of critical metastasis before initiating guideline supported treatment. Conclusions: IBM Watson is extracting information from free text medical records that supports building ML models to assist in selecting treatments for persons with lung cancers. This tool can select treatment options from consensus guidelines, and, through ML, it will identify personalized treatment plans. Training is ongoing to improve individualized decision making and optimize the web-based tool that connects with IBM Watson. [Table: see text]
PURPOSE We developed a system to automate analysis of the clinical oncology scientific literature from bibliographic databases and match articles to specific patient cohorts to answer specific questions regarding the efficacy of a treatment. The approach attempts to replicate a clinician’s mental processes when reviewing published literature in the context of a patient case. We describe the system and evaluate its performance. METHODS We developed separate ground truth data sets for each of the tasks described in the paper. The first ground truth was used to measure the natural language processing (NLP) accuracy from approximately 1,300 papers covering approximately 3,100 statements and approximately 25 concepts; performance was evaluated using a standard F1 score. The ground truth for the expert classifier model was generated by dividing papers cited in clinical guidelines into a training set and a test set in an 80:20 ratio, and performance was evaluated for accuracy, sensitivity, and specificity. RESULTS The NLP models were able to identify individual attributes with a 0.7-0.9 F1 score, depending on the attribute of interest. The expert classifier machine learning model was able to classify the individual records with a 0.93 accuracy (95% CI, 0.9 to 0.96, P < .0001), and sensitivity and specificity of 0.95 and 0.91, respectively. Using a decision boundary of 0.5 for the positive (expert) label, the classifier demonstrated an F1 score of 0.92. CONCLUSION The system identified and extracted evidence from the oncology literature with a high degree of accuracy, sensitivity, and specificity. This tool enables timely access to the most relevant biomedical literature, providing critical support to evidence-based practice in areas of rapidly evolving science.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.