Concordance assessment of a cognitive computing system in Thailand.

Abstract

6589 Background: IBM Watson for Oncology (WFO) was trained by Memorial Sloan Kettering and is a cognitive computing system that uses natural language processing to ingest patient data in structured and unstructured formats. The system provides physicians with treatment options that are derived from established guidelines, the medical literature, and training from patient cases. In this study, we assessed the degree of concordance between treatment recommendations proposed by WFO and oncologists at Bumrungrad International Hospital (BIH). BIH is a 580-bed multispecialty hospital in Bangkok, Thailand. Methods: Data from breast, colorectal, gastric, and lung cancer patients treated at BIH were entered into WFO in 2015 and 2016. Retrospective cases were entered after a treatment plan had been determined, and prospective cases were entered during patients’ treatment planning sessions. WFO recommendations were provided in 3 categories: “Recommended”, “For Consideration”, and “Not Recommended.” Concordance was analyzed by comparing the decisions made by the oncologists to those proposed by WFO. Concordance was achieved when the oncologist’s treatment suggestion was in the “Recommended” or “For Consideration” categories given by WFO. Results: A total of 211 cases were assessed, 92 were retrospective and 119 were prospective. The overall concordance rate was 83%; 89% for colorectal, 91% for lung, 76% for breast, and 78% for gastric cancer. Similar concordance rates were observed when retrospective and prospective cases were analyzed separately. Discordance was attributable in part to local oncologists’ preferences for non-U.S. guidelines for certain cancers, especially gastric cancer. Conclusions: There was a high degree of concordance between WFO treatment options and the decisions made by local oncologists. Similar results were recently reported in a breast cancer concordance study conducted using WFO in India (San Antonio Breast Cancer Symposium 2016, Somashekhar et al). WFO’s capabilities as a cognitive decision support tool can be further improved by incorporating regional guidelines. Future work will analyze reasons for discordance such as cost, insurance requirements, and patient and physician preference.

Concordance Study Between IBM Watson for Oncology and Real Clinical Practice for Cervical Cancer Patients in China: A Retrospective Analysis

“…A observational study from Korea showed a 73% concordance rate for colon cancer and a 49% concordance rate for gastric cancer (Somashekhar et al, 2016;Suwanvecho et al, 2017). And, a comparative Study from Korea indicated that WFO without the gene expression assay has limited clinical utility (Kim et al, 2018).…”

<p>Concordance of Treatment Recommendations for Metastatic Non-Small-Cell Lung Cancer Between Watson for Oncology System and Medical Team</p>

“…12 Consistency of treatment regimens between WFO and oncologists had been observed for breast cancer, lung cancer, gastric cancer, colon cancer, rectal cancer, cervical cancer, and ovarian cancer. 12,[15][16][17] However, the concordance varies significantly according to different countries and cancer types. 12,[15][16][17] There are many guidelines for NSCLC precise therapies.…”

“…12,[15][16][17] However, the concordance varies significantly according to different countries and cancer types. 12,[15][16][17] There are many guidelines for NSCLC precise therapies. The commonly used clinical guidelines include the ASCO (American Society of Clinical Oncology) lung cancer guideline, National Comprehensive Cancer Network (NCCN) Clinical Practice guideline, European Society of Cancer Internal Medicine (ESMO) guideline, and Chinese Society of Clinical Oncology (CSCO) guideline.…”

“…WFO was developed based on NCCN guidelines, whereas the treatment of lung cancer patients in China mostly follows CSCO guidelines. Although the WFO system as a new research and application field had been explored in lung cancer, 12,15,17 the samples of the associated studies have been too small. In particular, there is currently inadequate information available on mNSCLC in Chinese patients.…”

Differential impact of cognitive computing augmented by real world evidence on novice and expert oncologists

“…Comparison studies of WfO against recommendations at a tumor board in India noted concordance in 93% of breast, 96.4% of lung, 81.0% of colon and 92.7% of rectal cancer cases . A review in Thailand noted concordance rates of 89% for colorectal, 91% for lung, 76% for breast, and 78% for gastric cancer, with variance often driven by regional guidelines . An expansion of the 2016 Thailand study conducted in 2017 for a total of 590 cases found the overall concordance rate for the retrospective cases was 81%; 78% for bladder, 70% for breast, 56% for cervical, 90% for colon, 82% for gastric, 93% for lung, 79% for ovarian, 82% for prostate and 94% for rectal.…”