Concordance Rate between Clinicians and Watson for Oncology among Patients with Advanced Gastric Cancer: Early, Real-World Experience in Korea

Abstract

Backgrounds/Aims. Watson for Oncology (WFO) is a cognitive technology that processes medical information by analyzing the latest evidence and guidelines. However, studies of the concordance rate between WFO and clinicians for advanced gastric cancer (AGC) are lacking. Methods. We retrospectively reviewed 65 patients with AGC who consulted WFO and the Gachon Gil Medical Center multidisciplinary team (GMDT) in 2016 and 2017. The recommendations of WFO were compared with the opinions of the GMDT. WFO provided three treatment options: recommended (first treatment option), for consideration (second treatment option), and not recommended. Results. In total, 65 patients (mean age 61.0 years; 44 males and 21 females) were included in the study. The concordance rate between WFO and the GMDT was 41.5% (27/65) at the recommended level and 87.7% (57/65) at the for consideration level. The main causes of discordance between WFO and the GMDT were as follows. First, WFO did not consider the medical history. Second, WFO recommended the use of agents that are considered outdated in Korea. Third, some patients wanted to be involved in a clinical trial. Fourth, some patients refused to use the biologic agents recommended by WFO for financial reasons as they were not covered by medical insurance. Conclusions. The concordance rate at the recommended level was relatively low but was higher at the for consideration level. Discordances arose mainly from the different medical circumstances at the Gachon Gil Medical Center (GMC) and the Memorial Sloan Kettering Cancer Center (MSKCC), the main WFO consulting center. The utility of WFO as a tool for supporting clinical decision making could be further improved by incorporating regional guidelines.

Use of Natural Language Processing to Extract Clinical Cancer Phenotypes from Electronic Medical Records

“…Lee and colleagues (77) studied concordance of IBM Watson for Oncology, a commercial NLP-based treatment recommendation system, with the recommendations of local experts and it was 48.9%. Similar results are reported in (78,79). Furthermore, such applications are treated as Software as Medical Device (SaMD) by the FDA, which, justifiably, is a high bar to clear (80,81).…”

Who is afraid of black box algorithms? On the epistemological and ethical basis of trust in medical AI

“…Indeed, the degree of reliability of Watson for Oncology at the Memorial Sloan Kettering Cancer Center is different from those used in the Rigshospitalet in Copenhagen, as has been extensively discussed. [36][37][38] As for additional reliability indicators, we believe that transparency, understood as a process that informs the inner workings of black box algorithms is the type of right indicator that will contribute to the overall reliability of algorithms. As suggested before, transparency by itself is necessary, although not sufficient for entrenching the reliability of black box algorithms and the overall trustworthiness of their results.…”

Concordance Between Watson for Oncology and Multidisciplinary Teams in Colorectal Cancer: Prognostic Implications and Predicting Concordance

“…In general, we believe that WFO system could provide a great assistance to the MDT, as it provides treatment advices based on the updated knowledge and comprehensive evidence. It is worth mentioning that unlike other studies that only focused on consistency rate (3,10,11,13), a nomogram, which incorporated tumor type, and TNM stage, could indicate individual concordance possibilities between WFO and MDT recommendations with high sensitivity and specificity was first developed in the current study. Although we attempted to subdivide the patients by tumor type, and develop nomograms separately; however, we were unsuccessful because of the small sample size (data not shown).…”