Efficacy of Artificial-Intelligence-Driven Differential-Diagnosis List on the Diagnostic Accuracy of Physicians: An Open-Label Randomized Controlled Study

Abstract

Background: The efficacy of artificial intelligence (AI)-driven automated medical-history-taking systems with AI-driven differential-diagnosis lists on physicians’ diagnostic accuracy was shown. However, considering the negative effects of AI-driven differential-diagnosis lists such as omission (physicians reject a correct diagnosis suggested by AI) and commission (physicians accept an incorrect diagnosis suggested by AI) errors, the efficacy of AI-driven automated medical-history-taking systems without AI-driven differential-diagnosis lists on physicians’ diagnostic accuracy should be evaluated. Objective: The present study was conducted to evaluate the efficacy of AI-driven automated medical-history-taking systems with or without AI-driven differential-diagnosis lists on physicians’ diagnostic accuracy. Methods: This randomized controlled study was conducted in January 2021 and included 22 physicians working at a university hospital. Participants were required to read 16 clinical vignettes in which the AI-driven medical history of real patients generated up to three differential diagnoses per case. Participants were divided into two groups: with and without an AI-driven differential-diagnosis list. Results: There was no significant difference in diagnostic accuracy between the two groups (57.4% vs. 56.3%, respectively; p = 0.91). Vignettes that included a correct diagnosis in the AI-generated list showed the greatest positive effect on physicians’ diagnostic accuracy (adjusted odds ratio 7.68; 95% CI 4.68–12.58; p < 0.001). In the group with AI-driven differential-diagnosis lists, 15.9% of diagnoses were omission errors and 14.8% were commission errors. Conclusions: Physicians’ diagnostic accuracy using AI-driven automated medical history did not differ between the groups with and without AI-driven differential-diagnosis lists.

Effects of a Differential Diagnosis List of Artificial Intelligence on Differential Diagnoses by Physicians: An Exploratory Analysis of Data from a Randomized Controlled Study

“…As a result, that study showed that the AI-driven DDSS with AI-driven AMHT system also led to omission and commission errors at similar rates as did the type of DDSS that generated differential diagnoses after physicians enter the key symptoms and findings after a clinical encounter [ 10 ]. The following study, which we reported recently, also confirmed the findings [ 17 ]. However, these studies did not assess the mechanism of omission and commission errors when using this new type of DDSS.…”

“…According to a previous study, task performance can worsen by using an automated rather than a manual procedure when the reliability of automation is <70% [ 20 ]. As the reliability of AI in this study was set at 50% (the correct diagnosis was listed in 8 of 16 cases), approximately 15% of the effect of the DDSS on the physicians’ differential diagnosis may be associated with the commission errors observed in our main analysis [ 17 ]. The prevalence of diagnoses identical with the differential diagnosis of AI was also approximately 15% higher in the intervention group than in the control group, even in the subset of AI incorrect cases, which led to commission errors.…”

“…These results were consistent with previous reports that physicians who were more experienced and had less trust in automation were reluctant to accept the advice from automated decision support systems [ 21 , 22 , 23 ]. These two groups were associated with fewer commission errors in our main analysis [ 17 ]; therefore, to reduce commission errors when using a DDSS, physicians should consider the reliability of DDSSs, and less experienced physicians or physicians who trust AI should reduce their reliance on AI.…”

“…The primary analysis results of our randomized controlled study were previously reported [ 17 ]. In brief, this randomized controlled study was conducted in January 2021 at the Department of Diagnostic and Generalist Medicine of Dokkyo Medical University, in which 22 physicians (five interns, eight residents, and nine attending physicians) were randomly assigned in a 1:1 allocation to either reading 16 written clinical vignettes generated by AI on the basis of the information of actual patients with (intervention group, n = 11) or without (control group, n = 11) a top-10 differential diagnosis list of AI.…”

“…In addition, the initial diagnosis was reported to be important for accurate diagnosis [ 12 , 13 ], and the accuracy was high in physicians who received enjoy early support with a DDSS [ 14 , 15 , 16 ]. Therefore, the other types of DDSS that generate differential diagnosis from information entered by patients themselves before clinical encounter [ 10 , 17 ] can be expected to reduce DDSS-related diagnostic errors, particularly omission errors.…”