Benchmarking Triage Capability of Symptom Checkers Against That of Medical Laypersons: Survey Study

Abstract

Background Symptom checkers (SCs) are tools developed to provide clinical decision support to laypersons. Apart from suggesting probable diagnoses, they commonly advise when users should seek care (triage advice). SCs have become increasingly popular despite prior studies rating their performance as mediocre. To date, it is unclear whether SCs can triage better than those who might choose to use them. Objective This study aims to compare triage accuracy between SCs and their potential users (ie, laypersons). Methods On Amazon Mechanical Turk, we recruited 91 adults from the United States who had no professional medical background. In a web-based survey, the participants evaluated 45 fictitious clinical case vignettes. Data for 15 SCs that had processed the same vignettes were obtained from a previous study. As main outcome measures, we assessed the accuracy of the triage assessments made by participants and SCs for each of the three triage levels (ie, emergency care, nonemergency care, self-care) and overall, the proportion of participants outperforming each SC in terms of accuracy, and the risk aversion of participants and SCs by comparing the proportion of cases that were overtriaged. Results The mean overall triage accuracy was similar for participants (60.9%, SD 6.8%; 95% CI 59.5%-62.3%) and SCs (58%, SD 12.8%). Most participants outperformed all but 5 SCs. On average, SCs more reliably detected emergencies (80.6%, SD 17.9%) than laypersons did (67.5%, SD 16.4%; 95% CI 64.1%-70.8%). Although both SCs and participants struggled with cases requiring self-care (the least urgent triage category), SCs more often wrongly classified these cases as emergencies (43/174, 24.7%) compared with laypersons (56/1365, 4.10%). Conclusions Most SCs had no greater triage capability than an average layperson, although the triage accuracy of the five best SCs was superior to the accuracy of most participants. SCs might improve early detection of emergencies but might also needlessly increase resource utilization in health care. Laypersons sometimes require support in deciding when to rely on self-care but it is in that very situation where SCs perform the worst. Further research is needed to determine how to best combine the strengths of humans and SCs.

Patient's Perception of Digital Symptom Assessment Technologies in Rheumatology: Results From a Multicentre Study

“…Similar to a previous observational study analyzing the potential impact of Ada ( 12 ), the majority of patients (already being at the healthcare facility) declared that they would not have done anything differently after having used the DDSS. In line with previous work that showed a general risk adversity of symptom checkers ( 4 , 11 ), our results shows that DDSS suggestions would actually encourage patients to seek earlier care rather than turn to self-care or see a physician later. On the other hand, Meyer et al ( 13 ) reported that 14/26 (54%) of symptom checkers users given advice to proceed to the ED actually did.…”

“…Currently, DDSS evaluation studies largely focus on the evaluation of the diagnostic accuracy ( 4 , 10 , 11 ) and only few studies analyzed the usability and acceptability by their end-users, i.e., actual patients. A recent study conducted by the symptom checker company Ada Health suggests that the majority of patients (511/522, 97.8%) rated the symptom checker Ada as very easy or quite easy to use and would recommend it to a friend or relative (444/520, 85.3%) ( 12 ).…”

Avey: An Accurate AI Algorithm for Self-Diagnosis

“…Some work has also explored the triage capabilities of checkers [5, 51, 60]. Studying the utility and triage capabilities of checkers are beyond the scope of this paper and have been set as future work in Section 5.5.…”

“…In the US, almost two-thirds of adults search the Web for health information and roughly one-third utilize it for self-diagnosis , trying to discover by themselves the underlying causes of their health symptoms [36]. A recent study showed that half of the patients investigated their symptoms on search engines before visiting emergency departments [38, 51].…”

Correction: Benchmarking Triage Capability of Symptom Checkers Against That of Medical Laypersons: Survey Study

“…In the originally published manuscript, Reference 24 in the reference list was cited incorrectly. This citation has now been corrected to the reference listed below [1].…”