Performance of a new symptom checker in patient triage: Canadian cohort study

Abstract

Background Computerized algorithms known as symptom checkers aim to help patients decide what to do should they have a new medical concern. However, despite widespread implementation, most studies on symptom checkers have involved simulated patients. Only limited evidence currently exists about symptom checker safety or accuracy when used by real patients. We developed a new prototype symptom checker and assessed its safety and accuracy in a prospective cohort of patients presenting to primary care and emergency departments with new medical concerns. Method A prospective cohort study was done to assess the prototype’s performance. The cohort consisted of adult patients (≥16 years old) who presented to hospital emergency departments and family physician clinics. Primary outcomes were safety and accuracy of triage recommendations to seek hospital care, seek primary care, or manage symptoms at home. Results Data from 281 hospital patients and 300 clinic patients were collected and analyzed. Sensitivity to emergencies was 100% (10/10 encounters). Sensitivity to urgencies was 90% (73/81) and 97% (34/35) for hospital and primary care patients, respectively. The prototype was significantly more accurate than patients at triage (73% versus 58%, p<0.01). Compliance with triage recommendations in this cohort using this iteration of the symptom checker would have reduced hospital visits by 55% but cause potential harm in 2–3% from delay in care. Interpretation The prototype symptom checker was superior to patients in deciding the most appropriate treatment setting for medical issues. This symptom checker could reduce a significant number of unnecessary hospital visits, with accuracy and safety outcomes comparable to existing data on telephone triage.

Evaluation of Diagnostic and Triage Accuracy and Usability of a Symptom Checker in an Emergency Department: Observational Study

“…The study by Hill et al [ 24 ], which also replicated the study by Semigran et al [ 12 ], showed a mean triage accuracy of 49%, with stronger performance on emergency cases. The study by Chan et al [ 35 ] (reported earlier) showed significantly better triage accuracy by their locally developed symptom checker than the patients overall (73% vs 58%; P <.01), and performance on emergency and urgent cases was stronger than on routine or home care. A recent study on the use of Ada by 378 “walk-in” patients in urgent care compared its triage accuracy with the result of a triage nurse using the Manchester Triage System [ 44 ].…”

“…We are not aware of a study of patients directly entering data on their own symptoms to evaluate the accuracy of both diagnosis and triage in a general ED population. A study in 2 Canadian EDs and 13 primary care practices evaluated a symptom checker developed by the team that recommended a triage level out of 4 options [ 35 ]. For 281 hospital patients, the sensitivity for emergencies was 10/10 (100%) and, for urgent cases, it was 73/81 (90%), but performance for routine and home care was poorer at 52% and 29%, respectively.…”

Triage and Diagnostic Accuracy of Online Symptom Checkers: Systematic Review (Preprint)

“…This was surprising since triage advice, ie whether and where users should seek a healthcare assessment for their presenting symptoms is precisely one of the main functions of OSCs. In addition, as Chan et al 2021 [23] included in their review and as others have shown [26], patients deciding to present to ED does not automatically qualify them as requiring emergency treatment, thus undermining the pertinence of Berry et al 2019 [13] findings regarding triage accuracy.…”

“…Table 1 shows the main characteristics of included studies published between 2014 and 2022. Six [11][12][13][14][15][16] were conducted by researchers based in the USA, three in the UK [17][18][19], two in Australia [20,21], two in Canada [22,23], one in the Netherlands [24] and one in Hong Kong [25]. Eight studies [11,12,[14][15][16][19][20][21] used standardised patient vignettes; several were inspired by or included the 45 vignettes used by Semigran et al [11,12] The remaining seven studies used data from real patients either through their medical health records [13,25] or direct input by users [17,18,[22][23][24] in different settings, including primary care and emergent care settings.…”

“…Seven studies evaluated a single OSC [14,[16][17][18][22][23][24], while the other eight tested and compared the performance of two [25] to 36 [20] OSCs. Where provided, the most common justifications for selection were language (English), level of popularity among users, and accessibility (free).…”