Symptom checkers versus doctors: A prospective, head‐to‐head comparison for cough

Berry, Andrew C.; Berry, Nicholas A.; Wang, Bin; Mulekar, Madhuri S.; Melvin, Anne; Battiola, Richard J.; Bulacan, Frederick K; Berry, Bruce B.

doi:10.1111/crj.13135

Cited by 13 publications

(8 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is likely due in part to the lack of data on physical examinations and investigations available to the ED physician. Berry et al [ 39 ] compared the diagnostic accuracy of (1) physicians reviewing the symptom data items collected by the symptom checker (similar to this study) with (2) adding the clinical data collected by the reviewing physician. They confirmed that the additional clinical data significantly improved diagnostic accuracy.…”

Section: Discussionmentioning

confidence: 99%

“…Previous studies of symptom checkers in urgent or ED settings have generally been limited by the selection of less urgent patients—for example, Mediktor [ 41 ]—or lack of direct use of symptom checkers by patients. Berry et al [ 39 ] studied WebMD, iTriage, and FreeMD symptom checkers for the diagnosis of cough in primary care. The best-performing symptom checker had a sensitivity for the correct diagnosis of 34.5% (top 1 diagnosis) and 71.6% (top 3 diagnoses).…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Evaluation of Diagnostic and Triage Accuracy and Usability of a Symptom Checker in an Emergency Department: Observational Study

Fraser¹,

Cohan²,

Koehler³

et al. 2022

JMIR Mhealth Uhealth

View full text Add to dashboard Cite

Background Symptom checkers are clinical decision support apps for patients, used by tens of millions of people annually. They are designed to provide diagnostic and triage advice and assist users in seeking the appropriate level of care. Little evidence is available regarding their diagnostic and triage accuracy with direct use by patients for urgent conditions. Objective The aim of this study is to determine the diagnostic and triage accuracy and usability of a symptom checker in use by patients presenting to an emergency department (ED). Methods We recruited a convenience sample of English-speaking patients presenting for care in an urban ED. Each consenting patient used a leading symptom checker from Ada Health before the ED evaluation. Diagnostic accuracy was evaluated by comparing the symptom checker’s diagnoses and those of 3 independent emergency physicians viewing the patient-entered symptom data, with the final diagnoses from the ED evaluation. The Ada diagnoses and triage were also critiqued by the independent physicians. The patients completed a usability survey based on the Technology Acceptance Model. Results A total of 40 (80%) of the 50 participants approached completed the symptom checker assessment and usability survey. Their mean age was 39.3 (SD 15.9; range 18-76) years, and they were 65% (26/40) female, 68% (27/40) White, 48% (19/40) Hispanic or Latino, and 13% (5/40) Black or African American. Some cases had missing data or a lack of a clear ED diagnosis; 75% (30/40) were included in the analysis of diagnosis, and 93% (37/40) for triage. The sensitivity for at least one of the final ED diagnoses by Ada (based on its top 5 diagnoses) was 70% (95% CI 54%-86%), close to the mean sensitivity for the 3 physicians (on their top 3 diagnoses) of 68.9%. The physicians rated the Ada triage decisions as 62% (23/37) fully agree and 24% (9/37) safe but too cautious. It was rated as unsafe and too risky in 22% (8/37) of cases by at least one physician, in 14% (5/37) of cases by at least two physicians, and in 5% (2/37) of cases by all 3 physicians. Usability was rated highly; participants agreed or strongly agreed with the 7 Technology Acceptance Model usability questions with a mean score of 84.6%, although “satisfaction” and “enjoyment” were rated low. Conclusions This study provides preliminary evidence that a symptom checker can provide acceptable usability and diagnostic accuracy for patients with various urgent conditions. A total of 14% (5/37) of symptom checker triage recommendations were deemed unsafe and too risky by at least two physicians based on the symptoms recorded, similar to the results of studies on telephone and nurse triage. Larger studies are needed of diagnosis and triage performance with direct patient use in different clinical environments.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Evaluation of Diagnostic and Triage Accuracy and Usability of a Symptom Checker in an Emergency Department: Observational Study

Fraser¹,

Cohan²,

Koehler³

et al. 2022

JMIR Mhealth Uhealth

View full text Add to dashboard Cite

show abstract

“…Ultimately, there is likely no adequate “gold standard” differential diagnosis, and future work would benefit from assessing the inter-rater agreement between a larger sample of doctors. Further studies using real-world cohorts, for example within a health clinic ( Berry et al, 2019 ), will be required to demonstrate the relative performance of these systems to human doctors in more realistic contexts, where the ability to communicate with a patient is an additional factor in the diagnostic process. Such studies should ideally assess both algorithm performance and user interaction and could follow a multistage process whereby exposure to real-life clinical environments is gradually increased from early observational studies through randomised controlled trials to post-market surveillance during routine operational use ( Fraser et al, 2018 ).…”

Section: Discussionmentioning

confidence: 99%

A Comparison of Artificial Intelligence and Human Doctors for the Purpose of Triage and Diagnosis

Baker

Perov

Middleton

et al. 2020

Front. Artif. Intell.

View full text Add to dashboard Cite

AI virtual assistants have significant potential to alleviate the pressure on overly burdened healthcare systems by enabling patients to self-assess their symptoms and to seek further care when appropriate. For these systems to make a meaningful contribution to healthcare globally, they must be trusted by patients and healthcare professionals alike, and service the needs of patients in diverse regions and segments of the population. We developed an AI virtual assistant which provides patients with triage and diagnostic information. Crucially, the system is based on a generative model, which allows for relatively straightforward re-parameterization to reflect local disease and risk factor burden in diverse regions and population segments. This is an appealing property, particularly when considering the potential of AI systems to improve the provision of healthcare on a global scale in many regions and for both developing and developed countries. We performed a prospective validation study of the accuracy and safety of the AI system and human doctors. Importantly, we assessed the accuracy and safety of both the AI and human doctors independently against identical clinical cases and, unlike previous studies, also accounted for the information gathering process of both agents. Overall, we found that the AI system is able to provide patients with triage and diagnostic information with a level of clinical accuracy and safety comparable to that of human doctors. Through this approach and study, we hope to start building trust in AI-powered systems by directly comparing their performance to human doctors, who do not always agree with each other on the cause of patients’ symptoms or the most appropriate triage recommendation.

show abstract

“…The authors concluded that the quality of symptom checkers was variable and that triage applications were generally risk averse, often recommending higher levels of care than were needed. 46 An experimental study 47 comparing the accuracy of three symptom checkers for cough in an outpatient setting found that their diagnostic accuracy was poor when compared with physician assessment reference standard, with a 26.2% difference in correct diagnoses. A recent paper 48 described the rapid introduction of a digital self-triage and scheduling tool for COVID-19 integrated into a primary care system.…”

Section: Digital and Online Symptom Checkersmentioning

confidence: 99%

Impact of NHS 111 Online on the NHS 111 telephone service and urgent care system: a mixed-methods study

Turner

Knowles

Simpson

et al. 2021

Health Serv Deliv Res

View full text Add to dashboard Cite

Background The NHS emergency and urgent care system is under pressure as demand for services increases each year. NHS 111 is a telephone triage service designed to provide advice and signposting to appropriate services for people with urgent health-care problems. A new service, NHS 111 Online, has been introduced across England as a digital alternative that can be accessed using a website or a smartphone application. The effects and usefulness of this service are unknown. Objectives To explore the impact of NHS 111 Online on the related telephone service and urgent care system activity and the experiences of people who use those services. Design and methods A mixed-methods design of five related work packages comprising an evidence review; a quantitative before-and-after time series analysis of changes in call activity (18/38 sites); a descriptive comparison of telephone and online services with qualitative survey (telephone, n = 795; online, n = 3728) and interview (32 participants) studies of service users; a qualitative interview study (16 participants) of staff; and a cost–consequences analysis. Results The online service had little impact on the number of triaged calls to the NHS 111 telephone service. For every 1000 online contacts, triaged telephone calls increased by 1.3% (1.013, 95% confidence interval 0.996 to 1.029; p = 0.127). Recommendations to attend emergency and urgent care services increased between 6.7% and 4.2%. NHS 111 Online users were less satisfied than users of the telephone service (50% vs. 71%; p < 0.001), and less likely to recommend to others (57% vs. 69%; p < 0.001) and to report full compliance with the advice given (67.5% vs. 88%; p < 0.001). Online users were less likely to report contacting emergency services and more likely to report not making any contact with a health service (31% vs. 16%; p < 0.001) within 7 days of contact. Thirty-five per cent of online users reported that they did not want to use the telephone service, whereas others preferred its convenience and speed. NHS 111 telephone staff reported no discernible increase or decrease in their workload during the first year of operation of NHS 111 Online. If online and telephone services operate in parallel, then the annual costs will be higher unless ≥ 38% of telephone contacts move to online contacts. Conclusions There is some evidence that the new service has the potential to create new demand. The service has expanded significantly, so it is important to find ways of promoting the right balance in numbers of people who use the online service instead of the telephone service if it is to be effective. There is a clear need and preference by some people for an online service. Better information about when to use this service and improvements to questioning may encourage more uptake. Limitations The lack of control arm means that impact could have been an effect of other factors. This work took place during the early implementation phase, so findings may change as the service expands. Future work Further development of the online triage process to make it more ‘user friendly’ and to enable users to trust the advice given online could improve use and increase satisfaction. Better understanding of the characteristics of the telephone and online populations could help identify who is most likely to benefit and could improve information about when to use the service. Trial registration Current Controlled Trials ISRCTN51801112. Funding This project was funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research; Vol. 9, No. 21. See the NIHR Journals Library website for further project information.

show abstract

Symptom checkers versus doctors: A prospective, head‐to‐head comparison for cough

Cited by 13 publications

References 4 publications

Evaluation of Diagnostic and Triage Accuracy and Usability of a Symptom Checker in an Emergency Department: Observational Study

Evaluation of Diagnostic and Triage Accuracy and Usability of a Symptom Checker in an Emergency Department: Observational Study

A Comparison of Artificial Intelligence and Human Doctors for the Purpose of Triage and Diagnosis

Impact of NHS 111 Online on the NHS 111 telephone service and urgent care system: a mixed-methods study

Contact Info

Product

Resources

About