Assessment of Diagnosis and Triage in Validated Case Vignettes Among Nonphysicians Before and After Internet Search

Levine, David M.; Mehrotra, Ateev

doi:10.1001/jamanetworkopen.2021.3287

Cited by 22 publications

(25 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Users might also be more inclined to accept guiding advice from symptom checker apps before arriving at a health care facility. Interestingly, a web search seems to change only few people's urgency level [ 62 ]; this indicates a difference in advice-taking between symptom checker use and general web search.…”

Section: Discussionmentioning

confidence: 99%

Determinants of Laypersons’ Trust in Medical Decision Aids: Randomized Controlled Trial

Kopka¹,

Schmieding²,

Rieger³

et al. 2022

JMIR Hum Factors

View full text Add to dashboard Cite

Background Symptom checker apps are patient-facing decision support systems aimed at providing advice to laypersons on whether, where, and how to seek health care (disposition advice). Such advice can improve laypersons’ self-assessment and ultimately improve medical outcomes. Past research has mainly focused on the accuracy of symptom checker apps’ suggestions. To support decision-making, such apps need to provide not only accurate but also trustworthy advice. To date, only few studies have addressed the question of the extent to which laypersons trust symptom checker app advice or the factors that moderate their trust. Studies on general decision support systems have shown that framing automated systems (anthropomorphic or emphasizing expertise), for example, by using icons symbolizing artificial intelligence (AI), affects users’ trust. Objective This study aims to identify the factors influencing laypersons’ trust in the advice provided by symptom checker apps. Primarily, we investigated whether designs using anthropomorphic framing or framing the app as an AI increases users’ trust compared with no such framing. Methods Through a web-based survey, we recruited 494 US residents with no professional medical training. The participants had to first appraise the urgency of a fictitious patient description (case vignette). Subsequently, a decision aid (mock symptom checker app) provided disposition advice contradicting the participants’ appraisal, and they had to subsequently reappraise the vignette. Participants were randomized into 3 groups: 2 experimental groups using visual framing (anthropomorphic, 160/494, 32.4%, vs AI, 161/494, 32.6%) and a neutral group without such framing (173/494, 35%). Results Most participants (384/494, 77.7%) followed the decision aid’s advice, regardless of its urgency level. Neither anthropomorphic framing (odds ratio 1.120, 95% CI 0.664-1.897) nor framing as AI (odds ratio 0.942, 95% CI 0.565-1.570) increased behavioral or subjective trust (P=.99) compared with the no-frame condition. Even participants who were extremely certain in their own decisions (ie, 100% certain) commonly changed it in favor of the symptom checker’s advice (19/34, 56%). Propensity to trust and eHealth literacy were associated with increased subjective trust in the symptom checker (propensity to trust b=0.25; eHealth literacy b=0.2), whereas sociodemographic variables showed no such link with either subjective or behavioral trust. Conclusions Contrary to our expectation, neither the anthropomorphic framing nor the emphasis on AI increased trust in symptom checker advice compared with that of a neutral control condition. However, independent of the interface, most participants trusted the mock app’s advice, even when they were very certain of their own assessment. Thus, the question arises as to whether laypersons use such symptom checkers as substitutes rather than as aids in their own decision-making. With trust in symptom checkers already high at baseline, the benefit of symptom checkers depends on interface designs that enable users to adequately calibrate their trust levels during usage. Trial Registration Deutsches Register Klinischer Studien DRKS00028561; https://tinyurl.com/rv4utcfb (retrospectively registered).

show abstract

Section: Discussionmentioning

confidence: 99%

Determinants of Laypersons’ Trust in Medical Decision Aids: Randomized Controlled Trial

Kopka¹,

Schmieding²,

Rieger³

et al. 2022

JMIR Hum Factors

View full text Add to dashboard Cite

show abstract

“…Building on prior related work [12, 22, 27, 36, 52, 53], we adopted a clinical vignette approach to measure the performance of Avey alongside several other checkers. A seminal work at Harvard Medical School has established the value of this approach [22, 52, 53] for testing checkers, especially that it has been also a common method to test physicians on their diagnosis abilities [53].…”

Section: Methodsmentioning

confidence: 99%

“…For instance, in Australia, around 80% of people search the Internet for health information, and nearly 40% seek guidance online for self-treatment [13, 27]. In the US, almost two-thirds of adults search the Web for health information and roughly one-third utilize it for self-diagnosis , trying to discover by themselves the underlying causes of their health symptoms [36]. A recent study showed that half of the patients investigated their symptoms on search engines before visiting emergency departments [38, 51].…”

Section: Introductionmentioning

confidence: 99%

“…While search engines like Google and Bing are exceptional tools for educating people on almost any matter, they may facilitate misdiagnosis and induce risks stemmed from unrelated health content [36]. This is because Web search entails sifting through an ocean of results, which could emanate from all sorts of sources, and making personal judgements on which data to unveil.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Avey: An Accurate AI Algorithm for Self-Diagnosis

Hammoud

Douglas

Darmach

et al. 2022

Preprint

View full text Add to dashboard Cite

ObjectivesTo present our AI-based symptom checker, rigorously measure its accuracy, and compare it against existing popular symptom checkers and seasoned primary care physicians.DesignVignettes study.Setting400 gold-standard primary care vignettes.Intervention/ComparatorWe utilized 7 standard accuracy metrics for evaluating the performance of 6 symptom checkers. To this end, we developed and peer-reviewed 400 vignettes, each approved by at least 5 out of 7 independent and experienced general practitioners. To the best of our knowledge, this yielded the largest benchmark vignette suite in the field thus far. To establish a frame of reference and interpret the results of symptom checkers accordingly, we further directly compared the best-performing symptom checker against 3 primary care physicians with an average experience of 16.6 years.Primary Outcome MeasuresWe thoroughly studied the diagnostic accuracies of symptom checkers and physicians from 7 standard angles, including: (a) M1, M3, and M5 as measures of a symptom checker’s or a physician’s ability to return a vignette’s main diagnosis at the top, among the first 3 diseases, or among the first 5 diseases of their differential diagnosis, respectively (b) recall as a measure of the percentage of relevant diseases that are returned in a symptom checker’s or a physician’s differential diagnosis, (c) precision as a measure of the percentage of diseases in a symptom checker’s or a physician’s differential diagnosis that are relevant, (d) F1-measure as a trade-off measure between recall and precision, and (e) Normalized Discounted Cumulative Gain or NDCG as a measure of ranking quality of a symptom checker’s or a physician’s differential diagnosis.ResultsOur AI-based symptom checker, namely, Avey significantly outperformed 5 popular symptom checkers, namely, Ada, WebMD, K Health, Buoy, and Babylon by averages of 24.5%, 175.5%, 142.8%, 159.6%, 2968.1% using M1; 22.4%, 114.5%, 123.8%, 118.2%, 3392% using M3; 18.1%, 79.2%, 116.8%, 125%, 3114.2% using M5; 25.2%, 65.6%, 109.4%, 154%, 3545% using recall; 8.7%, 88.9%, 66.4%, 88.9%, 2084% using F1-measure; and 21.2%, 93.4%, 113.3%, 136.4%, 3091.6% using NDCG, respectively. Under precision, Ada outperformed Avey by an average of 0.9%, while Avey surpassed WebMD, K Health, Buoy, and Babylon by averages of 103.2%, 40.9%, 49.6%, and 1148.5%, respectively. To the contrary of symptom checkers, physicians outperformed Avey by averages of 37.1% and 1.2% using precision and F1-measure, while Avey exceeded them by averages of 10.2%, 20.4%, 23.4%, 56.4%, and 25.1% using M1, M3, M5, recall, and NDCG, respectively. To facilitate the reproducibility of our study and support future related studies, we made all our gold-standard vignettes publicly and freely available. Moreover, we posted online all the results of the symptoms checkers and physicians (i.e., 45 sets of experiments) to establish a standard of full transparency and enable verifying and cross validating our results.ConclusionsAvey tremendously outperformed the considered symptom checkers. In addition, it compared favourably to physicians, whereby it underperformed them under some accuracy metrics (e.g., precision and F1-measure), but outperformed them under some others (e.g., M1, M3, M5, recall, and NDCG). We will continue evolving Avey’s AI model. Furthermore, we will study its usability with real patients, examine how they respond to its suggestions, and measure its impact on their subsequent choices for care, among others.

show abstract

“…Ninety percent of databases used to create the DermAssist software extracted images from patients with lighter skin. 2 Melanoma Project, the largest available pigmented lesion dataset, relies heavily on images collected from patients with fair skin from the United Kingdom, United States, and Australia. 3 Given the skewed ethnic and geographic variability of the datasets, AI may proffer less accurate diagnoses for conditions encountered in patients with darker skin, such as acral melanoma or nail melanoma.…”

mentioning

confidence: 99%

The introduction of “Dr AI”: What dermatologists should consider

Dave

Nambudiri

Grant‐Kels

2023

Journal of the American Academy of Dermatology

View full text Add to dashboard Cite

Assessment of Diagnosis and Triage in Validated Case Vignettes Among Nonphysicians Before and After Internet Search

Abstract: This survey study evaluates the factors associated with accurate diagnosis and triage by nonphysician adults who use the internet to find health information.

Cited by 22 publications

References 23 publications

Determinants of Laypersons’ Trust in Medical Decision Aids: Randomized Controlled Trial

Determinants of Laypersons’ Trust in Medical Decision Aids: Randomized Controlled Trial

Avey: An Accurate AI Algorithm for Self-Diagnosis

The introduction of “Dr AI”: What dermatologists should consider

Contact Info

Product

Resources

About