Online Symptom Checkers: Recommendations for a Vignette-Based Clinical Evaluation Standard

Painter, Annabelle; Hayhoe, Benedict; Sasco, Eva Riboli; El-Osta, Austen

doi:10.2196/37408

Cited by 27 publications

(22 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We created 72 vignette cases total, 24 for each of our three conditions. The number of vignettes needed to evaluate symptom checkers is not well defined 72 . Other vignette symptom checker evaluations have used between 3-400 cases for testing with single-condition or single-system evaluations (e.g.…”

Section: Strengths and Limitationsmentioning

confidence: 99%

Assessing the accuracy of a digital symptom checker tool for suggestion of reproductive health conditions: a clinical vignettes study

Peven¹,

Wickham²,

Wilks³

et al. 2023

Preprint

View full text Add to dashboard Cite

Background Reproductive health conditions such as endometriosis, uterine fibroids and polycystic ovary syndrome affect a large proportion of women and people who menstruate worldwide. Prevalence estimates for these conditions range from 5-40% of women of reproductive age. Long diagnostic delays, up to 12 years, are common and contribute to health complications and increased healthcare costs. Symptom checker apps provide users with information and tools to better understand their symptoms and thus have the potential to reduce the time to diagnosis for reproductive health conditions. Objective This study aims to evaluate the accuracy of three symptom checkers developed by Flo Health assessing symptoms of endometriosis, uterine fibroids and polycystic ovary syndrome (PCOS) against current medical guidelines. Methods Independent general practitioners were recruited to create clinical case vignettes of simulated users with and without the conditions of interest. Vignettes were reviewed, modified and approved by separate general practitioners. A further independent panel of general practitioners reviewed the cases and designated a final classification. Vignettes were entered into the symptom checkers and the outcomes were compared with the final classification from the panel using accuracy metrics including percent agreement, sensitivity and specificity. Results A total of 24 cases were created per condition. Overall, exact matches between the vignette classification and the symptom checker outcome was 83.3% for endometriosis and uterine fibroids, and 87.5% for PCOS. While sensitivity was high for all conditions (>81%) and very high (100%) for PCOS, specificity was >81% for endometriosis and uterine fibroids and 75% for PCOS. Conclusion The single condition symptom checkers have high levels of accuracy for endometriosis, uterine fibroids and PCOS. Given long delays in diagnosis for many reproductive health conditions, which lead to increased medical costs and potential health complications for individuals and healthcare providers, innovative health apps and symptom checkers hold the potential to improve care pathways.

show abstract

Section: Strengths and Limitationsmentioning

confidence: 99%

Assessing the accuracy of a digital symptom checker tool for suggestion of reproductive health conditions: a clinical vignettes study

Peven¹,

Wickham²,

Wilks³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Specific evidence standards should be provided for OSCs to augment existing guidance, such as the NICE evidence standards framework and the evaluation requirements for medical device certification with the MHRA. A set of congruent requirements for standardised vignette-based clinical evaluation process of OSCs has been proposed with this aim [27].…”

Section: Implications For Research and Practicementioning

confidence: 99%

Triage and Diagnostic Accuracy of Online Symptom Checkers: Systematic Review (Preprint)

Sasco¹,

El-Osta²,

Alaa³

et al. 2022

Preprint

View full text Add to dashboard Cite

BACKGROUND In the context of a deepening global shortage of health workers, and particularly the COVID-19 pandemic, there is growing international interest in and use of online symptom checkers (OSCs). However, the evidence surrounding the safety and accuracy of OSCs remains inconclusive so far. The triage and diagnostic accuracy of these tools is an essential aspect that needs to be addressed before pushing any further implementation. OBJECTIVE This systematic review aimed to summarize the existing peer-reviewed literature evaluating the triage accuracy (directing users to appropriate services based on their presenting symptoms) and diagnostic accuracy of OSCs aimed at lay users for general health concerns. METHODS Searches were conducted in Medline, Embase, CINAHL, HMIC and Web of Science. We included peer-reviewed studies published in English between 1 January 2010 and 17 February 2022 with a quantitative assessment of triage and/or diagnostic accuracy of OSCs directed at lay users. We excluded tools supporting health professionals, and disease- or speciality-specific OSCs. Screening and data extraction were carried out independently by two reviewers for each study. We performed a descriptive narrative synthesis. RESULTS 21,284 studies were screened and 15 were included. Six studies reported on both triage and diagnostic accuracy, eight focused on triage accuracy, and one on diagnostic accuracy. Diagnostic and triage accuracy varied between studies and OSCs; most studies showed suboptimal diagnostic and triage accuracy. Frequency and urgency of the condition were the main variables that affected the levels of diagnostic and triage accuracy, along with specific features of the OSCs. The impact of each variable differed across tools and studies, making it difficult to draw any solid conclusions. Included studies had either a moderate or high risk of bias according to the revised tool for the Quality Assessment of Diagnostic Accuracy Studies 2. CONCLUSIONS While OSCs have significant potential to provide accessible and accurate health advice and triage recommendations to users, more research is needed to validate their triage and diagnostic accuracy prior to wide scale adoption in community and healthcare settings. Future studies should aim to use a common methodology and/or agreed standard for evaluation to facilitate objective benchmarking and validation.

show abstract

“…Evaluation studies of symptom checkers show high variability, however, with average accuracy estimates ranging from 27% to 90% (2,3). The reasons for the wide range of accuracy estimates are not entirely clear, although a first set of factors might relate to the choice of different evaluation methods, including testing procedures (19), types of case vignettes tested (19–21), and the gold standard solutions assigned to these cases (21,22). For instance, not all symptom checkers may be tested with every vignette (20) as some symptom checkers are designed for specialized tasks (such as only addressing pediatric cases), whereas other symptom checkers restrict the types of symptoms that may be entered and processed (16).…”

Section: Introductionmentioning

confidence: 99%

“…As a remedy researchers proposed solutions to standardize evaluation methodologies: Painter et al proposed several requirements, including standardizing the number of inputters, developing a standardized way of determining a gold standard solution to a case, or developing more reliable vignettes that are more representative for real-world cases (21). El-Osta et al examined variability in the vignette creation processes and urged the field to use real-world data instead of artificial vignettes (22).…”

Section: Introductionmentioning

confidence: 99%

symptomcheckR: an R package for analyzing and visualizing symptom checker performance

Kopka,

Feufel

2024

Preprint

View full text Add to dashboard Cite

Background: A major stream of research on symptom checkers aims at evaluating the technology′s predictive accuracy, but apart from general trends, the results are marked by high variability. Several authors suggest that this variability might in part be due to different assessment methods and a lack of standardization. To improve the reliability of symptom checker evaluation studies, several approaches have been suggested, including standardizing input procedures, the generation of test vignettes, and the assignment of gold standard solutions for these vignettes. Recently, we suggested a third approach—test-theoretic metrics for standardized performance reporting—to allow systematic and comprehensive comparisons of symptom checker performance. However, calculating these metrics is time-consuming and error prone, which could hamper the use and effectiveness of these metrics. Results: We developed the R package symptomcheckR as an open-source software to assist researchers in calculating standard metrics to evaluate symptom checker performance individually and comparatively and produce publication- ready figures. These metrics include accuracy (by triage level), safety of advice (i.e., rate of correct or overtriage), comprehensiveness (i.e., how many cases could be entered or were assessed), inclination to overtriage (i.e., how risk-averse a symptom checker is) and a capability comparison score (i.e., a score correcting for case difficulty and comprehensiveness that enables a fair and reliable comparison of different symptom checkers). Each metric can be obtained using a single command and visualized with another command. For the analysis of individual or the comparison of multiple symptom checkers, single commands can be used to produce a comprehensive performance profile that complements the standard focus on accuracy with additional metrics that reveal strengths and weaknesses of symptom checkers. Conclusions: Our package supports ongoing efforts to improve the quality of vignette-based symptom checker evaluation studies by means of standardized methods. Specifically, with our package, adhering to reporting standards and metrics becomes easier, simple, and time efficient. Ultimately, this may help users gain a more systematic understanding of the strengths and limitations of symptom checkers for different use cases (e.g., all-purpose symptom checkers for general medicine versus symptom checkers that aim at improving triage in emergency departments), which can improve patient safety and resource allocation.

show abstract

Online Symptom Checkers: Recommendations for a Vignette-Based Clinical Evaluation Standard

Cited by 27 publications

References 32 publications

Assessing the accuracy of a digital symptom checker tool for suggestion of reproductive health conditions: a clinical vignettes study

Assessing the accuracy of a digital symptom checker tool for suggestion of reproductive health conditions: a clinical vignettes study

Triage and Diagnostic Accuracy of Online Symptom Checkers: Systematic Review (Preprint)

symptomcheckR: an R package for analyzing and visualizing symptom checker performance

Contact Info

Product

Resources

About