Application of an interactive diagnosis ranking algorithm in a simulated vignette-based environment for general dermatology

Abstract

Background: Diagnostic algorithms may reduce noise and bias and improve interrater agreement of clinical decisions. In a practical sense, algorithms may serve as alternatives to specialist consultations or decision support in store-and-forward teledermatology. It is, however, unknown how dermatologists interact with algorithms based on questionnaires. 
 
 Objective: To evaluate the performance of a questionnaire-based diagnostic algorithm when applied by users with different expertise.
 
 Methods: We created 58 virtual test cases covering common dermatologic diseases and asked five raters with different expertise to complete a predefined clinical questionnaire, which served as input for a disease ranking algorithm. We compared the ranks of the correct diagnosis between users, analysed the similarity between inputs of different users, and explored the impact of different parts of the questionnaire on the final ranking. 
 
 Results: When applied by a board-certified dermatologist, the algorithm top-ranked the correct diagnosis in the majority of cases (median rank 1; IQR: 1.0; mean reciprocal rank 0.757). The median rank of the correct diagnosis was significantly lower when the algorithm was applied by four dermatology residents (median rank 2-5, p<.01 for all). The lowest similarity between inputs of the residents and the board-certified dermatologist was found for questions regarding morphology. Sensitivity analysis showed the highest deterioration in performance after omission of information on morphology and anatomic site. 
 Conclusions: A simple questionnaire-based disease ranking algorithm provides accurate ranking for a wide variety of dermatologic conditions. When applied in clinical practice, additional measures may be needed to ensure robustness of data entry for inexperienced users.

Evaluating a CDS Hook for FHIR Questionnaires in a SMART on FHIR App and an Existing Dermatological CDS System

“…Its goal is to collect responses for easy-to-answer atomic elements, collate them to a full virtual dermatologic examination, and provide reasonable feedback based on Bayesian-type search through a knowledge database. The usability of the algorithm was recently evaluated using patient vignettes [9] via a web-based interface 2 (see Figure 1 on the left). While harboring the theoretical potential for clinical application, the tool is currently used in teaching and an educational context.…”