Lessons Learned from the Usability Evaluation of a Simulated Patient Dialogue System

Abstract

Simulated consultations through virtual patients allow medical students to practice history-taking skills. Ideally, applications should provide interactions in natural language and be multi-case, multi-specialty. Nevertheless, few systems handle or are tested on a large variety of cases. We present a virtual patient dialogue system in which a medical trainer types new cases and these are processed without human intervention. To develop it, we designed a patient record model, a knowledge model for the history-taking task, and a termino-ontological model for term variation and out-of-vocabulary words. We evaluated whether this system provided quality dialogue across medical specialities (n = 18), and with unseen cases (n = 29) compared to the cases used for development (n = 6). Medical evaluators (students, residents, practitioners, and researchers) conducted simulated history-taking with the system and assessed its performance through Likert-scale questionnaires. We analysed interaction logs and evaluated system correctness. The mean user evaluation score for the 29 unseen cases was 4.06 out of 5 (very good). The evaluation of correctness determined that, on average, 74.3% (sd = 9.5) of replies were correct, 14.9% (sd = 6.3) incorrect, and in 10.7% the system behaved cautiously by deferring a reply. In the user evaluation, all aspects scored higher in the 29 unseen cases than in the 6 seen cases. Although such a multi-case system has its limits, the evaluation showed that creating it is feasible; that it performs adequately; and that it is judged usable. We discuss some lessons learned and pivotal design choices affecting its performance and the end-users, who are primarily medical students.

Virtual reality in medical emergencies training: benefits, perceived stress, and learning success

“…However, this requires the integration of sophisticated applications to enable speech recognition. Recent developments in natural language processing in virtual standardized patients only demonstrated a response accuracy of 74-92% in dialogs [34,35]. Especially in the management of emergencies, even higher levels of precision might prove mandatory, in order to prevent the misconception of information under time pressure.…”

Virtual reality in medical emergencies training: benefits, perceived stress, and learning success

“…However, this requires the integration of sophisticated applications to enable speech recognition. Recent developments in natural language processing in virtual standardized patients only demonstrated a response accuracy of 74-92% in dialogues [34,35]. Especially in the management of emergencies, even higher levels of precision might prove mandatory, in order to prevent the misconception of information under time pressure.…”

Towards believable and educational conversations with virtual patients

“…VP systems can provide explicit medical skills training recommended for health professionals' education to reduce the impact of future diagnostic errors and potential patient harm Cleland et al, 2009;Balogh et al, 2015. Several research projects are investigating the use of VP systems in the education of medical students Campillos-Llanos et al, 2020;Pantziaras et al, 2015;Mavrogiorgou et al, 2022;Graf et al, 2023b. Thereby, the art of VPs can vary from chatbots Cameron et al (2019) to embodied conversation virtual agents Campillos-Llanos et al, 2020;Pantziaras et al, 2015. They can be accessible via different devices like computers Pantziaras et al, 2015;Campillos-Llanos et al, 2020or VR headsets Mavrogiorgou et al, 2022Graf et al, 2023b. Several review articles have investigated the effectiveness of VP systems over the past years Cook et al, 2010;Milne-Ives et al, 2020;McGaghie et al, 2010;Plackett et al, 2022;Kocaballi et al, 2019;Isaza-Restrepo et al, 2018.…”

“…Extended reality systems like VR have become increasingly relevant as a means for medical education Kononowicz et al, 2015, Kononowicz et al, 2019Campillos-Llanos et al, 2020;Pantziaras et al, 2015;Mavrogiorgou et al, 2022;Graf et al, 2023b. Utilizing its high degree of sensory immersion and natural interaction affordances, VR enables the simulation of face-to-face interaction scenarios within an adaptive learning environment that is cost-effective, scalable, and applicable in a standardized way for different learners.…”