The use of mixed reality technology for the objective assessment of clinical skills: a validation study

Abstract

Background Mixed Reality technology may provide many advantages over traditional teaching methods. Despite its potential, the technology has yet to be used for the formal assessment of clinical competency. This study sought to collect validity evidence and assess the feasibility of using the HoloLens 2 mixed reality headset for the conduct and augmentation of Objective Structured Clinical Examinations (OSCEs). Methods A prospective cohort study was conducted to compare the assessment of undergraduate medical students undertaking OSCEs via HoloLens 2 live (HLL) and recorded (HLR), and gold-standard in-person (IP) methods. An augmented mixed reality scenario was also assessed. Results Thirteen undergraduate participants completed a total of 65 OSCE stations. Overall inter-modality correlation was 0.81 (p = 0.01), 0.98 (p = 0.01) and 0.82 (p = 0.01) for IP vs. HLL, HLL vs. HLR and IP vs. HLR respectively. Skill based correlations for IP vs. HLR were assessed for history taking (0.82, p = 0.01), clinical examination (0.81, p = 0.01), procedural (0.88, p = 0.01) and clinical skills (0.92, p = 0.01), and assessment of a virtual mixed reality patient (0.74, p = 0.01). The HoloLens device was deemed to be usable and practical (Standard Usability Scale (SUS) score = 51.5), and the technology was thought to deliver greater flexibility and convenience, and have the potential to expand and enhance assessment opportunities. Conclusions HoloLens 2 is comparable to traditional in-person examination of undergraduate medical students for both live and recorded assessments, and therefore is a valid and robust method for objectively assessing performance. The technology is in its infancy, and users need to develop confidence in its usability and reliability as an assessment tool. However, the potential to integrate additional functionality including holographic content, automated tracking and data analysis, and to facilitate remote assessment may allow the technology to enhance, expand and standardise examinations across a range of educational contexts.

Innovations in craniovertebral junction training: harnessing the power of mixed reality and head-mounted displays

“…The HoloLens 2, specifically as the medium of mixed reality, has been used to assess the clinical evaluation skills of medical students with good accuracy. 29 Because CVJ pathoanatomy is complex and variable, intuitive 3D understanding entails a steep learning curve. Previous studies have applied and proven the utility of extended reality modalities and simulators to help trainees better grasp CVJ anatomy and cultivate a 3D understanding of this anatomy.…”

COVID-19 impact on the assessment methodology of undergraduate medical students: a systematic review of the lessons learned

“…Of the 18 studies, four did not report the student's academic year (Alkhowailed et al, 2020;Prigoff et al, 2021;Yuda Handaya et al, 2021;Minty et al, 2022). Students were also in different phases of the medical curriculum.…”

“…Eleven studies were conducted in an online learning environment (Alkhowailed et al, 2020;Elzainy et al, 2020;Blythe et al, 2021;Co et al, 2021;Hernandez et al, 2021;Kronenfeld et al, 2021;Prigoff et al, 2021;Shaiba et al, 2021;Yuda Handaya et al, 2021;Chaudhuri et al, 2022;Minty et al, 2022), while the other 7 had a mix of online learning and face to face encounters (especially with students on clinical practice academical years) (Adeleke et al, 2020;Hanafy et al, 2021;Hope et al, 2021;Suwannaphisit et al, 2021;Aaraj et al, 2022;Co and Chu, 2022;Nathan et al, 2022). As expected, because of the restrictions imposed by the pandemic, no study had a just face-to-face traditional teaching methodology.…”

“…New teaching methodologies included: online lectures (recorded or live), live meetings, video interpretation (Suwannaphisit et al, 2021), simulation (Mileder et al, 2021;Palés-Argullós and Gomar-Sancho, 2021), Mixed reality (MR) (Minty et al, 2022) and Virtual Classroom Training (VCT) (Nathan et al, 2022).…”

Effect of Mixed Reality on Delivery of Emergency Medical Care in a Simulated Environment

“… 1 , 2 , 3 , 4 , 5 MR technology has been used across a range of clinical scenarios, including perioperative planning, surgical training, and 3D telemedicine support, 6 , 7 , 8 and deployed to support the delivery of clinical care during COVID-19. 9 With recent evidence suggesting an array of promising use cases for the technology, 10 , 11 , 12 , 13 there is increasing potential for MR to augment the delivery of emergency medical care by delivering clinically relevant holographic content and live communication across geographically disparate and remote environments. 14 In addition to its role in augmenting and enhancing direct clinical care, the technology has the potential to provide a means to capture, store, and annotate data obtained during routine clinical interactions that may populate digital data repositories to support future clinical care, education, and research.…”