Outcomes, Measurement Instruments, and Their Validity Evidence in Randomized Controlled Trials on Virtual, Augmented, and Mixed Reality in Undergraduate Medical Education: Systematic Mapping Review

Abstract

Background Extended reality, which encompasses virtual reality (VR), augmented reality (AR), and mixed reality (MR), is increasingly used in medical education. Studies assessing the effectiveness of these new educational modalities should measure relevant outcomes using outcome measurement tools with validity evidence. Objective Our aim is to determine the choice of outcomes, measurement instruments, and the use of measurement instruments with validity evidence in randomized controlled trials (RCTs) on the effectiveness of VR, AR, and MR in medical student education. Methods We conducted a systematic mapping review. We searched 7 major bibliographic databases from January 1990 to April 2020, and 2 reviewers screened the citations and extracted data independently from the included studies. We report our findings in line with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. Results Of the 126 retrieved RCTs, 115 (91.3%) were on VR and 11 (8.7%) were on AR. No RCT on MR in medical student education was found. Of the 115 studies on VR, 64 (55.6%) were on VR simulators, 30 (26.1%) on screen-based VR, 9 (7.8%) on VR patient simulations, and 12 (10.4%) on VR serious games. Most studies reported only a single outcome and immediate postintervention assessment data. Skills outcome was the most common outcome reported in studies on VR simulators (97%), VR patient simulations (100%), and AR (73%). Knowledge was the most common outcome reported in studies on screen-based VR (80%) and VR serious games (58%). Less common outcomes included participants’ attitudes, satisfaction, cognitive or mental load, learning efficacy, engagement or self-efficacy beliefs, emotional state, competency developed, and patient outcomes. At least one form of validity evidence was found in approximately half of the studies on VR simulators (55%), VR patient simulations (56%), VR serious games (58%), and AR (55%) and in a quarter of the studies on screen-based VR (27%). Most studies used assessment methods that were implemented in a nondigital format, such as paper-based written exercises or in-person assessments where examiners observed performance (72%). Conclusions RCTs on VR and AR in medical education report a restricted range of outcomes, mostly skills and knowledge. The studies largely report immediate postintervention outcome data and use assessment methods that are in a nondigital format. Future RCTs should include a broader set of outcomes, report on the validity evidence of the measurement instruments used, and explore the use of assessments that are implemented digitally.

Motivational Design for Web-Based Instruction in Health Professions Education: Protocol for a Systematic Review and Directed Content Analysis

“…Although previous reviews have focused on motivational design features of web-based instruction in HPE [ 38 , 44 , 49 - 58 ], none have sought to (1) achieve the specific goal of using existing evidence to refine models of motivational design, (2) propose which types of evidence will be required to meet this goal, (3) identify the study designs that can generate such evidence (eg, studies comparing motivational design strategies), and (4) appraise the degree to which studies have generated such evidence. Thus, the value of our review lies in its ability to appraise where we have been and to inform where we ought to go.…”

“…Studies will also be identified by hand searching the reference lists of previous systematic reviews related to web-based instruction in HPE [ 38 , 44 , 49 - 58 ].…”

Educational effects of and satisfaction with mixed-reality-based major trauma care simulator: A preliminary evaluation

“…[20] However, MR equipment is not widely distributed yet; therefore, the development of medical education programs using it is limited. [1] Although there are reports of education on the treatment of trauma patients using MR technology, there are cases where it was developed with the goal of learning specific skills rather than the entire process of treating trauma patients. [21,22] This study aimed to examine the educational effect and satisfaction of a tool to learn the entire process of actual trauma patient treatment using MR technology.…”

“…XR technology is subdivided into virtual reality (VR), augmented reality (AR), and mixed reality (MR). [1][2][3] It is classified into whether the head-mountdisplay (HMD) that provides the image delivers only complete digital information (VR) or whether virtual information is overlaid on real information through translucent glass (AR, MR). In addition, virtual information can be directly manipulated by the user (VR, MR), or information can be provided unilaterally (AR).…”

Effect of Mixed Reality on Delivery of Emergency Medical Care in a Simulated Environment

“…In addition to its role in augmenting and enhancing direct clinical care, the technology has the potential to provide a means to capture, store, and annotate data obtained during routine clinical interactions that may populate digital data repositories to support future clinical care, education, and research. It may also lead to better quality assurance and governance through novel performance and outcome insights . Despite MR’s potential, the technology remains relatively immature and there are important questions concerting its sustainability, scalability, and cost-effectiveness.…”