Information and Communication Technologies Combined with Mixed Reality as Supporting Tools in Medical Education

Abstract

The dynamic COVID-19 pandemic has destabilized education and forced academic centers to explore non-traditional teaching modalities. A key challenge this creates is in reconciling the fact that hands-on time in lab settings has been shown to increase student understanding and peak their interests. Traditional visualization methods are already limited and topics such as 3D molecular structures remain difficult to understand. This is where advances in Information and Communication Technologies (ICT), including remote meetings, Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), and Extended Reality (XR, so-called Metaverse) offer vast potential to revolutionize the education landscape. Specifically, how MR merges real and virtual life in a uniquely promising way and offers opportunities for entirely new educational applications. In this paper, we briefly overview and report our initial experience using MR to teach medical and pharmacy students. We also explore the future usefulness of MR in pharmacy education. MR mimics real-world experiences both in distance education and traditional laboratory classes. We also propose ICT-based systems designed to run on the Microsoft HoloLens2 MR goggles and can be successfully applied in medical and pharmacy coursework. The models were developed and implemented in Autodesk Maya and exported to Unity. Our findings demonstrate that MR-based solutions can be an excellent alternative to traditional classes, notably in medicine, anatomy, organic chemistry, and biochemistry (especially 3D molecular structures), in both remote and traditional in-person teaching modalities. MR therefore has the potential to become an integral part of medical education in both remote learning and in-person study.

Artificial Intelligence-Based Algorithms in Medical Image Scan Segmentation and Intelligent Visual-Content Generation

“…Image segmentation has a crucial role in creating both, medical diagnosing supported by image analysis and virtual object creation like the medical digital twin (DT) of organs [66,67], holograms of the human organs [81,82], and virtual medical simulators [68,83]. One can split the image segmentation process into semantic segmentation (i.e.…”

“…The segmented structures (in this case organs and their disorders) may be next applied to the development of the 3D virtual environment [105]. These 3D objects may be implemented through for example, holograms displayed in the head-mounted display (HDMs) like Mixed Reality glasses in medical diagnostics [113], pre-operative imaging [114], surgical assistance [115,116], robotics surgery [117], and medical education [81,82]. However, the crucial issue is connected with the quality of obtained segmented structures, and this process can be significantly accelerated and improved by the use of Artificial Intelligence.…”

“…Recently, computer-generated three-dimensional (3D) images have become increasingly important in medical diagnostics [5,6]. In particular, Extended Reality (XR) so-called Metaverse is increasingly used in health care and medical education, while it enables the deeper experience of the virtual world, especially through the development of depth perception, including the rendering of several modalities like vision, touch, and hearing [7]. In fact, medical images have different modalities and their accurate classification at the pixel level enables the accurate identification of disorders and abnormalities [8,7].…”

“…In particular, Extended Reality (XR) so-called Metaverse is increasingly used in health care and medical education, while it enables the deeper experience of the virtual world, especially through the development of depth perception, including the rendering of several modalities like vision, touch, and hearing [7]. In fact, medical images have different modalities and their accurate classification at the pixel level enables the accurate identification of disorders and abnormalities [8,7]. However, creating a 3D model of organs and/or their abnormalities is time-consuming and is often done manually or semi-automatically [10].…”

The Metaverse in University Education during COVID-19: A Systematic Review of Success Factors

“…So we also have the type of metaverse called mirror world that makes it possible to access the reflected real world through the provision and integration of information taken from the external environment (Mujica-Sequera, 2022); it can be recognized through virtual map applications and modeling using GPS technology, such as Google Earth, Google Maps, Naver Maps, and Airbnb (Gómez et al, 2022). Finally, virtual reality is about immersive technology that seeks to create the complete immersion of the user in a totally virtual world, simulated and alien to their real or physical environment (Mariscal et al, 2020;Portela, 2022). These alternative realities make it possible to leave the classroom and learn from what is seen, that is, it is discovery-based learning (Alvarado et al, 2019;Tzancoff et al, 2019).…”