Patient posture correction and alignment using mixed reality visualization and the HoloLens 2

Abstract

Purpose The purpose of this study was to develop and preliminarily test a radiotherapy system for patient posture correction and alignment using mixed reality (MixR) visualization. The write‐up of this work also provides an opportunity to introduce the concepts and technology of MixR for a medical physics audience who may be unfamiliar with the topic. Methods A MixR application was developed for on optical‐see‐through head‐mounted display (HoloLens 2) allowing a user to simultaneously and directly view a patient and a reference hologram derived from their simulation CT scan. The hologram provides a visual reference for the exact posture needed during treatment and is initialized in relation to the origin of a radiotherapy device using marker‐based tracking. The system further provides marker‐less tracking that allows the user tofreely navigate the room as they view and align the patient from various angles. The system was preliminarily tested using both a rigid (pelvis) and nonrigid (female mannequin) anthropomorphic phantom. Each phantom was aligned via hologram and accuracy quantified using CBCT and CT. Results A fully realized system was developed. Rigid registration accuracy was on the order of 3.0 ± 1.5 mm based on the performance of three users repeating alignment five times each. The lateral direction showed the most variability among users and was associated with the largest off‐sets (approximately 2.0 mm). For nonrigid alignment, the MixR setup outperformed a setup based on three‐point alignment and setup photos, the latter of which showed a difference in arm position of 2 cm and a torso roll of 6–7°. Conclusions MixR visualization is a rapidly emerging domain that has the potential to significantly impact the field of medicine. The current application is an illustration of this and highlights the advantages of MixR for patient setup in radiation oncology. The key feature of the system is the way in which it transforms nonrigid registration into rigid registration by providing an efficient, portable, and cost‐effective mechanism for reproducing patient posture without the use of ionizing radiation. Preliminary estimates of registration accuracy indicate clinical viability and form the foundation for further development and clinical testing.

Development and feasibility evaluation of an AR-assisted radiotherapy positioning system

“…In this study, an innovative AR-assisted radiotherapy positioning system was developed with HoloLens 2 as the only core hardware and provided patient tracking and virtual coordinate indication for radiation therapist, instead of relying on the human eye for virtual-real alignment ( 12 – 14 , 16 ). Compared with the related works, this improves the convenience ( 12 , 16 ), accuracy ( 17 ), and practicality of AR guidance systems.…”

“…In recent years, AR technology has been increasingly used in medicine, such as education and training (6)(7)(8) and hologram-guided surgery (9)(10)(11). Talbot et al first utilized the AR technique to guide the positioning of radiotherapy patients (12), which was subsequently explored and improved by Tarutani et al (13) and Johnson et al (14). However, there are some limitations for these methods.…”

Augmenting Performance: A Systematic Review of Optical See-Through Head-Mounted Displays in Surgery

“…Laparoscopic surgery ( ), reconstructive surgery ( ), radiotherapy ( ), dental surgery ( ), and heart surgery ( ) were the last categories surveyed. Reconstructive surgery included orbital floor reconstruction [ 69 ] and radiotherapy applications involved AR-based patient positioning [ 70 , 71 ]. Laparoscopic surgery applications included cholecystectomy [ 72 ] and general laparoscopic surgery [ 73 ].…”

Microsoft HoloLens 2 in Medical and Healthcare Context: State of the Art and Future Prospects

“…Four contributions [ 108 , 109 , 110 , 111 ] have been included in the “other applications” subgroup since, due to their characteristics, they cannot be configured as belonging to the subgroups mentioned. More specifically, in the study of Onishi et al [ 108 ], the authors implemented a prototype system, named Gaze-Breath, in which gaze and breathing are integrated, using an MR headset and a thermal camera, respectively, for hands-free or intuitive inputs to control the cursor three-dimensionally and facilitate switching between pointing and selection.…”

“…More specifically, in the study of Onishi et al [ 108 ], the authors implemented a prototype system, named Gaze-Breath, in which gaze and breathing are integrated, using an MR headset and a thermal camera, respectively, for hands-free or intuitive inputs to control the cursor three-dimensionally and facilitate switching between pointing and selection. Johnson et al [ 109 ] developed and preliminarily tested a radiotherapy system for patient posture correction and alignment using a mixed reality visualization. Kurazume et al [ 110 ] presented a comparative study of two AR training systems for Humanitude dementia care, a multimodal comprehensive care methodology for patients with dementia.…”