Cinematic Rendering in Mixed-Reality Holograms: A New 3D Preoperative Planning Tool in Pediatric Heart Surgery

Abstract

Cinematic rendering (CR) is based on a new algorithm that creates a photo-realistic three-dimensional (3D) picture from cross-sectional images. Previous studies have shown its positive impact on preoperative planning. To date, CR presentation has only been possible on 2D screens which limited natural 3D perception. To depict CR-hearts spatially, we used mixed-reality technology and mapped corresponding hearts as holograms in 3D space. Our aim was to assess the benefits of CR-holograms in the preoperative planning of cardiac surgery. Including 3D prints allowed a direct comparison of two spatially resolved display methods. Twenty-six patients were recruited between February and September 2019. CT or MRI was used to visualize the patient's heart preoperatively. The surgeon was shown the anatomy in cross-sections on a 2D screen, followed by spatial representations as a 3D print and as a high-resolution hologram. The holographic representation was carried out using mixed-reality glasses (HoloLens®). To create the 3D prints, corresponding structures were segmented to create STL files which were printed out of resin. In 22 questions, divided in 5 categories (3D-imaging effect, representation of pathology, structure resolution, cost/benefit ratio, influence on surgery), the surgeons compared each spatial representation with the 2D method, using a five-level Likert scale. The surgical preparation time was assessed by comparing retrospectively matched patient pairs, using a paired t-test. CR-holograms surpassed 2D-monitor imaging in all categories. CR-holograms were superior to 3D prints in all categories (mean Likert scale 4.4 ± 1.0 vs. 3.7 ± 1.3, P < 0.05). Compared to 3D prints it especially improved the depth perception (4.7 ± 0.7 vs. 3.7 ± 1.2) and the representation of the pathology (4.4 ± 0.9 vs. 3.6 ± 1.2). 3D imaging reduced the intraoperative preparation time (n = 24, 59 ± 23 min vs. 73 ± 43 min, P < 0.05). In conclusion, the combination of an extremely photo-realistic presentation via cinematic rendering and the spatial presentation in 3D space via mixed-reality technology allows a previously unattained level of comprehension of anatomy and pathology in preoperative planning.

Clinical Applications of Mixed Reality and 3D Printing in Congenital Heart Disease

“…A recent study by Gehrsitz et al has demonstrated the usefulness of MR in the pre-operative planning of pediatric heart surgeries. The use of MR was found to significantly improve the depth perception and portrayal of the pathology when compared to the two-dimensional (2D) medical images and 3DPHM [ 21 ]. The use of AR has also been shown constructive in Valve-in-Valve Transcatheter Heart Valve Implantation and percutaneous coronary intervention procedures [ 22 , 23 ].…”

“…Especially for MR, its superiority over the current visualization technique requires further validation. The published article to compare 3D printing and MR in pediatric heart surgeries is limited to one surgeon’s assessment based on a single-center experience [ 21 ].…”

Multimodal Imaging with 3D-Holograms for Preoperative Planning in Pediatric Cardiac Surgery: A Unique Case Report

“…Current clinical applications include minimally invasive cardiac procedures, mitral valve surgery, and trans-catheter interventional procedures [2,[7][8][9][10][11].…”

“…Augmented or mixed reality, in fact, accounting for head-sets made of translucent glasses, allows for real-time rendering of holographic 3D contents in the surrounding world, while preserving the interaction with the actual environment [1]. This technology is currently used for education, simulation, and procedural planning in the surgical and/or trans-catheter treatment of patients with congenital and acquired heart disease [2][3][4][5][6][7][8][9][10], but still presents some limitations.…”

Advances and Innovations in Ablative Head and Neck Oncologic Surgery Using Mixed Reality Technologies in Personalized Medicine

“…In particular, poor radiologic scans require manual reprocessing in contrast to cases with high-quality radiologic scans [ 39 ]. To improve visualization, voxel regions can be assigned specific color and opacity values, or supporting parameters such as textures or annotations can be used to mark or graphically represent structures [ 40 , 41 ]. By transmitting the processed treatment plans over a wireless network to a head-mounted MR device with the appropriate viewer software (Magic Leap 1, Plantation, Florida, USA; Viewer version 5.3, Brainlab AG, Munich, Germany), it is possible to view the radiological data as well as their reconstructions as 3D holograms (see Figure 1 ).…”