Augmenting Microsoft's HoloLens with vuforia tracking for neuronavigation

Abstract

Major hurdles for Microsoft's HoloLens as a tool in medicine have been accessing tracking data, as well as a relatively high-localisation error of the displayed information; cumulatively resulting in its limited use and minimal quantification. The following work investigates the augmentation of HoloLens with the proprietary image processing SDK Vuforia, allowing integration of data from its front-facing RGB camera to provide more spatially stable holograms for neuronavigational use. Continuous camera tracking was able to maintain hologram registration with a mean perceived drift of 1.41 mm, as well as a mean sub 2-mm surface point localisation accuracy of 53%, all while allowing the researcher to walk about a test area. This represents a 68% improvement for the later and a 34% improvement for the former compared with a typical HoloLens deployment used as a control. Both represent a significant improvement on hologram stability given the current state-of-the-art, and to the best of the authors knowledge are the first example of quantified measurements when augmenting hologram stability using data from the RGB sensor.

3D Holographic Visualization for Pre-procedural Planning of Spatially Complex Brain Aneurysm Treatment

“…The accuracy of the Vuforia SDK integration with the Microsoft HoloLens has been previously studied for a neuronavigational app, and is reported to have a mean hologram drift of 1.41 mm. 10 While accuracy was outside the scope of this current study, these results could be veri ed through methods described by Frantz et al in a future study. 10 Future accuracy measures could verify patient safety and pre-procedural plan con dence for INRs.…”

“…10 While accuracy was outside the scope of this current study, these results could be veri ed through methods described by Frantz et al in a future study. 10 Future accuracy measures could verify patient safety and pre-procedural plan con dence for INRs.…”

“…The use of the Microsoft HoloLens in combination with registration techniques, such as Vuforia SDK markers (for tracking and alignment of virtual models with the physical world), has been studied and proven to be accurate, with a mean hologram drift of 1.41 mm in a study by Frantz et al 10 In a previous study at the Cleveland Clinic, similar AR-based technology exhibited potential to be able to assist interventional radiologists in the performance of abdominal tumor ablation procedures in both a clinical and pre-procedural setting. 11 The objective of this project is to evaluate the use of a novel prototype imaging system for assisting an INR in visualization and performance of pre-procedural planning for treatment of a morphologically complex brain aneurysm.…”

Augmented reality improves procedural efficiency and reduces radiation dose for CT-guided lesion targeting: a phantom study using HoloLens 2

“…Automated registration of the 3D model to CT grid was performed using computer vision and Vuforia 9.0.12 with the CT grid as the image target. Features on the CT grid can be reliably and quickly detected by Vuforia 12 , and studies have validated the accuracy of Vuforia on HoloLens (v1) 10 , 13 – 17 . Registration accuracies were not directly validated in this study; registration fidelity was confirmed visually by the operator based on complete alignment of the virtual gridlines with the physical gridlines.…”

Infrared marker tracking with the HoloLens for neurosurgical interventions

“…Thus, a great variety of two dimensional markers exist, such as AprilTags [1], Aruco [2], and the commercially distributed product Vuforia [3]. Frantz et al use Vuforia markers to realize tracking of a patient with the HoloLens [4] in neurosurgical interventions. Other approaches realize patient tracking through surface or contour matching.…”