Evaluation of marker tracking using mono and stereo vision in Microsoft HoloLens for surgical navigation

Abstract

Optical-based navigation systems are widely used in surgical interventions. However, despite their great utility and accuracy, they are expensive and require time and effort to setup for surgeries. Moreover, traditional navigation systems use 2D screens to display instrument positions causing the surgeons to look away from the operative field. Head mounted displays such as the Microsoft HoloLens may provide an attractive alternative for surgical navigation that also permits augmented reality visualization. The HoloLens is equipped with multiple sensors for tracking and scene understanding. Mono and stereo-vision in the HoloLens have been both reported to be used for marker tracking, but no extensive evaluation on accuracy has been done to compare the two approaches. The objective of our work is to investigate the tracking performance of various camera setups in the HoloLens, and to study the effect of the marker size, marker distance from camera, and camera resolution on marker locating accuracy. We also investigate the speed and stability of marker pose for each camera setup. The tracking approaches are evaluated using ArUco markers. Our results show that mono-vision is more accurate in marker locating than stereo-vision when high resolution is used. However, this comes at the expense of higher frame processing time. Alternatively, we propose a combined low-resolution mono-stereo tracking setup that outperforms each tracking approach individually and is comparable to high resolution mono tracking, with a mean translational error of 1.8 ± 0.6mm for 10cm marker size at 50cm distance. We further discuss our findings and their implications for navigation in surgical interventions.

Assessment of Multiple Fiducial Marker Trackers on Hololens 2

“…Recent works in this field have mainly focused on assessing the maker tracking accuracy on medical applications, particularly within surgical procedures. This interest primarily stems from the fact that OST-HMDs hold promise as a potential solution for the current challenge faced by surgeons, which entails diverting their attention from the surgical site when utilizing conventional navigation systems [7], [8], [23]. The underlying objective of these AR applications is to streamline the interpretation of 3D spatial data derived from MRI and CT scans by digitally superimposing a virtual model onto the patient's body.…”

“…Based on the calibrateand-test procedure proposed in [1], Brand et al presented the residual error for uncalibrated, isometric, affine, and perspective calibration models, where the affine and perspective models had the lowest residual error with a negligible difference between them. Thabit et al [23] extended Brand's work by investigating the tracking performance of Aruco markers on the different cameras of HL2, as well as the effect of the marker size, distance from the camera, and camera resolution. While the HL2 stereo cameras were limited to a resolution of 640x480, the photo/video (PV) camera was tested at resolutions of 640x360, 760x428, 960x540, 1280x760, and 1920x1080 pixels.…”

“…Those using the frontal HL2 PV camera to track the marker were set to 1920x1080 resolution, which is the HL2 PV camera default. The only application that uses the HL2 stereo tracking cameras is the QR Code one, which is limited to a resolution of 640x480 [23]. Moreover, the Unity Robotics Hub (URH) was used to establish the communication between the HL2 and the ROSMaster via Wi-Fi.…”

“…Asserting the accuracy and precision of fiducial marker trackers in head-mounted displays echoes over current research trends on augmented reality [25], such as assessing the viability of using these devices in a range of applications, such as medicine [23], architecture and civil engineering [26], [27], simultaneous localization and mapping [4], human-robot collaboration [28], [29], and many others.…”

Accuracy assessment of marker recognition using ultra wide angle camera

“…ISSN 2664-9969 have a lower resolution of the light-sensitive matrix compared to monoscopic camera. This leads to obtaining of a lower level of recognition accuracy of fiducial markers [8,9].…”