Development of a surgical navigation system based on augmented reality using an optical see-through head-mounted display

Abstract

The surgical navigation system has experienced tremendous development over the past decades for minimizing the risks and improving the precision of the surgery. Nowadays, Augmented Reality (AR)-based surgical navigation is a promising technology for clinical applications. In the AR system, virtual and actual reality are mixed, offering real-time, high-quality visualization of an extensive variety of information to the users (Moussa et al., 2012) [1]. For example, virtual anatomical structures such as soft tissues, blood vessels and nerves can be integrated with the real-world scenario in real time. In this study, an AR-based surgical navigation system (AR-SNS) is developed using an optical see-through HMD (head-mounted display), aiming at improving the safety and reliability of the surgery. With the use of this system, including the calibration of instruments, registration, and the calibration of HMD, the 3D virtual critical anatomical structures in the head-mounted display are aligned with the actual structures of patient in real-world scenario during the intra-operative motion tracking process. The accuracy verification experiment demonstrated that the mean distance and angular errors were respectively 0.809±0.05mm and 1.038°±0.05°, which was sufficient to meet the clinical requirements.

On-patient see-through augmented reality based on visual SLAM

“…In surface-based registration techniques [2][3][4][5][6], a tablet-PC is mounted with a range camera, RGB-D sensor or stereo-vision to continuously capture the depth and color information, from which the skin surface is automatically extracted. This surface is then registered with the P3DM acquired from CT images, typically ICP (iterative closest point) is used.…”

“…Third, this kind of methods are not robust to partial occlusions of the skin. To achieve real-time performance, either parallel processing [4] or client/server architecture [6] or both [2] are used: A powerful server PC is necessary to process data, and the tablet-PC is used as a display tool only.…”

“…Indeed, markers are likely to be occluded by surgeon hand or a surgical instruments. Moreover, current on-patient visualization techniques typically evaluate their accuracy by measuring the registration errors of skin fiducials [2,3,5,6], the discrepancy in pixels in the image [8,10] and/or the processing time [4]. To address these drawbacks, a VSLAM-based method for on-patient visualization is proposed.…”

A review of computer-aided oral and maxillofacial surgery: planning, simulation and navigation

“…During the image-guided surgery, one of the limitations is that the surgeons have to switch between the actual operation site and computer screen which is inconvenient and impact the continuity of surgery [59]. Nowadays, with the use of wearable augmented reality (AR) devices, the real objects and virtual (computer-generated) objects are mixed in a real environment.…”

Comparison of optical see-through head-mounted displays for surgical interventions with object-anchored 2D-display

“…The classification is summarized in Table 1. Based on the classification, the use cases of HMDs for mixed reality interventions mainly fall into four categories: VST-HMD with head-anchored 2D-display [10,16,20,24,33,41], OST-HMD with head-anchored 2D-display [2,3,25,29,40], VST-HMD with object-anchored 3D-object [1,4–6,8,21], and OST-HMD with object-anchored 3D-object [7,37]. …”