Design and Testing of Augmented Reality-Based Fluorescence Imaging Goggle for Intraoperative Imaging-Guided Surgery

Abstract

The different pathways between the position of a near-infrared camera and the user’s eye limit the use of existing near-infrared fluorescence imaging systems for tumor margin assessments. By utilizing an optical system that precisely matches the near-infrared fluorescence image and the optical path of visible light, we developed an augmented reality (AR)-based fluorescence imaging system that provides users with a fluorescence image that matches the real-field, without requiring any additional algorithms. Commercial smart glasses, dichroic beam splitters, mirrors, and custom near-infrared cameras were employed to develop the proposed system, and each mount was designed and utilized. After its performance was assessed in the laboratory, preclinical experiments involving tumor detection and lung lobectomy in mice and rabbits by using indocyanine green (ICG) were conducted. The results showed that the proposed system provided a stable image of fluorescence that matched the actual site. In addition, preclinical experiments confirmed that the proposed system could be used to detect tumors using ICG and evaluate lung lobectomies. The AR-based intraoperative smart goggle system could detect fluorescence images for tumor margin assessments in animal models, without disrupting the surgical workflow in an operating room. Additionally, it was confirmed that, even when the system itself was distorted when worn, the fluorescence image consistently matched the actual site.

Low-cost augmented reality goggles enable precision fluorescence-guided cancer surgery

“…This dramatically simplifies the FGS system by removing the need for additional hardware and complex software associated with distance-specific camera signal transformation. A similar approach was introduced by Lee et al, but unlike our system, their work was limited by not having a coaligned visible camera feed 32 .…”

“…We overcame this limitation by modifying the Rokid Air to display the monocular signal to only one eye (Figure S15). We are not aware of a similar modification working with other off-the-shelf augmented reality glasses, and previously published non-stereoscopic FGS systems do not address methods for overcoming this double vision limitation 32 .…”

Global health open-source goggles for fluorescence-guided surgery

“…We previously introduced the GAINS system as an ergonomic real-time FGS system that overcame limitations of bulky, expensive, cart-based fluorescence imaging guidance systems by reducing cost, complexity, and hardware footprint( 17, 18, 29 ). Other groups have introduced complementary innovations, including multispectral sensors that obviate the need for separate white light and NIR cameras( 19, 25 ), stereoscopic multimodality systems( 21, 22, 27, 30 ), and optical methods for ensuring coalignment between the surgeon’s eyes and camera FOV( 31 ). Despite the innovations introduced by the GAINS and other more recent systems, FGS HMD systems continue to be wall-powered, complex, expensive, relatively bulky, and replete with significant mismatch between the camera and the surgeon’s view, which makes their adoption in low-resource settings more challenging.…”

Global health open-source goggles for fluorescence-guided surgery

“…We previously introduced the GAINS system as an ergonomic real-time FGS system that overcame limitations of bulky, expensive, cart-based uorescence imaging guidance systems by reducing cost, complexity, and hardware footprint 17,18,28 . Other groups have introduced complementary innovations, including multispectral sensors that obviate the need for separate white light and NIR cameras 19,29 , stereoscopic multimodality systems 21,22,26,30 , and optical methods for ensuring coalignment between the surgeon's eyes and camera FOV 31 . Despite the innovations introduced by the GAINS and other more recent systems, FGS HMD systems continue to be wall-powered, complex, expensive, relatively bulky, and replete with signi cant mismatch between the camera and the surgeon's view, which makes their adoption in low-resource settings more challenging.…”