Stereo vision-based tracking of soft tissue motion with application to online ablation control in laser microsurgery

Abstract

Recent research has revealed that image-based methods can enhance accuracy and safety in laser microsurgery. In this study, non-rigid tracking using surgical stereo imaging and its application to laser ablation is discussed. A recently developed motion estimation framework based on piecewise affine deformation modeling is extended by a mesh refinement step and considering texture information. This compensates for tracking inaccuracies potentially caused by inconsistent feature matches or drift. To facilitate online application of the method, computational load is reduced by concurrent processing and affine-invariant fusion of tracking and refinement results. The residual latency-dependent tracking error is further minimized by Kalman filter-based upsampling, considering a motion model in disparity space. Accuracy is assessed in laparoscopic, beating heart, and laryngeal sequences with challenging conditions, such as partial occlusions and significant deformation. Performance is compared with that of state-of-the-art methods. In addition, the online capability of the method is evaluated by tracking two motion patterns performed by a high-precision parallel-kinematic platform. Related experiments are discussed for tissue substitute and porcine soft tissue in order to compare performances in an ideal scenario and in a setup mimicking clinical conditions. Regarding the soft tissue trial, the tracking error can be significantly reduced from 0.72 mm to below 0.05 mm with mesh refinement. To demonstrate online laser path adaptation during ablation, the non-rigid tracking framework is integrated into a setup consisting of a surgical Er:YAG laser, a three-axis scanning unit, and a low-noise stereo camera. Regardless of the error source, such as laser-to-camera registration, camera calibration, image-based tracking, and scanning latency, the ablation root mean square error is kept below 0.21 mm when the sample moves according to the aforementioned patterns. Final experiments regarding motion-compensated laser ablation of structurally deforming tissue highlight the potential of the method for vision-guided laser surgery.

A dataset of laryngeal endoscopic images with comparative study on convolution neural network-based semantic segmentation

“…It has been shown that an ensemble of ErfNet and UNet yielded the most successful results on segmenting endoscopic image data. Schoob et al [30] reported that image-based online control of incision lasers for soft tissue undergoing motion is feasible with an imaging pipeline running at 73.5 Hz. Therefore, both ENet and ErfNet are well-suited in terms of efficiency for later use in autonomous or robot aided interventions.…”

μRALP and Beyond: Micro-Technologies and Systems for Robot-Assisted Endoscopic Laser Microsurgery

“…Subsequently, this system was integrated into the controller of a new robotic endoscope for non-contact FIGURE 12 | Real-time stereoscopic methods for (A) 3D reconstruction (Schoob et al, 2015). (B) intraoperative incision planning and visualization (Schoob et al, 2017), and (C) laser focus adjustment (Schoob et al, 2016 endolaryngeal laser surgery, enabling novel visualization concepts for eye-in-hand laser manipulation, as demonstrated in a user study with novice and expert users (Kundrat et al, 2020).…”

“… Real-time stereoscopic methods for (A) 3D reconstruction ( Schoob et al, 2015 ). (B) intraoperative incision planning and visualization ( Schoob et al, 2017 ), and (C) laser focus adjustment ( Schoob et al, 2016 ). …”

Preclinical Performance Evaluation of a Robotic Endoscope for Non-Contact Laser Surgery

“…25 Estimation of scene motion and deformation is implemented with a tracking algorithm that is optimised for soft tissues with errors of less than 0.1 mm. 26 The robotic eye-inhand setup with rigid laser optics integrated to the tip causes endoscopic visualisation with scene motion while displacing the laser spot. This is different from familiar scenarios in transoral laser surgery, where the beam is displaced on a static scene in the microscopic view.…”