Real-Time 3D Tracking of Articulated Tools for Robotic Surgery

Abstract

Abstract. In robotic surgery, tool tracking is important for providing safe tool-tissue interaction and facilitating surgical skills assessment. Despite recent advances in tool tracking, existing approaches are faced with major difficulties in real-time tracking of articulated tools. Most algorithms are tailored for offline processing with pre-recorded videos. In this paper, we propose a real-time 3D tracking method for articulated tools in robotic surgery. The proposed method is based on the CAD model of the tools as well as robot kinematics to generate online part-based templates for efficient 2D matching and 3D pose estimation. A robust verification approach is incorporated to reject outliers in 2D detections, which is then followed by fusing inliers with robot kinematic readings for 3D pose estimation of the tool. The proposed method has been validated with phantom data, as well as ex vivo and in vivo experiments.The results derived clearly demonstrate the performance advantage of the proposed method when compared to the state-of-the-art.

SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation With Surgical Robotics

“…3) Endoscopic Surgical Tool Tracking and Control: A recent literature survey by Bouget et al [20] gave a detailed summary of image-based surgical tool detection. Markerless with tracking algorithms [21], [22], [23] requires features which can be learned [24], [25], generated via template matching [26], or hand-crafted [27]. After the features have been extracted, they are fused with kinematic information and encoder readings to fully localize the surgical robotic tools [28].…”

“…To track the surgical tools accurately, T b− b will be estimated in real-time. Similar problem formulations have been utilized in prior works for surgical tool tracking [24], [25], [26].…”

Augmented Reality Predictive Displays to Help Mitigate the Effects of Delayed Telesurgery

“…Ye et al previously developed a tacking algorithm for the slave-tools and showed it to be accurate and robust to surgical environments [22]. It successfully tracked the slavetools in real-time by estimating the error of the initial handeye calibrations between the base of both slave arms and the left camera in real-time by using: 1) virtual slave-tool rendering to generate part-based templates online, 2) template matching between virtual slave-tool parts and image data, 3) geometric context to extract the best location estimates of the slave-tool parts, and 4) EKF to track the error of the hand-eye transform using the previously found 2D estimates for the update.…”

“…To initialize the EKF, the hand-eye transform from calibration completes steps 1 through 3, and instead of step 4, a Perspective-n-Point (PnP) solver on the detected features is applied [22]. To improve robustness, this is repeated when the distance or absolute difference in roll, pitch, or yaw between the corrected hand-eye and initial hand-eye from calibration is greater than a set threshold.…”

Implicit gaze-assisted adaptive motion scaling for highly articulated instrument manipulation

“…As the instrument shape can change significantly depending on its kinematic configuration and cause the detector to fail, a tracker was included to consider the temporal information of the tool motion. The salient image gradient features [16] were extracted inside a bounding box that encloses the end-effector. As can be seen from Fig.…”