Learning to Localize, Grasp, and Hand Over Unmodified Surgical Needles

Abstract

Robotic Surgical Assistants (RSAs) are commonly used to perform minimally invasive surgeries by expert surgeons. However, long procedures filled with tedious and repetitive tasks such as suturing can lead to surgeon fatigue, motivating the automation of suturing. As visual tracking of a thin reflective needle is extremely challenging, prior work has modified the needle with nonreflective contrasting paint. As a step towards automation of a suturing subtask without modifying the needle, we propose HOUSTON: Handoff of Unmodified, Surgical, Tool-Obstructed Needles, a problem and algorithm that uses a learned active sensing policy with a stereo camera to localize and align the needle into a visible and accessible pose for the other arm. To compensate for robot positioning and needle perception errors, the algorithm then executes a high-precision grasping motion that uses multiple cameras. In physical experiments using the da Vinci Research Kit (dVRK), HOUSTON successfully passes unmodified surgical needles with a success rate of 96.7% and is able to perform handover sequentially between the arms 32.4 times on average before failure. On needles unseen in training, HOUSTON achieves a success rate of 75 − 92.9%. To our knowledge, this work is the first to study handover of unmodified surgical needles. See https://tinyurl.com/houston-surgery for additional materials.

All You Need is LUV: Unsupervised Collection of Labeled Images using Invisible UV Fluorescent Indicators

“…This is commonly applied when training dynamics models that predict the resulting state after an action, both on images [7,10,16,47] and lower-dimensional state such as keypoints [19,29]. Recently, self-supervision has been applied in imitation learning to obtain ground-truth action labels for image-based policies by manually resetting the robot to a goal or known configuration, perturbing the end effector by a known displacement, and using the displacement with the initial pose to compute an action label [8,27,46]. Self-supervision is also a popular technique in reinforcement learning when automatic resets are readily available.…”

“…LUV is most similar to Qian et al [33], who use visible markers to label images for a network that predicts cloth features from depth images alone. In constrast to this work, LUV can be used to label RGB images, which is useful in tasks such as needle segmentation where active depth sensors tend to fail [46].…”

“…Segmenting surgical needles in images is a common perception task in surgical robotics research. Some prior works rely on painting the needle and performing color segmentation [3,40,46] or assuming that the rest of the scene is visually distinct from the needle [44]. Recent work uses fully-convolutional neural networks to predict these masks, using a combination of simulation and real data [46], but restrict their task to black backgrounds.…”

“…Supervised learning is a popular technique for training perception and planning systems for robots, with encouraging results in applications such as autonomous driving [13,23,36], robot object grasping [5,11,17,23,28], deformable manipulation [12,15,24,38,43,48,49], and robot-assisted surgery [19,31,46]. Supervised learning requires labeled data, and a common approach is for humans to handlabel images with segmentation masks, keypoints, and class labels [15,20,46]. However this is time-consuming, errorprone, and expensive [36], especially when dense or 3-D annotations are required [5,11,12,23,43].…”

“…To quantitatively measure these properties, we apply LUV to 3 real-world perception tasks in robot manipulation: locating fabric keypoints [12,23,38,39], cable segmentation [43], and surgical needle segmentation [40,44,46]. For each, we report the speed of data collection, qualitative invisibility of markings, transferability to unpainted images, and correspondence to human ground truth labels.…”