Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planning

Abstract

Regrasping a suture needle is an important process in suturing, and previous study has shown that it takes on average 7.4s before the needle is thrown again. To bring efficiency into suturing, prior work either designs a taskspecific mechanism or guides the gripper toward some specific pick-up point for proper grasping of a needle. Yet, these methods are usually not deployable when the working space is changed. These prior efforts highlight the need for more efficient regrasping and more generalizability of a proposed method. Therefore, in this work, we present rapid trajectory generation for bimanual needle regrasping via reinforcement learning (RL). Demonstrations from a sampling-based motion planning algorithm is incorporated to speed up the learning. In addition, we propose the ego-centric state and action spaces for this bimanual planning problem, where the reference frames are on the end-effectors instead of some fixed frame. Thus, the learned policy can be directly applied to any robot configuration and even to different robot arms. Our experiments in simulation show that the success rate of a single pass is 97%, and the planning time is 0.0212s on average, which outperforms other widely used motion planning algorithms. For the real-world experiments, the success rate is 73.3% if the needle pose is reconstructed from an RGB image, with a planning time of 0.0846s and a run time of 5.1454s. If the needle pose is known beforehand, the success rate becomes 90.5%, with a planning time of 0.0807s and a run time of 2.8801s.

Integration of Reinforcement Learning in a Virtual Robotic Surgical Simulation

“…This study serves as a novel framework for integrating RL methods with enhanced graphical simulation for rendering tissue dissection. The work here demonstrates the feasibility of integrating of the Unity package Machine Learning-Agents 34(p) (ML-Agents) with the powerful physics simulator, NVIDIA FleX system 35 by way of the uFlex package 36 available on the Unity3D Asset store. The development and testing of the environment was conducted entirely on an MSI Aegis 3 GTX 1070 using Windows operating system.…”

From Teleoperation to Autonomous Robot-assisted Microsurgery: A Survey

“…Deep RL was employed for an optimal tensioning policy of a pinch point for minimally invasive robotic surgery [178] . More recently, a discrete reinforcement learningbased approach has been developed to automate the needle hand-off task and collaborative suturing [179] . Moreover, RL has been used for rapid trajectory generation for a bimanual needle regrasping task, which is one of the most challenging sub-task of suturing [180] .…”

Mixline: A Hybrid Reinforcement Learning Framework for Long-Horizon Bimanual Coffee Stirring Task

“…Liu et al applied dual-arm Deep Deterministic Policy Gradient (DDPG) [5] based on Deep Reinforcement Learning (DRL) with Hindsight Experience Replay (HER) [6] to achieve cooperative tasks based on "rewarding cooperation and punishing competition" [7]. Chiu et al achieved bimanual regrasping by using demonstrations from a sampling-based motion planning algorithm and generalizing for non-specific trajectory [8]. Rajeswaran et al augmented the policy search process with a small number of human demonstrations [9].…”