Discrete soft actor-critic with auto-encoder on vascular robotic system

Abstract

Instrument delivery is critical part in vascular intervention surgery. Due to the soft-body structure of instruments, the relationship between manipulation commands and instrument motion is non-linear, making instrument delivery challenging and time-consuming. Reinforcement learning has the potential to learn manipulation skills and automate instrument delivery with enhanced success rates and reduced workload of physicians. However, due to the sample inefficiency when using high-dimensional images, existing reinforcement learning algorithms are limited on realistic vascular robotic systems. To alleviate this problem, this paper proposes discrete soft actor-critic with auto-encoder (DSAC-AE) that augments SAC-discrete with an auxiliary reconstruction task. The algorithm is applied with distributed sample collection and parameter update in a robot-assisted preclinical environment. Experimental results indicate that guidewire delivery can be automatically implemented after 50k sampling steps in less than 15 h, demonstrating the proposed algorithm has the great potential to learn manipulation skill for vascular robotic systems.

Recurrent neural networks for generalization towards the vessel geometry in autonomous endovascular guidewire navigation in the aortic arch

“…For each aortic arch the pathlength is calculated based on the centerlines of the individual geometry. This is an adaption of the reward utilized in [ 12 ] enhanced by the dense feature from [ 14 ] and [ 15 ]. Parameters for the constant penalty and change in pathlength are chosen such that they approximately equalize each other with an optimal action.…”

“…Recent research regarding autonomous control of endovascular instruments during navigation in interventions utilize learning-based approaches [ 8 – 15 ]. They utilize supervised learning [ 8 ], deep-q-networks [ 9 , 10 ], asynchronous advantage actor-critic [ 11 ], deep deterministic policy gradients with hindsight experience replay [ 12 ], generative adversarial imitation learning and proximal policy optimization [ 13 ], soft actor critic [ 14 ] or discrete soft actor critic with auto-encoder [ 15 ]. These approaches are all trained on one vessel geometry and show reduced success [ 13 ] on a different geometry.…”

A Survey on Deep Reinforcement Learning Algorithms for Robotic Manipulation

“…Reinforcement learning is used by Li et al [ 148 ] to solve the problem of automating instrument delivery in vascular intervention surgery. Existing reinforcement learning techniques are constrained by the non-linear connection between manipulation commands and instrument movements in actual vascular robotic systems.…”