Investigating exploration for deep reinforcement learning of concentric tube robot control

Abstract

Purpose Concentric tube robots are composed of multiple concentric, pre-curved, super-elastic, telescopic tubes that are compliant and have a small diameter suitable for interventions that must be minimally invasive like fetal surgery. Combinations of rotation and extension of the tubes can alter the robot's shape but the inverse kinematics are complex to model due to the challenge of incorporating friction and other tube interactions or manufacturing imperfections. We propose a model-free reinforcement learning approach to form the inverse kinematics solution and directly obtain a control policy. Method Three exploration strategies are shown for deep deterministic policy gradient with hindsight experience replay for concentric tube robots in simulation environments. The aim is to overcome the joint to Cartesian sampling bias and be scalable with the number of robotic tubes. To compare strategies, evaluation of the trained policy network to selected Cartesian goals and associated errors are analyzed. The learned control policy is demonstrated with trajectory following tasks. Results Separation of extension and rotation joints for Gaussian exploration is required to overcome Cartesian sampling bias. Parameter noise and Ornstein-Uhlenbeck were found to be optimal strategies with less than 1 mm error in all simulation environments. Various trajectories can be followed with the optimal exploration strategy learned policy at high joint extension values. Our inverse kinematics solver in evaluation has 0.44 mm extension and 0.3 • rotation error. ConclusionWe demonstrate the feasibility of effective model-free control for concentric tube robots. Directly using the control policy, arbitrary trajectories can be followed and this is an important step towards overcoming the challenge of concentric tube robot control for clinical use in minimally invasive interventions.

Data-Driven Steering of Concentric Tube Robots in Unknown Environments via Dynamic Mode Decomposition

“…Data-driven controllers that learn the complex dynamics of the robot and its interaction with the environment can overcome the aforementioned difficulties. Commonly, these methods employ machine learning to learn inverse/forward kinematic or dynamic models [14], [15] or learn a direct control policy for moving the robot using Reinforcement Learning (RL) techniques with/without prior knowledge about geometric models [16]- [18]. The major disadvantages of these methods are the requirement for numerous training data.…”

“…Additionally, relying only on simulation dataset for training leads to unsatisfactory results when the model is deployed on the real robot [20]. Moreover, these models that are trained offline on experimental datasets [19] or simulated [18] datasets cannot capture the robot's behaviour in contact with the environment or external forces, as it would require a very large training dataset considering numerous robot configurations with various forces.…”

Data augmentation for design of concentric tube continuum robots by generative adversarial networks

“…For the simulation problem, several works estimate the tip's pose or the entire CTCR shape in dependency of the actuation parameters [11][12][13]. The problem of path planning or control is addressed by the inverse relation with similar methods [12,[14][15][16]. Such approaches rely on a densely sampled and valid data set of high measurement accuracy, which is time-consuming to establish [17].…”

“…Such approaches rely on a densely sampled and valid data set of high measurement accuracy, which is time-consuming to establish [17]. For that reason, some authors stick with learning other physical models [14][15][16]. Kuntz et al are using a combination of measurements and simulation data [13].…”

Toward Learning Context-Dependent Tasks from Demonstration for Tendon-Driven Surgical Robots

“…Data-driven methods have also been applied to concentric tube robots, used in learning the forward and inverse kinematics [26], [27], the complete shape [28], and in estimating tip-contact forces [29]. Further, Iyengar et al [30] leverage deep reinforcement learning to control concentric tube robots, a method distinct from LfD.…”