Deep Reinforcement Learning for Concentric Tube Robot Control with a Goal-Based Curriculum

Abstract

Concentric Tube Robots (CTRs), a type of continuum robot, are a collection of concentric, pre-curved tubes composed of super elastic nickel titanium alloy. CTRs can bend and twist from the interactions between neighboring tubes causing the kinematics and therefore control of the end-effector to be very challenging to model. In this paper, we develop a control scheme for a CTR end-effector in Cartesian space with no prior kinematic model using a deep reinforcement learning (DRL) approach with a goal-based curriculum reward strategy. We explore the use of curricula by changing the goal tolerance through training with constant, linear and exponential decay functions. Also, relative and absolute joint representations as a way of improving training convergence are explored. Quantitative comparisons for combinations of curricula and joint representations are performed and the exponential decay relative approach is used for training a robust policy in a noise-induced simulation environment. Compared to a previous DRL approach, our new method reduces training time and employs a more complex simulation environment. We report mean Cartesian errors of 1.29 mm and a success rate of 0.93 with a relative decay curriculum. In path following, we report mean errors of 1.37 mm in a noise-induced path following task. Albeit in simulation, these results indicate the promise of using DRL in model free control of continuum robots and CTRs in particular.

Generalization for Deep Reinforcement Learning for Inverse Kinematics of Concentric Tube Robots

“…Using deep deterministic policy gradient (DDPG) [6] with hindsight experience replay (HER) [7], the training parameters are as follows. The number of training timesteps was 3 million, buffer size was 500, 000 with the policy network having 3 hidden networks with 256 units per layer, the initial goal tolerance and final goal tolerance were 20 mm and 1 mm applied over 1.5 million steps using a decay function [3]. Zero-mean Gaussian noise of 1.8 mm was applied to Δ𝛽 𝑖 and 0.025 radians to Δ𝛼 𝑖 .…”

“…This motivates the development of an end-to-end model-free control framework for CTRs. We extend our previous model-free deep reinforcement learning (deepRL) method [3] with an initial proof of concept for generalization. The task we give the agent then is to control the end-effector Cartesian robot tip position by means of actions that represent changes in joint values to reach a desired position in the robot workspace whilst considering a specific CTR system.…”

Data augmentation for design of concentric tube continuum robots by generative adversarial networks

“…For the simulation problem, several works estimate the tip's pose or the entire CTCR shape in dependency of the actuation parameters [11][12][13]. The problem of path planning or control is addressed by the inverse relation with similar methods [12,[14][15][16]. Such approaches rely on a densely sampled and valid data set of high measurement accuracy, which is time-consuming to establish [17].…”

“…Such approaches rely on a densely sampled and valid data set of high measurement accuracy, which is time-consuming to establish [17]. For that reason, some authors stick with learning other physical models [14][15][16]. Kuntz et al are using a combination of measurements and simulation data [13].…”

A Dataset and Benchmark for Learning the Kinematics of Concentric Tube Continuum Robots

“…For learning the kinematics of a CTCR task, we show in [4] that the accuracy and convergence are significantly improved by using this simple yet effective transformation to decorrelate the joint space. As of now, this approach has been used in publications [4], [5], [6] on machine learning. Yet, we are confident that disentanglement will also be useful for other research fields in the continuum robotics research community.…”