Deep Reinforcement Learning for Tactile Robotics: Learning to Type on a Braille Keyboard

Abstract

Artificial touch would seem well-suited for Reinforcement Learning (RL), since both paradigms rely on interaction with an environment. Here we propose a new environment and set of tasks to encourage development of tactile reinforcement learning: learning to type on a braille keyboard. Four tasks are proposed, progressing in difficulty from arrow to alphabet keys and from discrete to continuous actions. A simulated counterpart is also constructed by sampling tactile data from the physical environment. Using state-of-the-art deep RL algorithms, we show that all of these tasks can be successfully learnt in simulation, and 3 out of 4 tasks can be learned on the real robot. A lack of sample efficiency currently makes the continuous alphabet task impractical on the robot. To the best of our knowledge, this work presents the first demonstration of successfully training deep RL agents in the real world using observations that exclusively consist of tactile images. To aid future research utilising this environment, the code for this project has been released along with designs of the braille keycaps for 3D printing and a guide for recreating the experiments. A brief video summary is also available at https://youtu.be/eNylCA2uE_E.

Soft Biomimetic Optical Tactile Sensing with the TacTip: A Review

“…SoftBOT hands (middle row): tactile Model-M2 [69], tactile Model-GR2 [71], tactile Model-O [50], tactile Shadow Modular Grasper [74] and tactile SoftHand [30]. SoftBOT systems (bottom row): 3D-printed TacTip on ABB robot arm for tactile servo control [46], [77], TacTip on UR5 arm for reinforcement learning [75], TacFoot on walking robot [76], tactile Model-O on UR5 robot arm for grasping and slip detection [36], [50] and tacWhisker mounted on ABB robot arm [73].…”

“…New tactile capabilities developed with the TacTip include: (a) pose-based servo control, where a tactile fingertip mounted on a robot arm slides delicately over unknown complex 3D objects (Figure 5a; [77], [86], [87]); (b) pushing manipulation of unknown objects using only tactile sensing and proprioceptive knowledge of where the sensor is positioned relative to a goal location (Figure 5b; [88]); (c) acquiring the novel skill of single-fingered typing on a braille keyboard, learning the identity of keys and how to navigate the keyboard from touch (Figure 5c [86], [87] tactile & proprioceptive feedback [88] Learning to type Braille [75] (d) Object & grasp-success prediction (e) In-hand tactile manipulation (f) Fine control of contact with an underactuated tactile hand [50] for fully-actuated stable grasping with an anthropomorphic tactile hand For all examples, the use of convolutional neural networks was critical for the tactile capability to be reached. configuration with a fully-actuated tactile Shadow Modular Grasper (Figure 5e; [84]); (f) fine control of contact onto unknown objects placed in-hand using an anthropomorphic tactile hand based on the Pisa/IIT SoftHand (Figure 5f; [30]).…”

“…However, a complementary approach is to use tactile deep RL with a reward signal that indicates the desired interactions with a physical environment. The third task (Figure 5d) successfully applied this approach to learning to type with a braille keyboard [75], leading to a policy network that could efficiently navigate the keyboard to press a desired key. Our expectation is that all considered tasks could also be learnt using deep RL, but for sample efficiency much of this learning should take place in a simulated tactile environment [99].…”

Deep Reinforcement Learning for Guidewire Navigation in Coronary Artery Phantom

“…First, under the constraints of the discrete action space and training in a real-world setting, Rainbow [23] was applied (Figure 1a). Rainbow integrating Deep Q-Networks (DQN) [14] with recent advancements [24]- [28] in reinforcement learning has demonstrated outstanding performance in real world environments [29]. Replay memory [14], a representative an off-policy method to enhance sample efficiency and increase the training speed with quality data, was a key component to reduce the physical time requirement.…”

Using Tactile Sensing to Improve the Sample Efficiency and Performance of Deep Deterministic Policy Gradients for Simulated In-Hand Manipulation Tasks

“…The authors showed how the learning of tactile feedback can be made more efficient by reducing the dimensionality of the tactile information through spectral clustering and principal component analysis. Another study Church et al (2020) presented a challenging tactile robotics environment for learning to type on a braille keyboard with deep reinforcement learning algorithms. Preliminary results showed that successful learning can take place directly on a physical robot equipped with a biometric tactile sensor.…”