Exoskeleton-covered soft finger with vision-based proprioception and tactile sensing

Abstract

Soft robots offer significant advantages in adaptability, safety, and dexterity compared to conventional rigidbody robots. However, it is challenging to equip soft robots with accurate proprioception and exteroception due to their high flexibility and elasticity. In this work, we develop a novel exoskeleton-covered soft finger with embedded cameras and deep learning methods that enable high-resolution proprioceptive sensing and rich tactile sensing. To do so, we design features along the axial direction of the finger, which enable highresolution proprioceptive sensing, and incorporate a reflective ink coating on the surface of the finger to enable rich tactile sensing. We design a highly underactuated exoskeleton with a tendon-driven mechanism to actuate the finger. Finally, we assemble 2 of the fingers together to form a robotic gripper and successfully perform a bar stock classification task, which requires both shape and tactile information. We train neural networks for proprioception and shape (box versus cylinder) classification using data from the embedded sensors. The proprioception CNN had over 99% accuracy on our testing set (all six joint angles were within 1 • of error) and had an average accumulative distance error of 0.77 mm during live testing, which is better than human finger proprioception. These proposed techniques offer soft robots the high-level ability to simultaneously perceive their proprioceptive state and peripheral environment, providing potential solutions for soft robots to solve everyday manipulation tasks. We believe the methods developed in this work can be widely applied to different designs and applications.

GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot Finger

“…It can be difficult to place all 3 lights for some devices, e.g. GelFlex [27], where one axis of the sensor is too long and deformable for directional lights to travel. We aim to still estimate 3D geometries with the limited lights.…”

Learning to Prevent Grasp Failure with Soft Hands: From Online Prediction to Dual‐Arm Grasp Recovery

“…As an alternative to force sensing, the community is recently exploring the usage of other sensory sources, such as audio signals, [17] inertial sensing, [18] video streams, [19] and tactile sensors, [20,21] and infer contact forces through algorithms. However, little has been done so far to exploit such sensory information to predict when a grasp is going to fail, and to trigger reactive recovery primitives.…”