Tactile Gym 2.0: Sim-to-Real Deep Reinforcement Learning for Comparing Low-Cost High-Resolution Robot Touch

Abstract

High-resolution tactile sensing can provide accurate information about local contact in contact-rich robotic tasks. However, the deployment of such tasks in unstructured environments remains under-investigated. To improve the robustness of tactile robot control in unstructured environments, we propose and study a new concept: tactile saliency for robot touch, inspired by the human touch attention mechanism from neuroscience and the visual saliency prediction problem from computer vision. In analogy to visual saliency, this concept involves identifying key information in tactile images captured by a tactile sensor. While visual saliency datasets are commonly annotated by humans, manually labelling tactile images is challenging due to their counterintuitive patterns. To address this challenge, we propose a novel approach comprised of three interrelated networks: 1) a Contact Depth Network (ConDepNet), which generates a contact depth map to localize deformation in a real tactile image that contains target and noise features; 2) a Tactile Saliency Network (TacSalNet), which predicts a tactile saliency map to describe the target areas for an input contact depth map; 3) and a Tactile Noise Generator (TacNGen), which generates noise features to train the TacSalNet. Experimental results in contact pose estimation and edge-following in the presence of distractors showcase the accurate prediction of target features from real tactile images. Overall, our tactile saliency prediction approach gives robust sim-to-real tactile control in environments with unknown distractors. Project page: https://sites.google.com/ view/tactile-saliency/.

Advancements in Deep Reinforcement Learning and Inverse Reinforcement Learning for Robotic Manipulation: Toward Trustworthy, Interpretable, and Explainable Artificial Intelligence

“…Bi-Touch: Bimanual Tactile Manipulation with Sim-to-Real Deep Reinforcement Learning [86] Due to the complexity of designing effective controllers, bilateral manipulation with haptic feedback is less explored than single-handed manipulation.…”

Manipulación visual-táctil para la recogida de residuos domésticos en exteriores

“…Por ejemplo, en (Kolamuri et al, 2021) realizan un estudio para compensar la rotación indeseada de un objeto a partir de las imágenes táctiles de sensores Gelsight, mediante técnicas tradicionales de visión por computador. Otro ejemplo sería el trabajo de (Lin et al, 2022) donde aplican técnicas de aprendizaje por refuerzo para llevar a cabo diferentes tareas de manipulación con sensores táctiles como Gelsight, Digit o Tactip. En este trabajo se pretende continuar con la tendencia de uso de los sensores táctiles de bajo coste basados en imagen, ya que aportan más información sobre las características de los objetos que los sensores de fuerza o presión, ya que tienen una resolución espacial mayor.…”

ChinaSpec: a network for long-term in situ measurements of solar-induced fluorescence and reflectance in China

“…Nevertheless, the physical properties of the tactile sensor are neglected. Lin et al [25] employed an image-to-image translation GAN [26] to accomplish the sim-to-real transfer. Nonetheless, they solely evaluated the zero-shot performance in basic scenarios such as edge-following and surface-following.…”