Sim-to-Real Transfer for Robotic Manipulation with Tactile Sensory

In this paper, we propose a novel framework for tactile-based dexterous manipulation learning with a blind anthropomorphic robotic hand, i.e. without visual sensing. First, object-related states were extracted from the raw tactile signals by a graph-based perception model -TacGNN. The resulting tactile features were then utilized in the policy learning of an in-hand manipulation task in the second stage. This method was examined by a Baoding ball task -simultaneously manipulating two spheres around each other by 180 degrees in hand. We conducted experiments on object states prediction and in-hand manipulation using a reinforcement learning algorithm (PPO). Results show that TacGNN is effective in predicting objectrelated states during manipulation by decreasing the RMSE of prediction to 0.096cm comparing to other methods, such as MLP, CNN, and GCN. Finally, the robot hand could finish an in-hand manipulation task solely relying on the robotic own perception -tactile sensing and proprioception. In addition, our methods are tested on three tasks with different difficulty levels and transferred to the real robot without further training. https://sites.google.com/view/tacgnn

show abstract

“…All the fingers and the palm have been covered with our in-house custom-made resistive tactile sensors [28]. Fig.…”

Section: Experiments a Allegro Hand With Distributed Sensorsmentioning

confidence: 99%

TacGNN: Learning Tactile-Based In-Hand Manipulation With a Blind Robot Using Hierarchical Graph Neural Network

Yang

Huang

et al. 2023

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

show abstract

“…There is a sim-to-real gap [6,7,8] when grasps are trained in simulation. Robustness in these simulation settings has been achieved through domain randomization or general Monte-Carlo sampling [42,43,44].…”

Section: Robust Graspingmentioning

confidence: 99%

“…Instead of using a physics simulator, we propose to collect data entirely on a physical robot and use it to train a multi-object grasping function, MOG-Net, which is robust to state and control uncertainty and predicts the number of objects that will be grasped out of a target object group. We train in real to avoid the sim-to-real gap [6,7,8]. We also propose a necessary condition for frictional multi-object grasping to filter out inadmissible grasps and show that this 1 University of California, Berkeley, USA.…”

Section: Introductionmentioning

confidence: 99%

Learning to Efficiently Plan Robust Frictional Multi-Object Grasps

Agboh,

Sharma,

Srinivas

et al. 2023

2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

We consider a decluttering problem where multiple rigid convex polygonal objects rest in randomly placed positions and orientations on a planar surface and must be efficiently transported to a packing box using both single and multi-object grasps. Prior work considered frictionless multi-object grasping. In this paper, we introduce friction to increase the number of potential grasps for a given group of objects, and thus increase picks per hour. We train a neural network using real examples to plan robust multi-object grasps. In physical experiments, we find a 13.7% increase in success rate, a 1.6x increase in picks per hour, and a 6.3x decrease in grasp planning time compared to prior work on multi-object grasping. Compared to single-object grasping, we find a 3.1x increase in picks per hour.

show abstract

“…Ho et al proposed RetinaGAN, a GAN-based approach to achieve consistency in object detection when adapting simulated images to realistic ones [92]. Ding et al investigated a sim-to-real approach for incorporating tactile sensing into RL for tasks involving contact-rich interactions [93], while Lee et al studyed the problem of robotic stacking with complex objects and propose a set of challenging objects intended to necessitate sophisticated techniques beyond basic pick-and-place methods [94]. Pedersen et al proposed a method to transfer a grasping agent trained with DRL from a simulated environment to a physical robot [95].…”

Section: Simulation-to-real-world Transfermentioning

confidence: 99%

Review of Reinforcement Learning for Robotic Grasping: Analysis and Recommendations

Sekkat,

Moutik,

Ourabah

et al. 2023

Stat., optim. inf. comput.

View full text Add to dashboard Cite

This review paper provides a comprehensive analysis of over 100 research papers focused on the challenges of robotic grasping and the effectiveness of various machine learning techniques, particularly those utilizing Deep Neural Networks (DNNs) and Reinforcement Learning (RL). The objective of this review is to simplify the research process for others by gathering different forms of Deep Reinforcement Learning (DRL) grasping tasks in one place. Through a thorough analysis of the literature, the study emphasizes the critical nature of grasping for robots and how DRL techniques, particularly the Soft-Actor-Critic (SAC) strategy, have demonstrated high efficiency in handling the task. The results of this study hold significant implications for the development of more advanced and efficient grasping systems for robots. Continued research in this area is crucial to further enhance the capabilities of robots in handling complex and challenging tasks, such as grasping.

show abstract

Sim-to-Real Transfer for Robotic Manipulation with Tactile Sensory

Cited by 10 publications

References 27 publications

TacGNN: Learning Tactile-Based In-Hand Manipulation With a Blind Robot Using Hierarchical Graph Neural Network

TacGNN: Learning Tactile-Based In-Hand Manipulation With a Blind Robot Using Hierarchical Graph Neural Network

Learning to Efficiently Plan Robust Frictional Multi-Object Grasps

Review of Reinforcement Learning for Robotic Grasping: Analysis and Recommendations

Contact Info

Product

Resources

About