Learning Generalizable Dexterous Manipulation from Human Grasp Affordance

Wu, Yueh-Hua; Wang, Jiashun; Wang, Xiaolong

doi:10.48550/arxiv.2204.02320

Cited by 3 publications

(12 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For dexterous object manipulation, using natural hand poses has a significant impact on the success of natural object grasping. Several previous studies have conducted dexterous object manipulation with hand pose estimation and DRL [32,41]. Yueh-Hua Wu et al [32] used GraspCVAE and DRL to manipulate objects with an ADROIT robot hand.…”

Section: Discussionmentioning

confidence: 99%

“…Several previous studies have conducted dexterous object manipulation with hand pose estimation and DRL [32,41]. Yueh-Hua Wu et al [32] used GraspCVAE and DRL to manipulate objects with an ADROIT robot hand. GraspCVAE is based on a variational autoencoder that estimates the natural hand pose from the object affordance.…”

Section: Discussionmentioning

confidence: 99%

“…Recently, reinforcement learning approaches were introduced to train an anthropomorphic robot hand with the estimated natural hand poses [18,32]. Furthermore, Demo Augmented Policy Gradient (DAPG) was introduced [33].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Dexterous Object Manipulation with an Anthropomorphic Robot Hand via Natural Hand Pose Transformer and Deep Reinforcement Learning

Lopez

Jeong

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

Dexterous object manipulation using anthropomorphic robot hands is of great interest for natural object manipulations across the areas of healthcare, smart homes, and smart factories. Deep reinforcement learning (DRL) is a particularly promising approach to solving dexterous manipulation tasks with five-fingered robot hands. Yet, controlling an anthropomorphic robot hand via DRL in order to obtain natural, human-like object manipulation with high dexterity remains a challenging task in the current robotic field. Previous studies have utilized some predefined human hand poses to control the robot hand’s movements for successful object-grasping. However, the hand poses derived from these grasping taxonomies are limited to a partial range of adaptability that could be performed by the robot hand. In this work, we propose a combinatory approach of a deep transformer network which produces a wider range of natural hand poses to configure the robot hand’s movements, and an adaptive DRL to control the movements of an anthropomorphic robot hand according to these natural hand poses. The transformer network learns and infers the natural robot hand poses according to the object affordance. Then, DRL trains a policy using the transformer output to grasp and relocate the object to the designated target location. Our proposed transformer-based DRL (T-DRL) has been tested using various objects, such as an apple, a banana, a light bulb, a camera, a hammer, and a bottle. Additionally, its performance is compared with a baseline DRL model via natural policy gradient (NPG). The results demonstrate that our T-DRL achieved an average manipulation success rate of 90.1% for object manipulation and outperformed NPG by 24.8%.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Dexterous Object Manipulation with an Anthropomorphic Robot Hand via Natural Hand Pose Transformer and Deep Reinforcement Learning

Lopez

Jeong

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…DAPG [1] exploits human demonstrations collected in virtual reality to improve sample efficiency of DRL in a high-dimensional 30 DoFs hand space. Following the same human demonstration strategy, ILAD [2], GRAFF [3], and DexVIP [4] extends the demonstrations into larger scale trajectories or grasping poses collected from human interaction with real daily objects. GRAFF [3] leverages contacts from hand-object interactions for agents to learn to approach objects more effectively.…”

Section: Introductionmentioning

confidence: 99%

“…DexVIP [4] learns human pose priors from videos and imposes these priors into DRL by incorporating auxiliary reward functions favoring robot poses similar to the human ones in videos. ILAD [2] trains a generator to synthesize grasping trajectories with largescale demonstrations instead of using human demonstrations directly.…”

Section: Introductionmentioning

confidence: 99%

DexRepNet: Learning Dexterous Robotic Grasping Network with Geometric and Spatial Hand-Object Representations

Liu¹,

Chen²,

Sun³

et al. 2023

Preprint

View full text Add to dashboard Cite

Robotic dexterous grasping is a challenging problem due to the high degree of freedom (DoF) and complex contacts of multi-fingered robotic hands. Existing deep reinforcement learning (DRL) based methods leverage human demonstrations to reduce sample complexity due to the high dimensional action space with dexterous grasping. However, less attention has been paid to hand-object interaction representations for high-level generalization. In this paper, we propose a novel geometric and spatial hand-object interaction representation, named DexRep, to capture dynamic object shape features and the spatial relations between hands and objects during grasping. DexRep comprises Occupancy Feature for rough shapes within sensing range by moving hands, Surface Feature for changing hand-object surface distances, and Local-Geo Feature for local geometric surface features most related to potential contacts. Based on the new representation, we propose a dexterous deep reinforcement learning method to learn a generalizable grasping policy DexRepNet. Experimental results show that our method outperforms baselines using existing representations for robotic grasping dramatically both in grasp success rate and convergence speed. It achieves a 93% grasping success rate on seen objects and higher than 80% grasping success rates on diverse objects of unseen categories in both simulation and real-world experiments.

show abstract

One Policy to Dress Them All: Learning to Dress People with Diverse Poses and Garments

Wang,

Sun,

Erickson

et al. 2023

Robotics: Science and Systems XIX

View full text Add to dashboard Cite

Learning Generalizable Dexterous Manipulation from Human Grasp Affordance

Cited by 3 publications

References 40 publications

Dexterous Object Manipulation with an Anthropomorphic Robot Hand via Natural Hand Pose Transformer and Deep Reinforcement Learning

Dexterous Object Manipulation with an Anthropomorphic Robot Hand via Natural Hand Pose Transformer and Deep Reinforcement Learning

DexRepNet: Learning Dexterous Robotic Grasping Network with Geometric and Spatial Hand-Object Representations

One Policy to Dress Them All: Learning to Dress People with Diverse Poses and Garments

Contact Info

Product

Resources

About