Joint Hand-Object Pose Estimation with Differentiably-Learned Physical Contact Point Analysis

Abstract

Hand-object pose estimation aims to jointly estimate 3D poses of hands and the held objects. During the interaction between hands and objects, the position and motion of keypoints in hands and objects are tightly related and there naturally exist some physical restrictions, which is usually ignored by most previous methods. To address this issue, we propose a learnable physical affinity loss to regularize the joint estimation of hand and object poses. The physical constraints mainly focus on enhancing the stability of grasping, which is the most common interaction manner between hands and objects. Together with the physical affinity loss, a context-aware graph network is also proposed to jointly learn independent geometry prior and interaction messages. The whole pipeline consists of two components. First an image encoder is used to predict 2D keypoints from RGB image and then a contextual graph module is designed to convert 2D keypoints into 3D estimations. Our graph module treats the keypoints of hands and objects as two sub-graphs and estimates initial 3D coordinates according to their topology structure separately. Then the two sub-graphs are merged into a whole graph to capture the interaction information and further refine the 3D estimation results. Experimental results show that both our physical affinity loss and our context-aware graph network can effectively capture the relationship and improve the accuracy of 3D pose estimation. CCS CONCEPTS• Computing methodologies → Interest point and salient region detections; Reconstruction; Activity recognition and understanding.

3D-Augmented Contrastive Knowledge Distillation for Image-based Object Pose Estimation

“…Similarly, [7] proposes a lighter version of [61] by encoding the 3D shapes into graphs using node embeddings [15]. However, these multi-modal methods are limited as 3D shapes are oftentimes unavailable at testing [43,57,58]; The other category is the imagebased methods [13,37,59,61,63,66,68], that is only images are exploited for pose estimation. [13,37] regard corners of the 3D bounding box as generic keypoints, which only focus on cubic objects with simple geometric shape.…”

“…However, 3D shape is usually unavailable and acquiring 3D shapes is time consuming and labor intensive on-site. Therefore, the pure image-based object pose estimation without any 3D shape information has emerged [13,37,59,63,66,68]. In these works, one common approach [13,63,66,68] is to leverage keypoint features for pose estimation after detecting and re-projection of 2D keypoints, which requires for a suitable design of category-agnostic keypoints on various object geometries.…”

“…Therefore, the pure image-based object pose estimation without any 3D shape information has emerged [13,37,59,63,66,68]. In these works, one common approach [13,63,66,68] is to leverage keypoint features for pose estimation after detecting and re-projection of 2D keypoints, which requires for a suitable design of category-agnostic keypoints on various object geometries. Another approach [59] applies contrastive learning [2] to exploiting RGB-based geometric similarities shared between various categories, which heavily depends on the feature representation.…”