CFAM: Estimating 3D Hand Poses from a Single RGB Image with Attention

Abstract

Precise 3D hand pose estimation can be used to improve the performance of human–computer interaction (HCI). Specifically, computer-vision-based hand pose estimation can make this process more natural. Most traditional computer-vision-based hand pose estimation methods use depth images as the input, which requires complicated and expensive acquisition equipment. Estimation through a single RGB image is more convenient and less expensive. Previous methods based on RGB images utilize only 2D keypoint score maps to recover 3D hand poses but ignore the hand texture features and the underlying spatial information in the RGB image, which leads to a relatively low accuracy. To address this issue, we propose a channel fusion attention mechanism that combines 2D keypoint features and RGB image features at the channel level. In particular, the proposed method replans weights by using cascading RGB images and 2D keypoint features, which enables rational planning and the utilization of various features. Moreover, our method improves the fusion performance of different types of feature maps. Multiple contrast experiments on public datasets demonstrate that the accuracy of our proposed method is comparable to the state-of-the-art accuracy.

3D Hand Pose Estimation Based on Five-Layer Ensemble CNN

“…Figure 9 shows the comparisons with state-of-the-art methods [ 8 , 12 , 13 , 14 , 15 , 19 , 25 , 36 , 37 , 38 , 39 , 40 , 41 , 42 ] on the STB dataset; Ours represents the proposed 5LENet. It is worth noting that in addition to the pose estimation error, the process of hand positioning will also produce the error, so the methods involved in the comparison also need to add the corresponding localization error if they do not have it.…”

“…The high precision estimation result of ours again verifies the effectiveness and advancement of the five-layer network and five newly added 3D finger constraints. There is a substantial improvement compared with the 0.991 of Wang et al [ 13 ] and Yang et al [ 39 ], while Wang et al [ 13 ] use an RHD-trained localization segmentation network to crop the hand area, which cannot adapt well to the real environment compared with the OneHand10K-trained network. Yang et al [ 39 ] achieve image synthesis and pose estimation by learning disentangled representations of hand poses and hand images, but to a certain extent, the disentangling process will lead to the missing information that helps to generate useful data.…”

“…In addition, each network is trained separately, which is prone to falling into local optimization. Wang et al [ 13 ] estimated 2D keypoint features with RGB image features and introduced a channel domain attention mechanism as a weighting scheme to reweight the concatenated features, which can exploit features more efficiently to estimate 3D hand pose. In order to solve the problem of local optimization in literature [ 12 ], Liu et al [ 27 ] presented a three-stage cascaded CNN, which realized the mutual promotion between estimation stages, so as to achieve global optimization.…”

“…At present, mainstream 3D hand pose estimation methods can be classified into two categories: holistic estimation method based on the hand [ 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 ] and hierarchical estimation method based on hand structure [ 21 , 22 , 23 , 24 , 25 , 26 ]. The holistic estimation method based on the hand aims to directly use a complete hand structure for estimation, which has developed into a mainstream method in recent years [ 12 , 14 ].…”