Trajectory‐based view‐invariant hand gesture recognition by fusing shape and orientation

Abstract

Traditional studies in vision‐based hand gesture recognition remain rooted in view‐dependent representations, and hence users are forced to be fronto‐parallel to the camera. To solve this problem, view‐invariant gesture recognition aims to make the recognition result independent of viewpoint changes. However, in current works the view‐invariance is achieved at the price of mixing different gesture patterns that have similar trajectory curve shape but different semantic meanings. For example, the gesture ‘push’ can be mistaken as ‘drag’ from another viewpoint. To address this shortcoming, in this study, the authors use a shape descriptor to extract the view‐invariant features of a three‐dimensional (3D) trajectory. As the shape features are invariant to omnidirectional viewpoint changes, the orientation features are then added into weight different rotation angles so that similar trajectory shapes are better separated. The proposed method was conducted on two different databases, including a popular Australian Sign Language database and a challenging Kinect Hand Trajectory database. Experimental results show that the proposed algorithm achieves a higher average recognition rate than the state‐of‐the‐art approaches, and can better distinguish confusing gestures while meeting the view‐invariant condition.

Taiwan Sign Language Recognition System Using LC-KSVD Sparse Coding Method

“…In [9], A.-M. Cretu et al use depth information and the threshold of distance to capture hand shapes; nevertheless, it is hard to catch hand shapes when the hand is near body. In [10], X. Wu et al use depth joints features to draw hand location and trajectory; although, trajectory features do not used with hand shapes for gestures recognition. The above-mentioned problems are considered as the opinions of features choice.…”

Design and evaluation of novel gesture-centric interaction models in smart spaces

“…Most of the works about gesture recognition approaches in the review literature are focused on gesture data from optical [107,31,96] or inertial sensor [52,18,37], or the fusion of these two [21,65,20]. This work, however, explores a unified solution to the gesture recognition problem based on both types of sensors, widening the base of compatible gesture input devices in terms of sensing devices.…”

“…Some works in this category use traditional color cameras like [87,31] while others recognize gestures based on RGB-D data, such as [96,10,105,110,9]. The gesture trajectory can be obtained directly through Kinect-like devices and software, and the works [101,14,107,31] start the recognition from the obtained gesture trajectory. Although the vision-based tracking enables the users to do free gestures without cumbersome contact-based devices like data glove, it does have some limitations, such as being prone to be interfered by varying lighting conditions and cluttered background and relatively low sampling rate of normal cameras [21].…”

“…To make the Kinect-based recognition position-invariant, the coordinates of gesture trajectory are transformed from camerabased to body-based and the displacement of the coordinates between test sample and template is compensated in [14]. Based on the Kinect as [14], the work of [107] aims at making recognition view-invariant by extracting shape features that are invariant to rotation, scale and translation transformation of the gesture trajectory. In addition, the orientation features indicating the rotation state of a trajectory are complement of the shape features and help distinguish gestures with similar shapes.…”

“…It will have problems when users train the gestures with the body facing the device but do gestures with the body facing perpendicular to the device orientation. The shape features, which are proposed in [107] and describe the gesture trajectory in a way invariant to rotation, scale and translation, could be able to enhance the recognition approach to be view-invariant.…”