Comparing gesture recognition accuracy using color and depth information

Abstract

In human-computer interaction applications, gesture recognition has the potential to provide a natural way of communication between humans and machines. The technology is becoming mature enough to be widely available to the public and real-world computer vision applications start to emerge. A typical example of this trend is the gaming industry and the launch of Microsoft's new camera: the Kinect. Other domains, where gesture recognition is needed, include but are not limited to: sign language recognition, virtual reality environments and smart homes. A key challenge for such real-world applications is that they need to operate in complex scenes with cluttered backgrounds, various moving objects and possibly challenging illumination conditions. In this paper we propose a method that accommodates such challenging conditions by detecting the hands using scene depth information from the Kinect. On top of our detector we employ a dynamic programming method for recognizing gestures, namely Dynamic Time Warping (DTW). Our method is translation and scale invariant which is a desirable property for many HCI systems. We have tested the performance of our approach on a digits recognition system. All experimental datasets include hand signed digits gestures but our framework can be generalized to recognize a wider range of gestures.

Toward a 3D body part detection video dataset and hand tracking benchmark

“…Figures 5 and 6 show the results for the skeletal tracker and its comparison to the single hand detector, respectively. We can see in figure 6, for example, that 50% of the signs had a maximum pixel error of about 22 pixels or less when the comparison method of [4] was used to detect the hands. shows an example frame with good accuracy using the skeletal tracker on both hands in a two-handed sign.…”

“…This operation was performed on each frame of the signs, and the accuracy was calculated to serve as the benchmark for the evaluation of future methods. As an example comparison, we processed the one-handed signs with the single hand locator described in [4]-a method based on movement and depth alone-and calculated the results using the same pixel Euclidean distance similarity measure.…”

“…One of the earlier methods using the Kinect, proposed by Doliotis, et al, uses a combination of depth video motion analysis and scene distance information [4]. The algorithm isolates the person performing the gesture using segmentation on the depth data, and a score is calculated for each pixel belonging to the person based on distance to the camera and motion.…”

Real-Time Dynamic Sign Language Recognition Based on Hierarchical Matching Strategy

“…For example, in the [9] literature, palm position in each frame is extracted and thus a trajectory curve is formed. The frame rate of Kinect is 30 frames per second, and if the dynamic sign language "graduation certificate" lasts for 2.13 second, then totally 64 frames and 64 corresponding palm points can be obtained.…”

Image based recognition of Pakistan sign language

“…Several methods have been suggested to detect the signs easily and accurately. Among the noteworthy methodologies used for sign recognition are Template Matching (Liu & Fujimura, 2004), Conditional Random Fields (CRF) (Saad et al, 2012) and Dynamic Time Warping (DTW) (Doliotis et al, 2011). Dynamic signs are the signs which rely on hands, head and body motion.…”