One-Shot Learning Gesture Recognition from RGB-D Data Using Bag of Features

Abstract

For one-shot learning gesture recognition, two important challenges are: how to extract distinctive features and how to learn a discriminative model from only one training sample per gesture class. For feature extraction, a new spatio-temporal feature representation called 3D enhanced motion scale-invariant feature transform (3D EMoSIFT) is proposed, which fuses RGB-D data. Compared with other features, the new feature set is invariant to scale and rotation, and has more compact and richer visual representations. For learning a discriminative model, all features extracted from training samples are clustered with the k-means algorithm to learn a visual codebook. Then, unlike the traditional bag of feature (BoF) models using vector quantization (VQ) to map each feature into a certain visual codeword, a sparse coding method named simulation orthogonal matching pursuit (SOMP) is applied and thus each feature can be represented by some linear combination of a small number of codewords. Compared with VQ, SOMP leads to a much lower reconstruction error and achieves better performance. The proposed approach has been evaluated on ChaLearn gesture database and the result has been ranked amongst the top best performing techniques on ChaLearn gesture challenge (round 2).

Local frame match distance: A novel approach for exemplar gesture recognition

“…Long Short Term Memory (LSTM) networks have proven successful for this task [22]. With the release of ChaLearn Gesture Challenge data set [23], there have been a number of works in oneshot learning, in which a single training example is used per gesture class [24], [25], [26]. A third focus is on developing methods that work on well-established gesture sets, such as sign languages.…”

Design and evaluation of novel gesture-centric interaction models in smart spaces

“…Most of the works about gesture recognition approaches in the review literature are focused on gesture data from optical [107,31,96] or inertial sensor [52,18,37], or the fusion of these two [21,65,20]. This work, however, explores a unified solution to the gesture recognition problem based on both types of sensors, widening the base of compatible gesture input devices in terms of sensing devices.…”

“…Most of the reviewed recognition approaches [21,54,110] are tested with more than one training samples. Some works [96,105,52] are dedicated to approaches with only one training sample. Others ones [20,3] report impressive accuracy with user-independent approaches, which do not require training from new users.…”

“…The detection and tracking are basically computer vision problems. Some works in this category use traditional color cameras like [87,31] while others recognize gestures based on RGB-D data, such as [96,10,105,110,9]. The gesture trajectory can be obtained directly through Kinect-like devices and software, and the works [101,14,107,31] start the recognition from the obtained gesture trajectory.…”

“…Based on the bag of feature model, the 3D enhanced motion scale-invariant feature transformation fuses the RGB-D data, and the features are clustered with the k-mean algorithm to learn a visual codebook to which the features will be mapped to have the recognition result [96]. In [87], the descriptors that characterize the local motion patterns are indexed and the gestures are recognized by matching against the database through a term frequency-inverse document frequency (TF-IDF) weighting scheme.…”