Recognition of Hand Gestures Observed by Depth Cameras

Abstract

We focus on gesture recognition based on 3D information in the form of a point cloud of the observed scene. A descriptor of the scene is built on the basis of a Viewpoint Feature Histogram (VFH). To increase the distinctiveness of the descriptor the scene is divided into smaller 3D cells and VFH is calculated for each of them. A verification of the method on publicly available Polish and American sign language datasets containing dynamic gestures as well as hand postures acquired by a time-of-flight (ToF) camera or Kinect is presented. Results of cross-validation test are given. Hand postures are recognized using a nearest neighbour classifier with city-block distance. For dynamic gestures two types of classifiers are applied: (i) the nearest neighbour technique with dynamic time warping and (ii) hidden Markov models. The results confirm the usefulness of our approach.

Comparing and combiningof point cloud descriptors features in static hand posture recognition

“…W [4] zaproponowano modyfikację sposobu liczenia VFH w celu rozpoznawania gestów dynamicznych, którą następnie wykorzystano do rozpoznawania statycznych postur dłoni [5]. Polegała ona na dzieleniu obserwowanej sceny na mniejsze prostopadłościenne komórki i wyznaczaniu cech deskryptora dla każdej z nich (niezależnie od pozostałych).…”

“…Aby zwiększyć dystynktywność deskryptorów, bryła brzegowa jest dzielona na prostopadłościenne komórki i cechy deskryptora liczone są dla każdej z nich niezależnie. Nasze eksperymenty przeprowadzone w [5] pokazały, że dzielenie bryły brzegowej na 3 horyzontalne komórki (patrz rys. 5) daje najlepsze rezultaty w rozpoznawaniu gestów wykonywanych dłonią, np.…”

Recognition of Hand Posture Based on a Point Cloud Descriptor and a Feature of Extended Fingers

“…Instead, VFH calculated for every descriptor value is represented by its mean and standard deviation. We use the two VFH values: |d| and cos(Φ) since our previous work [6] indicates that this combination is the best in terms of the hand posture recognition and adding new values or replacing some of them does not yield better results. The experiments undertaken in the mentioned work and other previous works also led us to the conclusion that VFH calculated for the entire point cloud (without the division into cells) was significantly less distinctive, and in this case the postures recognition most often resulted in misclassification, even for the seemingly dissimilar hand shapes.…”

“…The voxel dimensions V x × V y × V z are the parameters of the filter. We decided to use cubic voxels and set the dimensions as follows: V x × V y × V z =0.00439 m, which in our previous work [6] turned out to be the optimal value in terms of postures recognition rate. An example of a rotated and downsampled point cloud is shown in Fig 3h. …”

“…In [5] an approach to recognition of dynamic hand gestures based on VFH and time-of-flight data has been described. Our later work includes the recognition of dynamic hand gestures as well as hand postures using both time-of-flight and Kinect TM data [6]. In these works a modification of VFH calculation has been proposed, which consists in dividing the observed scene into smaller cuboidal cells and calculating VFH for each of them.…”