Hand gesture recognition with jointly calibrated Leap Motion and depth sensor

Abstract

Novel 3D acquisition devices like depth cameras and the Leap Motion have recently reached the market. Depth cameras allow to obtain a complete 3D description of the framed scene while the Leap Motion sensor is a device explicitly targeted for hand gesture recognition and provides only a limited set of relevant points. This paper shows how to jointly exploit the two types of sensors for accurate gesture recognition. An ad-hoc solution for the joint calibration of the two devices is firstly presented. Then a set of novel feature descriptors is introduced both for the Leap Motion and for depth data. Various schemes based on the distances of the hand samples from the centroid, on the curvature of the hand contour and on the convex hull of the hand shape are employed and the use of Leap Motion data to aid feature extraction is also considered. The proposed feature sets are fed to two different classifiers, one based on multi-class SVMs and one exploiting Random Forests. Different feature selection algorithms have also been tested in order to reduce the complexity of the approach. Experimental results show that a very high accuracy can be obtained from the proposed method. The current implementation is also able to run in real-time

Structural Motion Grammar for Universal Use of Leap Motion: Amusement and Functional Contents Focused

“…The Leap Motion's movement recognition has also been investigated [16][17][18][19][20][21]. Marin et al [16,17] conducted research on the multiclass classifier by coupling Leap Motion with a Kinect and depth camera, while Vikram et al [18] studied the recognition of handwritten characters.…”

“…Marin et al [16,17] conducted research on the multiclass classifier by coupling Leap Motion with a Kinect and depth camera, while Vikram et al [18] studied the recognition of handwritten characters. Lu et al [19] proposed the Hidden Conditional Neural Field (HCNF) classifier to recognize the moving gestures.…”

“…Some studies on the use of SVM were reported in [9], and studies using HMM were investigated in [10][11][12]. However, these studies use machine learning, which requires feature extraction, normalization, and time-consuming training procedures.…”

Computer Access Using Leap Motion Sensor

“…Many gesture recognition methods have been put forward under difference environments. Marin et.al [Marin et al 2015] works on hand gestures recognition using Leap Motion Controller and kinect devices. Ad-hoc features are built based on fingertips positions and orientations.…”

Theoretical Frames for Sign and Dactyl Recognition System GESL data

“…not graphical /visual format) the data depends on the particular device and on its technical characteristics. It can be as a custom made device as well as any other existing devises, like Microsoft Kinect (Marin at al 2014, Yi Li 2012, Murata & Shin 2014Tang 2011), or LeapMotion (Marin at al 2014, Marin at al 2015, McCartney, at al 2015, Nowicki, et al 2014. These devices are cheap and do not require additional supplies or a special studio environment.…”