Deep Belief Networks for Recognizing Handwriting Captured by Leap Motion Controller

Abstract

Leap Motion controller is an input device that can track hands and fingers position quickly and precisely. In some gaming environment, a need may arise to capture letters written in the air by Leap Motion, which cannot be directly done right now. In this paper, we propose an approach to capture and recognize which letter has been drawn by the user with Leap Motion. This approach is based on Deep Belief Networks (DBN) with Resilient Backpropagation (Rprop) fine-tuning. To assess the performance of our proposed approach, we conduct experiments involving 30,000 samples of handwritten capital letters, 8,000 of which are to be recognized. Our experiments indicate that DBN with Rprop achieves an accuracy of 99.71%, which is better than DBN with Backpropagation or Multi-Layer Perceptron (MLP), either with Backpropagation or with Rprop. Our experiments also show that Rprop makes the process of fine-tuning significantly faster and results in a much more accurate recognition compared to ordinary Backpropagation. The time needed to recognize a letter is in the order of 5,000 microseconds, which is excellent even for online gaming experience.

TARNet: An Efficient and Lightweight Trajectory-Based Air-Writing Recognition Model Using a CNN and LSTM Network

“…These wearable devices were difficult (and sometimes impossible) to handle. The recent Kinect [27,29,41], RealSense [22,23], and Leap Motion [12,21,25,28,42,43] devices are vision-based and very user-friendly and do not require additional devices. The RealSense camera is straightforward to ease.…”

“…We used four air-writing datasets: the RTD [22], RTC [23], smart-band [24], and Abas datasets [25]. The RTD and RTC vision-based datasets were collected using an Intel RealSense sr300 camera.…”

“…In this research, we propose a versatile deep learning model that fuses both CNN and LSTM features and achieves outstanding results. The model was trained and tested using the publicly available RealSense trajectory digit (RTD) [22], RealSense trajectory character (RTC) [23], smart-band [24], and Abas datasets [25]. The variety of datasets proves that the proposed model is invariant to a different dataset.…”

Trajectory-Based Air-Writing Recognition Using Deep Neural Network and Depth Sensor

“…Most of the research has been done by using the trajectory information directly, i.e., using the temporal information. On the other hand, Setiawan and Pulungan [30] proposed a 2D mapping approach, in which trajectories were collected by the Leap Motion device and converted to the 2D image matrix like the popular MNIST dataset. Nowadays, WiFi and Radar-based technology have become popular.…”

A Haptic feedback system based on leap motion controller for prosthetic hand application

“…It consists of three parts, two cameras, LEDs, and a microcontroller in which the cameras capture successive images of the hand and then they are passed to the controller to process the images and extract spatial information of the hand and fingers. Leap Motion has been used in several projects to recognize hand gestures [8][9][10][11][12][13]. Although approaching the dexterity of a human hand to control robotic arm is difficult, LM was used to do this task in [14][15][16], where hand gestures have been translated to joint angles to perform a specific task.…”