Air Writing: Recognizing Multi-Digit Numeral String Traced in Air Using RNN-LSTM Architecture

Abstract

Air writing provides a more natural and immersive way of interacting with devices, with the potential of having significant application in fields like augmented reality and education. However, such systems often rely on expensive hardware, making them less accessible for general purposes. In this study, we propose a robust and inexpensive system for the recognition of multi-digit numerals traced in an air-writing environment which uses only a generic device camera for input. We employ a sliding window-based algorithm to isolate a small segment of the input for processing. A dual network configuration consisting of RNN-LSTM networks are used for noise elimination and digit recognition. We conduct our experiments on English numerals using the MNIST dataset as the baseline model to allow easy adaptability of our method. Our results are further improved by the use of Pendigits and ISI-Air online datasets. We observed a drop in accuracy with increase in the number of digits owing to the accumulation of transition noise. However, bi-directional scanning considerably reduces the impact of such noise on the recognition accuracy. Under standard conditions, our system produced an accuracy of 98.75% and 85.27% for single and multi-digit English numerals, respectively. Incorporation of selective frame skipping in the sliding window algorithm resulted in a 60% reduction in computational time, significantly improving the system performance. We provide a link to the source code of our system at the end of this paper.

An End-to-End Air Writing Recognition Method Based on Transformer

“…Intermediate steps Recognition steps Fingertip tracking recognition [23] Use fingertip-tracking algorithms to obtain trajectory graphs Using a handwritten digit dataset (MNIST) as a training set to identify the content of the trajectory graph Depth-Camera [25] Collects fingertip coordinates (XYZ) via depth sensors air-writing recognition based on coordinate data using CNN networks Hand Skeleton Recognition [26] Obtaining hand key point coordinates and joint angles using a deep learning framework Air writing recognition using the RNN-LSTM model based on hand key points and joint angles…”

“…They then detect fingertips based on finger contour features and proposed an improved deep neural network to recognize handwritten characters (2010) [24]. In addition to trajectory recognition, M. S. Alam et al used a depth camera to recognize air-writing by tracking the three-dimensional data changes of fingertip positions in space (2020) [25], A. Rahman et al proposed a bone recognition-based method that uses hand key points and joint angles as input data and an RNN-LSTM structure to recognize handwritten characters (2021) [26].…”

Air-Writing Recognition Based on Deep Convolutional Neural Networks

“…The proposed method achieved 97.7%, 95.4% and 93.7% recognition rates in person-independent evaluations over English, Bengali and Devanagari numerals, respectively. To incorporate flexibility in marker choice and stable motion tracking under varying lighting conditions, Rahman et al [18] improved the marker tip tracking scheme by a marker calibration mechanism. They presented a dual network configuration consisting of RNN-LSTM (recurrent neural network-long short-term memory) networks for noise elimination and digit recognition.…”