Sign Language Recognition Using Wearable Electronics: Implementing k-Nearest Neighbors with Dynamic Time Warping and Convolutional Neural Network Algorithms

Abstract

We propose a sign language recognition system based on wearable electronics and two different classification algorithms. The wearable electronics were made of a sensory glove and inertial measurement units to gather fingers, wrist, and arm/forearm movements. The classifiers were k-Nearest Neighbors with Dynamic Time Warping (that is a non-parametric method) and Convolutional Neural Networks (that is a parametric method). Ten sign-words were considered from the Italian Sign Language: cose, grazie, maestra, together with words with international meaning such as google, internet, jogging, pizza, television, twitter, and ciao. The signs were repeated one-hundred times each by seven people, five male and two females, aged 29–54 y ± 10.34 (SD). The adopted classifiers performed with an accuracy of 96.6% ± 3.4 (SD) for the k-Nearest Neighbors plus the Dynamic Time Warping and of 98.0% ± 2.0 (SD) for the Convolutional Neural Networks. Our system was made of wearable electronics among the most complete ones, and the classifiers top performed in comparison with other relevant works reported in the literature.

Analysis of Influence of Segmentation, Features, and Classification in sEMG Processing: A Case Study of Recognition of Brazilian Sign Language Alphabet

“…Other recent studies presented frameworks and techniques that have been developed to solve the SLR problem, focusing on signal processing for sensors with different natures. For example, a leap motion sensor was used with a hidden Markov layer for pattern recognition of 24 gestures of ASL [41] and instrumented gloves with flex [42] and/or inertial sensors [43] with machine learning techniques reached accuracies ranging from 86% to 98%. A combination of images and deep learning techniques demonstrated high performance for this task, such as for Persian Sign Language [44] and CSL [45].…”

Recognition and Classification of Dynamic Hand Gestures by a Wearable Data-Glove

“…To the best of our knowledge, studies reported in [27,32] and [33] adopt approaches comparable with the technique here presented. In [27], the commercial sensory glove Cyberglove TM is exploited to acquire data about hand configuration, while the Flock of Birds ® motion tracker provides data about hand position and orientation.…”

“…They reached 91% recognition accuracy for a dataset of 95 categories of Australian sign language. In [33], a wearable system with 10 flex sensors in a glove and 3 IMUs has been used for data acquisition. In that paper, a k-Nearest Neighbors with Dynamic Time Warping and Convolutional Neural Networks has been applied to classify among 10 signed-words from the Italian Sign Language.…”

“…Talking Hands does not use any external device to acquire data about the absolute position of the hand, differently from the other works in literature that perform a translation on a set of dynamic signs of comparable size (cf. [27,32,33,37]).…”

Objective Assessment of Walking Impairments in Myotonic Dystrophy by Means of a Wearable Technology and a Novel Severity Index

“…On the other end, wearable technology has been proving to produce convincing and useful results in evaluating motor impairments of subjects suffering from (among others) Parkinson disease [10][11][12], attention deficit hyperactivity disorder/developmental coordination disorder (ADHD/DCD) [13], neuro-degenerative diseases [14], or spinal cord injury [15], when measured in a clinical environment, and in evaluating motor (dis)abilities of subjects when doing daily chores in home environment too [16][17][18]. The effective sensitivity of instruments as accelerometers, gyroscopes [19,20], and electromyography (EMG) [21], supports and extends the naked-eye analysis of the medical doctors [22] for the balance and gait analysis: Chapron et al adopted wearable IMUs on patients during physical rehabilitation programs within domestic environments [23]. Storm et al found that IMUs applied to the six-minute walk trial can provide clinically meaningful information [24].…”