Real-time Embedded Recognition of Sign Language Alphabet Fingerspelling in an IMU-Based Glove

Mummadi, Chaithanya Kumar; Leo, Frederic Philips Peter; Verma, Keshav Deep; Kasireddy, Shivaji; Scholl, Philipp M.; Laerhoven, Kristof Van

doi:10.1145/3134230.3134236

Cited by 20 publications

(7 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the mobile and ubiquitous computing community, there have been efforts to exploit sensing platforms for sign language translation [14,34,37,39,44,57,60]. These previous works use devices such as RGB cameras [5,21,27,29,33,35,46,54,55], motion sensors (e.g., Leap Motion) [14,41], depth cameras/sensors (e.g., Kinect) [6,10,11,16,38,48,51], or electromyogram (EMG) sensors [53,57] to capture user hand motions and combine sensing results with various machine learning models to infer the word being expressed.…”

Section: :2 • Park Et Almentioning

confidence: 99%

“…Mohandes [37] designed a two-handed sign recognition system for Unified Arabic Sign Language using CyberGloves, and a support vector machine (SVM) was used as a classifier. Mummadi et al [39] designed a real-time recognition system for the French Sign Language alphabet using a custom-developed IMU sensor-based glove. In this work, the authors exploit a multi-layer perceptron (MLP) to classify between different alphabet letters.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Enabling Real-time Sign Language Translation on Mobile Platforms with On-board Depth Cameras

Park

Lee

2021

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

In this work we present SUGO, a depth video-based system for translating sign language to text using a smartphone's front camera. While exploiting depth-only videos offer benefits such as being less privacy-invasive compared to using RGB videos, it introduces new challenges which include dealing with low video resolutions and the sensors' sensitiveness towards user motion. We overcome these challenges by diversifying our sign language video dataset to be robust to various usage scenarios via data augmentation and design a set of schemes to emphasize human gestures from the input images for effective sign detection. The inference engine of SUGO is based on a 3-dimensional convolutional neural network (3DCNN) to classify a sequence of video frames as a pre-trained word. Furthermore, the overall operations are designed to be light-weight so that sign language translation takes place in real-time using only the resources available on a smartphone, with no help from cloud servers nor external sensing components. Specifically, to train and test SUGO, we collect sign language data from 20 individuals for 50 Korean Sign Language words, summing up to a dataset of ~5,000 sign gestures and collect additional in-the-wild data to evaluate the performance of SUGO in real-world usage scenarios with different lighting conditions and daily activities. Comprehensively, our extensive evaluations show that SUGO can properly classify sign words with an accuracy of up to 91% and also suggest that the system is suitable (in terms of resource usage, latency, and environmental robustness) to enable a fully mobile solution for sign language translation.

show abstract

Section: :2 • Park Et Almentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Enabling Real-time Sign Language Translation on Mobile Platforms with On-board Depth Cameras

Park

Lee

2021

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

show abstract

“…Hand Gesture Recognition was previously explored in [1,4,5,6]. In [18] a similar glove prototype was built to recognize french sign language characters by estimating the rotation of each finger. A single-sensor glove prototype called GestGlove [19] was built to recognize a simple set of hand gestures allowing phone control.…”

Section: Related Workmentioning

confidence: 99%

Waveglove: Transformer-Based Hand Gesture Recognition Using Multiple Inertial Sensors

Králik¹,

Šuppa²

2021

2021 29th European Signal Processing Conference (EUSIPCO)

View full text Add to dashboard Cite

Hand Gesture Recognition (HGR) based on inertial data has grown considerably in recent years, with the state-of-theart approaches utilizing a single handheld sensor and a vocabulary comprised of simple gestures. In this work we explore the benefits of using multiple inertial sensors. Using WAVE-GLOVE, a custom hardware prototype in the form of a glove with five inertial sensors, we acquire two datasets consisting of over 11000 samples. To make them comparable with prior work, they are normalized along with 9 other publicly available datasets, and subsequently used to evaluate a range of Machine Learning approaches for gesture recognition, including a newly proposed Transformer-based architecture. Our results show that even complex gestures involving different fingers can be recognized with high accuracy. An ablation study performed on the acquired datasets demonstrates the importance of multiple sensors, with an increase in performance when using up to three sensors and no significant improvements beyond that.

show abstract

“…The algorithms for HAR can be classified into shallow and deep learning methods. Common shallow methods in HAR include SVM [13], [20], [23], k-nearest neighbors (kNN) [16], [24], linear discriminant analysis (LDA) [9], and random forest (RF) [21]. Deep learning approaches, such as LSTM [7], [15], CNN-LSTM [25], [27], CNN [22], and convLSTM [26], have shown impressive leaps in performance compared to their shallow counterparts by learning to automatically extract features from raw sensor data, thus dropping the need for having human experts to provide hand-engineered features.…”

Section: Background a Human Activity Recognition (Har)mentioning

confidence: 99%

Improving Cross-Subject Activity Recognition via Adversarial Learning

Leite

Xiao

2020

IEEE Access

View full text Add to dashboard Cite

Deep learning has been widely used for implementing human activity recognition from wearable sensors like inertial measurement units. The performance of deep activity recognition is heavily affected by the amount and variability of the labeled data available for training the deep learning models. On the other hand, it is costly and time-consuming to collect and label data. Given limited training data, it is hard to maintain high performance across a wide range of subjects, due to the differences in the underlying data distribution of the training and the testing sets. In this work, we develop a novel solution that applies adversarial learning to improve cross-subject performance by generating training data that mimic artificial subjects-i.e. through data augmentation-and enforcing the activity classifier to ignore subject-dependent information. Contrary to domain adaptation methods, our solution does not utilize any data from subjects of the test set (or target domain). Furthermore, our solution is versatile as it can be utilized together with any deep neural network as the classifier. Considering the open dataset PAMAP2, nearly 10% higher crosssubject performance in terms of F1-score can be achieved when training a CNN-LSTM-based classifier with our solution. A performance gain of 5% is also observed when our solution is applied to a stateof-the-art HAR classifier composed of a combination of inception neural network and recurrent neural network. We also investigate different influencing factors of classification performance (i.e. selection of sensor modalities, sampling rates and the number of subjects in the training data), and summarize a practical guideline for implementing deep learning solutions for sensor-based human activity recognition.

show abstract

Real-time Embedded Recognition of Sign Language Alphabet Fingerspelling in an IMU-Based Glove

Cited by 20 publications

References 16 publications

Enabling Real-time Sign Language Translation on Mobile Platforms with On-board Depth Cameras

Enabling Real-time Sign Language Translation on Mobile Platforms with On-board Depth Cameras

Waveglove: Transformer-Based Hand Gesture Recognition Using Multiple Inertial Sensors

Improving Cross-Subject Activity Recognition via Adversarial Learning

Contact Info

Product

Resources

About