Sensor Fusion of Motion-Based Sign Language Interpretation with Deep Learning

Abstract

Sign language was designed to allow hearing-impaired people to interact with others. Nonetheless, knowledge of sign language is uncommon in society, which leads to a communication barrier with the hearing-impaired community. Many studies of sign language recognition utilizing computer vision (CV) have been conducted worldwide to reduce such barriers. However, this approach is restricted by the visual angle and highly affected by environmental factors. In addition, CV usually involves the use of machine learning, which requires collaboration of a team of experts and utilization of high-cost hardware utilities; this increases the application cost in real-world situations. Thus, this study aims to design and implement a smart wearable American Sign Language (ASL) interpretation system using deep learning, which applies sensor fusion that “fuses” six inertial measurement units (IMUs). The IMUs are attached to all fingertips and the back of the hand to recognize sign language gestures; thus, the proposed method is not restricted by the field of view. The study reveals that this model achieves an average recognition rate of 99.81% for dynamic ASL gestures. Moreover, the proposed ASL recognition system can be further integrated with ICT and IoT technology to provide a feasible solution to assist hearing-impaired people in communicating with others and improve their quality of life.

Backhand-Approach-Based American Sign Language Words Recognition Using Spatial-Temporal Body Parts and Hand Relationship Patterns

“…They can be divided into two groups. First, the sensor-based group [15][16][17][18][19][20] includes devices worn on the users' hands, such as gloves, EMG sensors, cables, accelerometers, touch sensors, and flexion sensors. The degree of flexion of the fingers and the fingers' motion are used as the extracted features.…”

“…In the final stage of feature extraction, the four features consist of the spatial-temporal body parts and hand relationship patterns (H 1 (t)), the spatial-temporal finger joint angle patterns (H 2 (t)), the spatial-temporal double-hand relationship patterns (H 3 (t)), and the spatial-temporal 3D hand motion trajectory patterns (H 4 (t)). These features are concatenated into one-dimensional data by means of a concatenation technique, of which the equations are shown in Equation (15). The final feature extraction in terms of spatial temporal patterns (X (t)) is the input of a stacked BiLSTM in the classification process.…”

“…Equation ( 18) expresses the cell states generated from the updated cell state in terms of the forget gate ( 𝑓 ), the input The structure of the designed two-layer BiLSTM model consists of the input, BiLSTM hidden neurons, dropout, and classification layers, as shown in Figure 12. The input layer contains time series data, as demonstrated in Equation (15). Then, the BiLSTM hidden layer consists of four main gates: the forget gate, the input gate, the input modulation gate, and the output gate.…”

A Systematic Review on Systems-Based Sensory Gloves for Sign Language Pattern Recognition: An Update From 2017 to 2022

“…Thirdly, aside from the aforementioned, it is recommended that two technical aspects be employed such as using a pair of gloves for data collection, as stated in [85], [85], [86], [87]. Using a combination of two gloves instead of a wide range of hand gestures can be added [88], [26].…”

Interpreting Arabic sign alphabet by utilizing a glove with sensors

“…Figure 7 shows the LSTM internal architecture. f t is the forget gate of a previous cell state C t−1 ,  is the logistic sigmoid non-linear activation functions [Lee et al, 2020] and  is applied on C t−1 to determine which the retrieved LSTM memory should be kept. tanh is then applied to the value between 1 and -1 (derived from the second sigmond function) with the following parameters: C t a new cell state of the LSTM memory, and x t a new input to the LSTM memory.…”