Spatial Attention-Based 3D Graph Convolutional Neural Network for Sign Language Recognition

Al-Hammadi, Muneer; Bencherif, Mohamed A.; Alsulaiman, Mansour; Muhammad, Ghulam; Mekhtiche, Mohamed Amine; Abdul, Wadood; Al-Ohali, Yousef; Al-Rayes, Tareq S.; Mathkour, Hassan; Faisal, Mohammed; Algabri, Mohammed; Altaheri, Hamdi; Alfakih, Taha; Ghaleb, Hamid

doi:10.3390/s22124558

Cited by 17 publications

(16 citation statements)

References 43 publications

(51 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Comparatively, the proposed Two-Stream GCN model significantly outperformed existing models, achieving 34.41% accuracy. Specifically, ST-GCN [22] generated 16.48%, and the 3DGCN [27] model, which only utilized spatial context information, achieved 25.05%, demonstrating the superiority of our proposed approach. These results emphasize the effectiveness of the Two-Stream GCN model in addressing the challenges posed by largescale sign language datasets.…”

Section: G Performance Accuracy With Asllvd Datasetmentioning

confidence: 79%

“…The proposed Two-Stream GCN model exhibits a top accuracy of 34.41%, setting a benchmark for largescale sign language datasets. To contextualize this result, a comparative analysis was performed with existing models, specifically ST-GCN [22] and 3DGCN [27]. Hammadi et al introduced a recent method for ASLLVD data, employing a graph-based convolutional neural network with separable 3DGCN layers and a spatial attention mechanism [27].…”

Section: G Performance Accuracy With Asllvd Datasetmentioning

confidence: 99%

“…To contextualize this result, a comparative analysis was performed with existing models, specifically ST-GCN [22] and 3DGCN [27]. Hammadi et al introduced a recent method for ASLLVD data, employing a graph-based convolutional neural network with separable 3DGCN layers and a spatial attention mechanism [27]. Their approach achieved 25.05% accuracy for all classes, with notable performance for partial classes but with limited effectiveness for the entire dataset.…”

Section: G Performance Accuracy With Asllvd Datasetmentioning

confidence: 99%

“…Numerous researchers have focused on developing systems to enhance the performance accuracy of the ASLLVD dataset. Notable among them are STGCN [22] and 3DGCN [27], considered the latest systems in this field. Amorim et al employed an STGCN for sign language recognition [22] based on the skeleton key points.…”

Section: G Performance Accuracy With Asllvd Datasetmentioning

confidence: 99%

“…In addition, Matyas [26] employed a spotter model, and they only show a few class label configurations of the WLASL dataset. Another large-scale dataset, ASSLLVD [22], is used by many researchers [22], but their system is still facing problems in achieving high-performance accuracy [22], [27]. The main drawback of the systems is lower performance accuracy and may fail to achieve the system's generalised properties due to a lack of effective features.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Sign Language Recognition Using Graph and General Deep Neural Network Based on Large Scale Dataset

Miah,

Hasan,

Nishimura

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Sign Language Recognition (SLR) represents a revolutionary technology aiming to establish communication between deaf and non-deaf communities, surpassing traditional interpreter-based approaches. Existing efforts in automatic sign recognition predominantly rely on hand skeleton joint information, steering clear of image pixels to address challenges like partial occlusion and redundant backgrounds. Many researchers have been working to develop automatic sign recognition using hand skeleton joint information instead of image pixels to overcome partial occlusion and redundant background problems. However, body motion and facial expression play an essential role in increasing the inner gesture variance in expressing sign language emotion besides hand information for large-scale sign word datasets. Recently, some researchers have been working to develop muti-gesture-based SLR recognition systems, but their performance accuracy and efficiency are unsatisfactory for real-time deployment. Addressing these limitations, we propose a novel approach -a two-stream multistage graph convolution with attention and residual connection (GCAR) -designed to extract spatial-temporal contextual information. The multistage GCAR system, incorporating a channel attention module, dynamically enhances attention levels, particularly for non-connected skeleton points during specific events within spatial-temporal features. The methodology involves capturing joint skeleton points and motion, offering a comprehensive understanding of a person's entire body movement during sign language gestures and feeding this information into two streams. In the first stream, joint key features undergo processing through sep-TCN, graph convolution, deep learning layer, and a channel attention module across multiple stages, generating intricate spatial-temporal features in sign language gestures. Simultaneously, the joint motion is processed in the second stream, mirroring the steps of the first branch. The fusion of these two features yields the final feature vector, which is then fed into the classification module. The model excels in capturing discriminative structural displacements and short-range dependencies by leveraging unified joint features projected onto a high-dimensional space. Owing to the effectiveness of these features, the proposed method achieved significant accuracies: 90.31%, 94.10%, 99.75%, and 34.41%, for the WLASL, PSL, MSL, and ASLLVD large-scale datasets, respectively, with 0.69 million parameters. The high-performance accuracy, coupled with stable computational complexity, demonstrates the superiority of the proposed model. This innovative approach is anticipated to redefine the landscape of sign language recognition, setting a new standard in the field.

show abstract

Section: G Performance Accuracy With Asllvd Datasetmentioning

confidence: 79%

Section: G Performance Accuracy With Asllvd Datasetmentioning

confidence: 99%

Section: G Performance Accuracy With Asllvd Datasetmentioning

confidence: 99%

Section: G Performance Accuracy With Asllvd Datasetmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Sign Language Recognition Using Graph and General Deep Neural Network Based on Large Scale Dataset

Miah,

Hasan,

Nishimura

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

Double-handed dynamic gesture recognition using contour-based hand tracking and maximum mean probability ensembling (MMPE) for Indian Sign Language

Sruthi

Lijiya²

2022

Vis Comput

View full text Add to dashboard Cite

Spatial–temporal attention with graph and general neural network-based sign language recognition

Miah,

Hasan,

Okuyama

et al. 2024

Pattern Anal Applic

View full text Add to dashboard Cite

Automatic Sign Language Recognition (SLR) stands as a vital aspect within the realms of Human-Computer Interaction (HCI) and computer vision, facilitating the conversion of hand signs-utilized by individuals with significant hearing and speech impairments-into equivalent text or voice. Researchers have recently used hand skeleton joint information instead of the image pixel due to light illumination and complex background-bound problems. However, besides the hand information, body motion and facial gestures play an essential role in expressing sign language emotion. Also, a few researchers have been working to develop an SLR system by taking a multi-gesture dataset, but their performance accuracy and time complexity are not sufficient. In light of these limitations, we introduce a Spatial and Temporal Attention Model amalgamated with a General Neural Network designed for the SLR system. The main idea of our architecture is first to construct a fully connected graph to project the skeleton information. We employ self-attention mechanisms to extract insights from node Article Title and edge features across spatial and temporal domains. Our architecture bifurcates into three branches: a graph-based spatial branch, a graph-based temporal branch, and a general neural network branch, which collectively synergize to contribute to the final feature integration. Specifically, the spatial branch discerns spatial dependencies, while the temporal branch amplifies temporal dependencies embedded within the sequential hand skeleton data. Further, the general neural network branch enhances the architecture's generalization capabilities, bolstering its robustness. In our evaluation, utilizing the Mexican Sign Language(MSL), Pakistani Sign Langauge(PSL) datasets and American Sign Langauge Large Video Dataset(ASSLVD) -which comprises 3D joint coordinates for face, body, and hands that conducted experiments on individual gestures and their combinations. Impressively, our model demonstrated notable efficacy, achieving an accuracy rate of 99.96% for the MSL dataset, 92.00% for PSL, and 26.00% for the ASLLVD dataset, which includes 2745 classes. These exemplary performance metrics, coupled with the model's computationally efficient profile, underscore its preeminence compared to contemporaneous methodologies in the field.

show abstract

Spatial Attention-Based 3D Graph Convolutional Neural Network for Sign Language Recognition

Cited by 17 publications

References 43 publications

Sign Language Recognition Using Graph and General Deep Neural Network Based on Large Scale Dataset

Sign Language Recognition Using Graph and General Deep Neural Network Based on Large Scale Dataset

Double-handed dynamic gesture recognition using contour-based hand tracking and maximum mean probability ensembling (MMPE) for Indian Sign Language

Spatial–temporal attention with graph and general neural network-based sign language recognition

Contact Info

Product

Resources

About