An Efficient Two-Stream Network for Isolated Sign Language Recognition Using Accumulative Video Motion

Abstract

Sign language is the primary communication medium for persons with hearing impairments. This language depends mainly on hand articulations accompanied by nonmanual gestures. Recently, there has been a growing interest in sign language recognition. In this paper, we propose a trainable deep learning network for isolated sign language recognition, which can effectively capture the spatiotemporal information using a small number of signs' frames. We propose a hierarchical sign learning module that comprises three networks: dynamic motion network (DMN), accumulative motion network (AMN), and sign recognition network (SRN). Additionally, we propose a technique to extract key postures for handling the variations in the sign samples performed by different signers. The DMN stream uses these key postures to learn the spatiotemporal information pertaining to the signs. We also propose a novel technique to represent the statical and dynamic information of sign gestures into a single frame. This approach preserves the spatial and temporal information of the sign by fusing the sign's key postures in the forward and backward directions to generate an accumulative video motion frame. This frame was used as an input to the AMN stream, and the extracted features were fused with the DMN features to be fed into the SRN for the learning and classification of signs. The proposed approach is efficient for isolated sign language recognition, especially for recognizing static signs. We evaluated this approach on the KArSL-190 and KArSL-502 Arabic sign language datasets, and the obtained results on KArSL-190 outperformed other techniques by 15% in the signer-independent mode. Additionally, the proposed approach outperformed the state-of-the-art techniques on the Argentinian sign language dataset LSA64. The code is available at https://github.com/Hamzah-Luqman/SLR_AMN.

A Review on Dynamic Hand Gesture Recognition Techniques

“…The feature learning phase receives these two volumes, out of which one of them is dedicated to the hand region, another represents the whole gesture area. [11] B. Feature Learning Hand configuration's precise spatial and temporal properties are learned by the maiden C3D instance.…”

“…The result of this step is two feature vectors, each with 4096 size. [11] C. Feature Fusion and Classification After dimension reduction, we may acquire a precise representation of integrated features, resulting in reduced computing difficulty and higher face identification accuracy. Feature fusion aids in the complete learning of picture characteristics for the description of their rich internal information.…”

“…Combining training picture features vector from the common weight network layer with extracted features made up of other numerical data allows the proposed model to use as many features as feasible for the subsequent classification. [11] Frame work…”

ST-TGR: Spatio-Temporal Representation Learning for Skeleton-Based Teaching Gesture Recognition

“…At the same time, gesture recognition can help create a richer and more interactive learning experience in online teaching environments [ 4 ]. For special education, such as students with hearing impairments, gesture recognition can also be used to identify and learn sign language [ 5 , 6 ]. Teaching gesture recognition brings many possibilities to education by improving teaching quality and enhancing student learning experience.…”

Enhanced Weak Spatial Modeling Through CNN-Based Deep Sign Language Skeletal Feature Transformation

“…Skeletal hand features can be obtained from sensor devices [4], such as depth cameras, or hand pose estimators, which use deep learning to detect and track the key points of the hand skeleton from images or videos. Some sign language models use skeleton-based methods alone, while others combine them with other modalities, such as RGB and depth, to provide complementary information [13]. Hamza [13] proposes an efficient hand key posture method for isolated sign language recognition to address variations in background and lighting.…”