Independent Sign Language Recognition with 3d Body, Hands, and Face Reconstruction

Kratimenos, Agelos; Pavlakos, Georgios; Maragos, Petros

doi:10.1109/icassp39728.2021.9414278

Cited by 12 publications

(4 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…hand gesture, body pose, and facial expression. The authors in [24] utilized SMPL-X, a modern parametric model that allows the extraction of 3D body shape, face, and hand information from a single image. By using this comprehensive 3D reconstruction, the authors conducted SLR and found that it resulted in greater accuracy compared to recognizing information from raw RGB images or 2D skeletons.…”

Section: A Depth Pose Alignment Of Imagementioning

confidence: 99%

Deep Learning Based Multi Pose Human Face Matching System

Sohail,

Shoukat,

Khan

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Current techniques for multi-pose human face matching yield suboptimal outcomes because of the intricate nature of pose equalization and face rotation. Deep learning models, such as YOLO-V5, etc., that have been proposed to tackle these complexities, suffer from slow frame matching speeds and therefore exhibit low face recognition accuracy. Concerning this, certain literature investigated multi-pose human face detection systems; however, those studies are of elementary level and do not adequately analyze the utility of those systems. To fill this research gap, we propose a real-time face matching algorithm based on YOLO-V5. Our algorithm utilizes multi-pose human patterns and considers various face orientations, including organizational faces and left, right, top, and bottom alignments, to recognize multiple aspects of people. Using face poses, the algorithm identifies face positions in a dataset of images obtained from mixed pattern live streams, and compares faces with a specific piece of the face that has a relatively similar spectrum for matching with a given dataset. Once a match is found, the algorithm displays the face on Google Colab, collected during the learning phase with the Robo-flow key, and tracks it using the YOLO-V5 face monitor. Alignment variations are broken up into different positions, where each type of face is uniquely learned to have its own study demonstrated. This method offers several benefits for identifying and monitoring humans using their labeling tag as a pattern name, including high face-matching accuracy and minimum speed of owing face-to-pose variations. Furthermore, the algorithm addresses the face rotation issue by introducing a mixture of error functions for execution time, accuracy loss, frame-wise failure, and identity loss, attempting to guide the authenticity of the produced image frame. Experimental results confirm effectiveness of the algorithm in terms of improved accuracy and reduced delay in the face-matching paradigm.

show abstract

Section: A Depth Pose Alignment Of Imagementioning

confidence: 99%

Deep Learning Based Multi Pose Human Face Matching System

Sohail,

Shoukat,

Khan

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…It is also recognized for independent sign language, comprising a 3D body, hands, and facial expressions. For this purpose, SMPL-X extracts features of body shape, face, and writing using a single image [14]. The 3D reconstructed output of the 3D model is utilized for SLR resulting in higher accuracy than raw RGB images.…”

Section: Literature Reviewmentioning

confidence: 99%

A Novel Action Transformer Network for Hybrid Multimodal Sign Language Recognition

Javaid¹,

Rizvi²

2023

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

Sign language fills the communication gap for people with hearing and speaking ailments. It includes both visual modalities, manual gestures consisting of movements of hands, and non-manual gestures incorporating body movements including head, facial expressions, eyes, shoulder shrugging, etc. Previously both gestures have been detected; identifying separately may have better accuracy, but much communicational information is lost. A proper sign language mechanism is needed to detect manual and non-manual gestures to convey the appropriate detailed message to others. Our novel proposed system contributes as Sign Language Action Transformer Network (SLATN), localizing hand, body, and facial gestures in video sequences. Here we are expending a Transformer-style structural design as a "base network" to extract features from a spatiotemporal domain. The model impulsively learns to track individual persons and their action context in multiple frames. Furthermore, a "head network" emphasizes hand movement and facial expression simultaneously, which is often crucial to understanding sign language, using its attention mechanism for creating tight bounding boxes around classified gestures. The model's work is later compared with the traditional identification methods of activity recognition. It not only works faster but achieves better accuracy as well. The model achieves overall 82.66% testing accuracy with a very considerable performance of computation with 94.13 Giga-Floating Point Operations per Second (G-FLOPS). Another contribution is a newly created dataset of Pakistan Sign Language for Manual and Non-Manual (PkSLMNM) gestures.

show abstract

“…For example, Yang et al [70] generate sequences of body language predictions from estimated human poses and feed them to an RNN for emotion interpretation and psychiatric symptom prediction. Kratimenos et al [32] extract a holistic 3D body shape, including hands and face, from a single image and feed them also to an RNN for sign language recognition. Singh et al [61] use handcrafted features to analyze body language for estimating a person's emotions and state of mind.…”

Section: Namementioning

confidence: 99%

Bodily Behaviors in Social Interaction: Novel Annotations and State-of-the-Art Evaluation

Balazia,

Müller,

Tánczos

et al. 2022

Preprint

View full text Add to dashboard Cite

Body language is an eye-catching social signal and its automatic analysis can significantly advance artificial intelligence systems to understand and actively participate in social interactions. While computer vision has made impressive progress in low-level tasks like head and body pose estimation, the detection of more subtle behaviors such as gesturing, grooming, or fumbling is not well explored. In this paper we present BBSI, the first set of annotations of complex Bodily Behaviors embedded in continuous Social Interactions in a group setting. Based on previous work in psychology, we manually annotated 26 hours of spontaneous human behavior in the MPIIGroupInteraction dataset with 15 distinct body language classes. We present comprehensive descriptive statistics on the resulting dataset as well as results of annotation quality evaluations. For automatic detection of these behaviors, we adapt the Pyramid Dilated Attention Network (PDAN), a state-of-the-art approach for * Authors contributed equally.

show abstract

Independent Sign Language Recognition with 3d Body, Hands, and Face Reconstruction

Cited by 12 publications

References 29 publications

Deep Learning Based Multi Pose Human Face Matching System

Deep Learning Based Multi Pose Human Face Matching System

A Novel Action Transformer Network for Hybrid Multimodal Sign Language Recognition

Bodily Behaviors in Social Interaction: Novel Annotations and State-of-the-Art Evaluation

Contact Info

Product

Resources

About