Multimodal 3D American sign language recognition for static alphabet and numbers using hand joints and shape coding

Mahdikhanlou, Khadijeh; Ebrahimnezhad, Hossein

doi:10.1007/s11042-020-08982-8

Cited by 15 publications

(6 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Subsequently, a 3D CNN was applied to improve the accuracy of hand sign language recognition, pose evaluation, and motion recognition. In [12], a hand recognition system that employs a leap motion controller (LMC) and a webcam was proposed. Using the LMC, the 3D positions of the hand joints were extracted, and from the RGB images of the hand obtained through the webcam, outline-based features were extracted, and the two sets of data were combined to achieve recognition.…”

Section: A Study Using Skeleton Methodsmentioning

confidence: 99%

ASL Recognition by the Layered Learning Model Using Clustered Groups

Shin¹,

Jung²

2023

Computer Systems Science and Engineering

View full text Add to dashboard Cite

American Sign Language (ASL) images can be used as a communication tool by determining numbers and letters using the shape of the fingers. Particularly, ASL can have an key role in communication for hearing-impaired persons and conveying information to other persons, because sign language is their only channel of expression. Representative ASL recognition methods primarily adopt images, sensors, and pose-based recognition techniques, and employ various gestures together with hand-shapes. This study briefly reviews these attempts at ASL recognition and provides an improved ASL classification model that attempts to develop a deep learning method with meta-layers. In the proposed model, the collected ASL images were clustered based on similarities in shape, and clustered group classification was first performed, followed by reclassification within the group. The experiments were conducted with various groups using different learning layers to improve the accuracy of individual image recognition. After selecting the optimized group, we proposed a meta-layered learning model with the highest recognition rate using a deep learning method of image processing. The proposed model exhibited an improved performance compared with the general classification model.

show abstract

Section: A Study Using Skeleton Methodsmentioning

confidence: 99%

ASL Recognition by the Layered Learning Model Using Clustered Groups

Shin¹,

Jung²

2023

Computer Systems Science and Engineering

View full text Add to dashboard Cite

show abstract

“…Those features are then classified using random forest. Grzejszczak [46] proposed a method for the localization of landmarks in RGB images. They analyzed a skin-masked directional image using hand transform and template matching.…”

Section: Hgr Via Landmarks Featuresmentioning

confidence: 99%

Hand Gesture Recognition Based on Auto-Landmark Localization and Reweighted Genetic Algorithm for Healthcare Muscle Activities

et al. 2021

View full text Add to dashboard Cite

Due to the constantly increasing demand for the automatic localization of landmarks in hand gesture recognition, there is a need for a more sustainable, intelligent, and reliable system for hand gesture recognition. The main purpose of this study was to develop an accurate hand gesture recognition system that is capable of error-free auto-landmark localization of any gesture dateable in an RGB image. In this paper, we propose a system based on landmark extraction from RGB images regardless of the environment. The extraction of gestures is performed via two methods, namely, fused and directional image methods. The fused method produced greater extracted gesture recognition accuracy. In the proposed system, hand gesture recognition (HGR) is done via several different methods, namely, (1) HGR via point-based features, which consist of (i) distance features, (ii) angular features, and (iii) geometric features; (2) HGR via full hand features, which are composed of (i) SONG mesh geometry and (ii) active model. To optimize these features, we applied gray wolf optimization. After optimization, a reweighted genetic algorithm was used for classification and gesture recognition. Experimentation was performed on five challenging datasets: Sign Word, Dexter1, Dexter + Object, STB, and NYU. Experimental results proved that auto landmark localization with the proposed feature extraction technique is an efficient approach towards developing a robust HGR system. The classification results of the reweighted genetic algorithm were compared with Artificial Neural Network (ANN) and decision tree. The developed system plays a significant role in healthcare muscle exercise.

show abstract

“…Hand gesture is an excellent way of communicating with the deaf and dumb people. One of the systematic sets of hand gestures belongs to sign language (Starner and Pentland 1997;Sharma and Singh 2021;Mahdikhanlou and Ebrahimnezhad 2020). HCI technology is increasingly penetrating the medical field.…”

Section: Related Workmentioning

confidence: 99%

Object manipulation and deformation using hand gestures

Mahdikhanlou

Ebrahimnezhad

2021

J Ambient Intell Human Comput

Self Cite

View full text Add to dashboard Cite

Compared to the mouse, as a two-dimensional and precise interface device, hand gestures provide more degrees of freedom for users to interact with computers by employing intelligent computing methods. Leap Motion Controller is gaining more popularity due to its ability to detect and track hand joints in three dimensions. However, in some cases, the Leap Motion Controller measurements are not correct enough. We show that the occlusion, palm angle, and the limited field of view are the main downsides of the Leap Motion Controller. In this paper, a framework is proposed to manipulate and deform a three-dimensional object by hand gestures. We select only a few gestures so that the system instructions can be easily memorized. The gestures are not defined very strictly, so users can do them properly without getting tired. We propose that calculating a reliable space from a Leap Motion Controller can significantly reduce these problems. To deform objects, the Free Form Deformation technique is used, which allows for more local deformation. The selected gestures and determined space for interaction make the deformation framework achieve a balance between the accuracy, user-factors, required tasks for deformation, and limitation of the hand tracking device. The proposed method, compared to related studies, offers more creative methods for deforming objects and more natural movements to interact with the system. According to the conducted user study, a significant difference is observed between hand gesture interaction and mouse in terms of speed and number of attempts.

show abstract

Multimodal 3D American sign language recognition for static alphabet and numbers using hand joints and shape coding

Cited by 15 publications

References 40 publications

ASL Recognition by the Layered Learning Model Using Clustered Groups

ASL Recognition by the Layered Learning Model Using Clustered Groups

Hand Gesture Recognition Based on Auto-Landmark Localization and Reweighted Genetic Algorithm for Healthcare Muscle Activities

Object manipulation and deformation using hand gestures

Contact Info

Product

Resources

About