In sign language, when signed letters are continuously spelled based on backhand view, a previous signed letter influences the trajectory of fingers and a hand to approach the pause duration for signing the current signed letter. Since those varied trajectories are regarded as parts of the current signed letter, hand gesture during pause duration of the current signed letter is regarded as insufficient for recognition of the current signed letter. The previous signed letters, trajectories of fingers, and hand between the previous and the current signed letters should be included as data for classification. This paper proposes a method of backhand-view-based continuous-signed-letter recognition using a rewound video sequence with previous signed letter. In the method, a hand shape of previous signed letter and trajectories of finger joints moving from the previous signed letter to the current one are detected, features are then extracted, and finally, the features are classified for signed letter recognition. To evaluate performance of the proposed method, experiments with 10 participants were performed 20 times each, and the results revealed 96.07% accuracy approximately which were improved significantly from the conventional methods using forehand and backhand.
Most of the existing methods focus mainly on the extraction of shape-based, rotation-based, and motion-based features, usually neglecting the relationship between hands and body parts, which can provide significant information to address the problem of similar sign words based on the backhand approach. Therefore, this paper proposes four feature-based models. The spatial–temporal body parts and hand relationship patterns are the main feature. The second model consists of the spatial–temporal finger joint angle patterns. The third model consists of the spatial–temporal 3D hand motion trajectory patterns. The fourth model consists of the spatial–temporal double-hand relationship patterns. Then, a two-layer bidirectional long short-term memory method is used to deal with time-independent data as a classifier. The performance of the method was evaluated and compared with the existing works using 26 ASL letters, with an accuracy and F1-score of 97.34% and 97.36%, respectively. The method was further evaluated using 40 double-hand ASL words and achieved an accuracy and F1-score of 98.52% and 98.54%, respectively. The results demonstrated that the proposed method outperformed the existing works under consideration. However, in the analysis of 72 new ASL words, including single- and double-hand words from 10 participants, the accuracy and F1-score were approximately 96.99% and 97.00%, respectively.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.