Non-manual cues in automatic sign language recognition

Asteriadis, Stylianos; Karpouzis, Kostas

doi:10.1007/s00779-012-0615-1

Cited by 23 publications

(6 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most of the previous researches have focused on individual modalities, such as the face ( [4], [5]), head pose [6], mouth [7]- [9], eye-gaze [10] and body pose ( [11], [12]), where the features can be classified as, manual features (intentional expressions when performing a sign such as hand gesture and body movement) and, non-manual features (un-intentional expressions when performing a sign such as lip movement and eye gaze). Researchers have mainly attended on using the manual features for SLR [13]- [15], and have ignored the important and rich information International Journal on Advances in ICT for Emerging Regions in the non-manual features.…”

Section: Multi-modalitymentioning

confidence: 99%

Multi-modal Deep Learning Approach to Improve Sentence level Sinhala Sign Language Recognition

Haputhanthri,

Tennakoon,

Wijesekara

et al. 2023

Int J on Adv. in ICT for Emerging Countries

View full text Add to dashboard Cite

Sign language is used across the world for communication purposes within hearing-impaired communities. Hearing people are not well versed in sign language and most hearing-impaired are not good in general text, creating a communication barrier. Research on Sign Language Recognition (SLR) systems have shown admirable solutions for this issue. In Sri Lanka, machine learning along with neural networks has been the prominent domain of research in Sinhala SLR. All previous research is mainly focused on word-level SLR using hand gestures for translation. While this works for a certain vocabulary, there are many signs interpreted through other spatial cues like lip movements and facial expressions. Therefore, translation is limited and sometimes the interpretations can be misleading. In this research, we propose a multi-modal Deep Learning approach that can effectively recognize sentence-level sign gestures using hand and lip movements and translate to Sinhala text. The model consists of modules for visual feature extraction (ResNet), contextual relationship modeling (transformer encoder with multi-head attention), alignment (CTC) and decoding (Prefix beam search). A dataset consisting 22 of sentences used for evaluations was collected under controlled conditions for a specific day-to-day scenario (a conversation between a vendor and a customer in a shop). The proposed model achieves a best Word Error Rate (WER) of 12.70 on the testing split, improving over the single-stream model which shows a best WER of 17.41, suggesting a multi-modal approach improves overall SLR.

show abstract

Section: Multi-modalitymentioning

confidence: 99%

Multi-modal Deep Learning Approach to Improve Sentence level Sinhala Sign Language Recognition

Haputhanthri,

Tennakoon,

Wijesekara

et al. 2023

Int J on Adv. in ICT for Emerging Countries

View full text Add to dashboard Cite

show abstract

“…For the doubt expression case, we found that the distance and angle features presented small changes between cases where labels were positive or negative, mainly due to the type of expression which is characterized by a slight contraction of the eyes and mouth; thus, we included two extra features for the left eyebrow (90,91,92,93,94,21,22,23,24,25), mouth (48,49,50,51,52,53,54,55,56,57,58,59), and left eye (0, 1, 2, 3, 4, 5, 6, 7), which are the enclosed area [66] and the principal axes ratio or eccentricity [67], which allowed us to better identify the characteristic patterns of this expression. Additionally, for the doubt expression case, we did not concatenate the features that belong to each frame in a window.…”

Section: Feature Extraction and Perspective Constructionmentioning

confidence: 99%

“…GFE has been used in data-driven machine learning approaches for various sign languages, such as American (ASL) [21][22][23], German [24], Czech [25], and Turkish [26], among others. In the literature, it has been proposed that a classifier should learn to recognize syntactic-type GFE in "Libras" sign language (Brazilian sign language) using a vector of features composed of distance, angles and deep points, extracted from the points of the contour of the face (which were captured by a deep camera) [9].…”

Section: Introductionmentioning

confidence: 99%

Facial Expression Recognition from Multi-Perspective Visual Inputs and Soft Voting

Aguileta

Brena

Molino-Minero-Re

et al. 2022

Sensors

View full text Add to dashboard Cite

Automatic identification of human facial expressions has many potential applications in today’s connected world, from mental health monitoring to feedback for onscreen content or shop windows and sign-language prosodic identification. In this work we use visual information as input, namely, a dataset of face points delivered by a Kinect device. The most recent work on facial expression recognition uses Machine Learning techniques, to use a modular data-driven path of development instead of using human-invented ad hoc rules. In this paper, we present a Machine-Learning based method for automatic facial expression recognition that leverages information fusion architecture techniques from our previous work and soft voting. Our approach shows an average prediction performance clearly above the best state-of-the-art results for the dataset considered. These results provide further evidence of the usefulness of information fusion architectures rather than adopting the default ML approach of features aggregation.

show abstract

“…Classification was most often carried out using: hidden Markov models (HMM), e.g., [30][31][32], artificial neural networks (ANN), e.g., [33][34][35], dynamic time warping (DTW), e.g., [27,36], and other methods. Vision methods allow natural interaction and inclusion of non-manual features [37,38], but they are dependent on lighting conditions, background colors, and the user's clothing. Therefore, such solutions work only in controlled laboratory conditions.…”

Section: Recent Workmentioning

confidence: 99%

Recognition of Signed Expressions in an Experimental System Supporting Deaf Clients in the City Office

Kapuściński

Wysocki

2020

Sensors

View full text Add to dashboard Cite

The paper addresses the recognition of dynamic Polish Sign Language expressions in an experimental system supporting deaf people in an office when applying for an ID card. A method of processing a continuous stream of RGB-D data and a feature vector are proposed. The classification is carried out using the k-nearest neighbors algorithm with dynamic time warping, hidden Markov models, and bidirectional long short-term memory. The leave-one-subject-out protocol is used for the dataset containing 121 Polish Sign Language sentences performed five times by four deaf people. A data augmentation method is also proposed and tested. Preliminary observations and conclusions from the use of the system in a laboratory, as well as in real conditions with an experimental installation in the Office of Civil Affairs are given.

show abstract

Non-manual cues in automatic sign language recognition

Cited by 23 publications

References 27 publications

Multi-modal Deep Learning Approach to Improve Sentence level Sinhala Sign Language Recognition

Multi-modal Deep Learning Approach to Improve Sentence level Sinhala Sign Language Recognition

Facial Expression Recognition from Multi-Perspective Visual Inputs and Soft Voting

Recognition of Signed Expressions in an Experimental System Supporting Deaf Clients in the City Office

Contact Info

Product

Resources

About