Aligning accumulative representations for sign language recognition

Kındıroğlu, Ahmet Alp; Özdemir, Oğulcan; Akarun, Lale

doi:10.1007/s00138-022-01367-x

Cited by 10 publications

(3 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 9 shows the studies using the 744-class BosphorusSign22k dataset. Among the past highest performances, Gökçe et al (2020) [57] and Kındıroglu et al (2023) [69] utilized the MC3-18 model, a powerful model capable of processing spatial and temporal information, while Sincan and Keles (2021) [46] benefited from the I3D model. In our proposed method, the relatively simpler MobileNetV2 and MultiChannel-MobileNetV2 models are used, and temporal information is obtained from video data using the frame difference method.…”

Section: Comparison With Other Studiesmentioning

confidence: 99%

Multi-Stream Isolated Sign Language Recognition Based on Finger Features Derived from Pose Data

Akdag,

Baykan

2024

Electronics

View full text Add to dashboard Cite

This study introduces an innovative multichannel approach that focuses on the features and configurations of fingers in isolated sign language recognition. The foundation of this approach is based on three different types of data, derived from finger pose data obtained using MediaPipe and processed in separate channels. Using these multichannel data, we trained the proposed MultiChannel-MobileNetV2 model to provide a detailed analysis of finger movements. In our study, we first subject the features extracted from all trained models to dimensionality reduction using Principal Component Analysis. Subsequently, we combine these processed features for classification using a Support Vector Machine. Furthermore, our proposed method includes processing body and facial information using MobileNetV2. Our final proposed sign language recognition method has achieved remarkable accuracy rates of 97.15%, 95.13%, 99.78%, and 95.37% on the BosphorusSign22k-general, BosphorusSign22k, LSA64, and GSL datasets, respectively. These results underscore the generalizability and adaptability of the proposed method, proving its competitive edge over existing studies in the literature.

show abstract

Section: Comparison With Other Studiesmentioning

confidence: 99%

Multi-Stream Isolated Sign Language Recognition Based on Finger Features Derived from Pose Data

Akdag,

Baykan

2024

Electronics

View full text Add to dashboard Cite

show abstract

“…More recently, SLR researchers have shifted their attention to the successful spatio-temporal deep learning approaches in which they employ 2D and 3D CNNs (Huang et al, 2015 ; Camgöz et al, 2016b ; Koller et al, 2016 ; Joze and Koller, 2018 ; Mittal et al, 2019 ; Gökçe et al, 2020 ; Rastgoo et al, 2020 ; Abdullahi and Chamnongthai, 2022 ; Samaan et al, 2022 ). Moreover, Kındıroğlu et al ( 2019 ), Kındıroglu et al ( 2023 ) have proposed a temporal representation approach for modeling the movement-hold pattern of SLs (Liddell and Johnson, 1989 ), in which authors adapted the pose-based motion representation, PoTion (Choutas et al, 2018 ), to the SLR task.…”

Section: Related Workmentioning

confidence: 99%

Multi-cue temporal modeling for skeleton-based sign language recognition

2023

Self Cite

View full text Add to dashboard Cite

Sign languages are visual languages used as the primary communication medium for the Deaf community. The signs comprise manual and non-manual articulators such as hand shapes, upper body movement, and facial expressions. Sign Language Recognition (SLR) aims to learn spatial and temporal representations from the videos of the signs. Most SLR studies focus on manual features often extracted from the shape of the dominant hand or the entire frame. However, facial expressions combined with hand and body gestures may also play a significant role in discriminating the context represented in the sign videos. In this study, we propose an isolated SLR framework based on Spatial-Temporal Graph Convolutional Networks (ST-GCNs) and Multi-Cue Long Short-Term Memorys (MC-LSTMs) to exploit multi-articulatory (e.g., body, hands, and face) information for recognizing sign glosses. We train an ST-GCN model for learning representations from the upper body and hands. Meanwhile, spatial embeddings of hand shape and facial expression cues are extracted from Convolutional Neural Networks (CNNs) pre-trained on large-scale hand and facial expression datasets. Thus, the proposed framework coupling ST-GCNs with MC-LSTMs for multi-articulatory temporal modeling can provide insights into the contribution of each visual Sign Language (SL) cue to recognition performance. To evaluate the proposed framework, we conducted extensive analyzes on two Turkish SL benchmark datasets with different linguistic properties, BosphorusSign22k and AUTSL. While we obtained comparable recognition performance with the skeleton-based state-of-the-art, we observe that incorporating multiple visual SL cues improves the recognition performance, especially in certain sign classes where multi-cue information is vital. The code is available at: https://github.com/ogulcanozdemir/multicue-slr.

show abstract

“…In this context, sign language has emerged as a crucial mode of communication for the Deaf and hard-of-hearing community ( Sruthi and Lijiya, 2019 ; Wadhawan and Kumar, 2020 ; Ganguly and Deb, 2022 ). Sign languages, which vary across regions and cultures ( Rastgoo et al, 2021 ; Kındıroglu et al, 2023 ), provide a visual and gestural means of expressing thoughts and ideas ( Thakur et al, 2020 ; Kodandaram et al, 2021 ). They offer a bridge to the world for those who may not have access to spoken language.…”

Section: Introductionmentioning

confidence: 99%

A machine learning-driven web application for sign language learning

Orovwode,

Ibukun,

Abubakar

2024

Front. Artif. Intell.

View full text Add to dashboard Cite

Addressing the increasing demand for accessible sign language learning tools, this paper introduces an innovative Machine Learning-Driven Web Application dedicated to Sign Language Learning. This web application represents a significant advancement in sign language education. Unlike traditional approaches, the application’s unique methodology involves assigning users different words to spell. Users are tasked with signing each letter of the word, earning a point upon correctly signing the entire word. The paper delves into the development, features, and the machine learning framework underlying the application. Developed using HTML, CSS, JavaScript, and Flask, the web application seamlessly accesses the user’s webcam for a live video feed, displaying the model’s predictions on-screen to facilitate interactive practice sessions. The primary aim is to provide a learning platform for those who are not familiar with sign language, offering them the opportunity to acquire this essential skill and fostering inclusivity in the digital age.

show abstract

Aligning accumulative representations for sign language recognition

Cited by 10 publications

References 39 publications

Multi-Stream Isolated Sign Language Recognition Based on Finger Features Derived from Pose Data

Multi-Stream Isolated Sign Language Recognition Based on Finger Features Derived from Pose Data

Multi-cue temporal modeling for skeleton-based sign language recognition

A machine learning-driven web application for sign language learning

Contact Info

Product

Resources

About