Enabling Real-time Sign Language Translation on Mobile Platforms with On-board Depth Cameras

Park, Hyeonjung; Lee, Mi Kyung; Ko, JeongGil

doi:10.1145/3463498

Cited by 25 publications

(5 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The depth stream assists in learning more complex features by ignoring the video background. Recently, the SUGO model based on 3D-CNN is proposed that uses data acquired through LIDAR [31]. Appearance-based methods for SLR critically suffer from high computational complexity in terms of memory requirements and processing power and have significantly low accuracies.…”

Section: Related Workmentioning

confidence: 99%

Signgraph: An Efficient and Accurate Pose-Based Graph Convolution Approach Toward Sign Language Recognition

et al. 2023

View full text Add to dashboard Cite

Sign language recognition (SLR) enables the deaf and speech-impaired community to integrate and communicate effectively with the rest of society. Word level or isolated SLR is a fundamental yet complex task with the main objective of using models to correctly recognize signed words. Sign language consists of very fast and complex hand, body, face movements, and mouthing cues that make the task very challenging. Several input modalities; RGB, optical Flow, RGB-D, and pose/skeleton have been proposed for SLR. However, the complexity of these modalities and the state-of-the-art (SOTA) methodologies tend to be exceedingly sophisticated and over-parametrized. In this paper, our focus is to use the hands and body poses as an input modality. One major problem in pose-based SLR is extracting the most valuable and distinctive features for all skeleton joints. In this regard, we propose an accurate, efficient, and lightweight pose-based pipeline leveraging a graph convolution network (GCN) along with residual connections and a bottleneck structure. The proposed architecture not only facilitates efficient learning during model training providing significantly improved accuracy scores but also alleviates computational complexity. With the proposed architecture in place, we are able to achieve improved accuracies on three different subsets of the WLASL dataset and the LSA-64 dataset. Our proposed model outperforms previous SOTA pose-based methods by providing a relative improvement of 8.91%, 27.62%, and 26.97% for WLASL-100, WLASL-300, and WLASL-1000 subsets. Moreover, our proposed model also outperforms previous SOTA appearance-based methods by providing a relative improvement of 2.65% and 5.15% for WLASL-300 and WLASL-1000 subsets. For the LSA-64 dataset, our model is able to achieve 100% test recognition accuracy. We are able to achieve this improved performance with far less computational cost as compared to existing appearancebased methods.

show abstract

Section: Related Workmentioning

confidence: 99%

Signgraph: An Efficient and Accurate Pose-Based Graph Convolution Approach Toward Sign Language Recognition

et al. 2023

View full text Add to dashboard Cite

show abstract

“…The low complexity of such calculations implies low processing power requirements [7], and, thus, ToF cameras are expected to become ubiquitously available on smartphones [15,81]. In fact, ToF cameras on smartphones have been recently used in novel applications such as translating sign language [57], recognizing hand gestures [86], and improving lighting estimation for AR [97]. ToF cameras are known to suffer from several systematic errors caused by the internal temperature of the camera, integration time settings, and infrared demodulation errors [18,19].…”

Section: A Primer On Tof Cameras In Ar Devicesmentioning

confidence: 99%

InDepth

Zhang

Scargill

Vaishnav

et al. 2022

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

Mobile Augmented Reality (AR) demands realistic rendering of virtual content that seamlessly blends into the physical environment. For this reason, AR headsets and recent smartphones are increasingly equipped with Time-of-Flight (ToF) cameras to acquire depth maps of a scene in real-time. ToF cameras are cheap and fast, however, they suffer from several issues that affect the quality of depth data, ultimately hampering their use for mobile AR. Among them, scale errors of virtual objects - appearing much bigger or smaller than what they should be - are particularly noticeable and unpleasant. This article specifically addresses these challenges by proposing InDepth, a real-time depth inpainting system based on edge computing. InDepth employs a novel deep neural network (DNN) architecture to improve the accuracy of depth maps obtained from ToF cameras. The DNN fills holes and corrects artifacts in the depth maps with high accuracy and eight times lower inference time than the state of the art. An extensive performance evaluation in real settings shows that InDepth reduces the mean absolute error by a factor of four with respect to ARCore DepthLab. Finally, a user study reveals that InDepth is effective in rendering correctly-scaled virtual objects, outperforming DepthLab.

show abstract

“…introduced to clinical applications and are being applied to various domains [11][12][13][14][15][16][17][18]. In this work, we follow the paradigm of exploiting machine learning algorithms together with clinical data and compare candidate models that can suit our purposes.…”

Section: Plos Onementioning

confidence: 99%

Predicting speech discrimination scores from pure-tone thresholds—A machine learning-based approach using data from 12,697 subjects

et al. 2021

Self Cite

View full text Add to dashboard Cite

Diagnostic tests for hearing impairment not only determines the presence (or absence) of hearing loss, but also evaluates its degree and type, and provides physicians with essential data for future treatment and rehabilitation. Therefore, accurately measuring hearing loss conditions is very important for proper patient understanding and treatment. In current-day practice, to quantify the level of hearing loss, physicians exploit specialized test scores such as the pure-tone audiometry (PTA) thresholds and speech discrimination scores (SDS) as quantitative metrics in examining a patient’s auditory function. However, given that these metrics can be easily affected by various human factors, which includes intentional (or accidental) patient intervention, there are needs to cross validate the accuracy of each metric. By understanding a “normal” relationship between the SDS and PTA, physicians can reveal the need for re-testing, additional testing in different dimensions, and also potential malingering cases. For this purpose, in this work, we propose a prediction model for estimating the SDS of a patient by using PTA thresholds via a Random Forest-based machine learning approach to overcome the limitations of the conventional statistical (or even manual) methods. For designing and evaluating the Random Forest-based prediction model, we collected a large-scale dataset from 12,697 subjects, and report a SDS level prediction accuracy of 95.05% and 96.64% for the left and right ears, respectively. We also present comparisons with other widely-used machine learning algorithms (e.g., Support Vector Machine, Multi-layer Perceptron) to show the effectiveness of our proposed Random Forest-based approach. Results obtained from this study provides implications and potential feasibility in providing a practically-applicable screening tool for identifying patient-intended malingering in hearing loss-related tests.

show abstract

Enabling Real-time Sign Language Translation on Mobile Platforms with On-board Depth Cameras

Cited by 25 publications

References 49 publications

Signgraph: An Efficient and Accurate Pose-Based Graph Convolution Approach Toward Sign Language Recognition

Signgraph: An Efficient and Accurate Pose-Based Graph Convolution Approach Toward Sign Language Recognition

InDepth

Predicting speech discrimination scores from pure-tone thresholds—A machine learning-based approach using data from 12,697 subjects

Contact Info

Product

Resources

About