Gesture and Sign Language Recognition with Temporal Residual Networks

Abstract

Gesture and sign language recognition in a continuous video stream is a challenging task, especially with a large vocabulary. In this work, we approach this as a framewise classification problem. We tackle it using temporal convolutions and recent advances in the deep learning field like residual networks, batch normalization and exponential linear units (ELUs). The models are evaluated on three different

Deep Learning for Sign Language Recognition: Current Techniques, Benchmarks, and Open Issues

“…CNNs have also been used in attempts to resolve the challenging task of gesture and sign language recognition in a constant video stream. For instance, Pigou et al [126] used a deep learning approach and temporal convolutions to address this problem. The CNN model featured certain improvements that made it easier to conduct the classification process.…”

“…However, this approach proved to be extremely challenging because the animations were difficult to work with after processing. While exploring the challenges of continuous translation, Pigou et al [126] observed that deep residual networks can be used to learn patterns in continuous videos containing gestures and signs. The use of deep residual networks can minimize the need for preprocessing.…”

“…In addition, methods that rely on sensors or customized input devices were not given proper consideration. Individual ap- Sign Language [13], [39], [44], [52], [72]- [74], [76], [95], [96], [116]- [120], [125], [128], [131], [134] American Sign Language [38] Italian Sign Language [40], [121], [169] Arabic Sign Language [49]- [51], [65], [71], [81], [97] Chinese Sign Language [70] Argentine Sign Language [63], [64] Danish and New Zealand Sign Language [45], [122] Bengali Sign Language [3], [66], [77], [95], [133] German Sign Language [67] Japanese Sign Language [68], [115], [123], [124] Indian Sign Language [69], [130], [132] Indonesian Sign Language [46] Portuguese Sign Language [126] Dutch Sign Language…”

Word-Level ASL Recognition and Trigger Sign Detection with RF Sensors

“…Although there has been much research related to technologies for the deaf or hard of hearing (HoH) over the past three decades, much of this work has focused on the translation of sign language into voice or text using camera-based or wearable devices. Although sensor augmented gloves [1]- [3] have been reported to typically yield higher gesture recognition rates than camera-based systems [4]- [6], they cannot capture the intricacies of sign languages presented through head and body movements. In contrast, video can capture facial expressions; but require adequate light and a direct line-of-sight to be effective.…”

“…Because it is much easier to recruit hearing participants than deaf participants, many studies on ASL recognition (e.g. [6], [11]- [13]) have used imitation signing data, despite its differences from native ASL data.…”