Sign Transition Modeling and a Scalable Solution to Continuous Sign Language Recognition for Real-World Applications

Abstract

We propose a new approach to modeling transition information between signs in continuous Sign Language Recognition (SLR) and address some scalability issues in designing SLR systems. In contrast to Automatic Speech Recognition (ASR) in which the transition between speech sounds is often brief and mainly addressed by the coarticulation effect, the sign transition in continuous SLR is far from being clear and usually not easily and exactly characterized. Leveraging upon hidden Markov modeling techniques from ASR, we proposed a modeling framework for continuous SLR having the following major advantages, namely: (i) the system is easy to scale up to large-vocabulary SLR; (ii) modeling of signs as well as the transitions between signs is robust even for noisy data collected in real-world SLR; and (iii) extensions to training, decoding, and adaptation are directly applicable even with new deep learning algorithms. A pair of low-cost digital gloves affordable for the deaf and hard of hearing community is used to collect a collection of training and testing data for real-world SLR interaction applications. Evaluated on 1,024 testing sentences from five signers, a word accuracy rate of 87.4% is achieved using a vocabulary of 510 words. The SLR speed is in real time, requiring an average of 0.69s per sentence. The encouraging results indicate that it is feasible to develop real-world SLR applications based on the proposed SLR framework.

Towards Domain-independent Complex and Fine-grained Gesture Recognition with RFID

“…Wearable-sensor based methods leverage the sensors embedded in wearable devices to capture the hand or finger movements. For example, researchers integrate gyroscopes and accelerometers into a glove to track finger movements [15]. ArmTrak [23] fuse inertial signals of a smartwatch and the anatomy of arm joints to trace the geometric motion of the arm.…”

Lexicon-free fingerspelling recognition from video: Data, models, and signer adaptation

“…A number of linguistically motivated representations of handshape and motion have been used in some prior work, e.g., [34,4,47,48,56,57,46,58,59], and multiple systems of phonological and phonetic features have been developed by linguists [60,2]. One of the unique aspects of sign language is that transitional movements occupy a larger portion of the signal than steady states, and some researchers have developed approaches for explicitly modeling the transitions as units [61,62,63,64].…”

Chinese Sign Language Recognition Based on an Optimized Tree-Structure Framework

“…Gao et al [10] firstly reported a continuous SLR work over 5113 signs with two sensory gloves and three trackers, and achieved 90.8% average classification rate. Li et al [11] collected real-world data using a pair of low-cost digital gloves and achieved 87.4% word accuracy in the evaluation of 1024 testing sentences involving 510 Chinese sign language (CSL) words. Nevertheless, wearing the cumbersome data glove to collect hand and finger movements will potentially go against the convenient, efficient and natural intention of HCI [1], [4], [9].…”