Tracking Benchmark Databases for Video-Based Sign Language Recognition

Abstract

A survey of video databases that can be used within a continuous sign language recognition scenario to measure the performance of head and hand tracking algorithms either w.r.t. a tracking error rate or w.r.t. a word error rate criterion is presented in this work. Robust tracking algorithms are required as the signing hand frequently moves in front of the face, may temporarily disappear, or cross the other hand. Only few studies consider the recognition of continuous sign language, and usually special devices such as colored gloves or blue-boxing environments are used to accurately track the regions-of-interest in sign language processing. Ground-truth labels for hand and head positions have been annotated for more than 30k frames in several publicly available video databases of different degrees of difficulty, and preliminary tracking results are presented.

The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities

“…In recent years, sequence modeling tools have received an increasing amount of attention, as shown, for instance, in the survey by Weinland et al [24], in part because of their ability to model the temporal structure of actions at multiple levels of granularity. HMMs are a particularly popular framework as they have been shown to work well not only for the recognition of events but also for the parsing and segmentation of videos [10] with applications ranging from sign language understanding [6,14] to the evaluation of motor skills including the training of surgeons [26]. In the context of the recognition of human actions in video, Chen and Aggarval [5] use the output of an SVM to classify complete activities with HMMs, reaching a recognition accuracy of 90.9% on the KTH dataset.…”

“…As a result, behavioral analysis with HMMs is often done with motion capture data or other sensors [10] or, in the case of video-based action recognition, with object, hand and head trajectories [6,26]. This has typically forced researchers working with HMMs to work in controlled environments with restrictive setups.…”

Review of Literature for the Development of Indian Sign Language Recognition System

“…Dreuw et al [21] provided criteria to evaluate hand and head tracking algorithms; the proposed criteria were tracking error rate (TER) and word error rate (WER). Dynamic programming tracking (DPT) avoided taking possibly wrong local decisions by tracking back from the last frame of the sequence to the first frame to get a best path with the highest score given by a score function, which was calculated for every frame starting from the very beginning.…”

Classification of extreme facial events in sign language videos

“…However, manual annotation is conducted by experts and is a highly time-consuming task (in [7] is described as 'enormous', resulting on annotations of 'only a small proportion of data'), justifying their general lack. Simultaneously, more SL data, many of which lack facial annotations, are built or accumulated on the web [8][9][10][11]. All the above led on efforts towards the development of automatic or semiautomatic annotation tools [12][13][14] for the processing of corpora.…”