Enhanced Level Building Algorithm for the Movement Epenthesis Problem in Sign Language Recognition

Abstract

One of the hard problems in automated sign language recognition is the movement epenthesis (me) problem. Movement epenthesis is the gesture movement that bridges two consecutive signs. This effect can be over a long duration and involve variations in hand shape, position, and movement, making it hard to explicitly model these intervening segments. This creates a problem when trying to match individual signs to full sign sentences since for many chunks of the sentence, corresponding to these mes, we do not have models. We present an approach based on version of a dynamic programming framework, called Level Building, to simulataneously segment and match signs to continuous sign language sentences in the presence of movement epenthesis (me). We enhance the classical Level Building framework so that it can accomodate me labels for which we do not have explicit models. This enhanced Level Building algroithm is then coupled with a trigram grammar model to optimally segment and label sign language sentences. We demonstrate the efficiency of the algorithm using a single view video dataset of continuous sign language sentences. We obtain 83% word level recognition rate with the enhanced Level Building approach, as opposed to a 20% recognition rate using a classical Level Building framework on the same dataset. The proposed approach is novel since it does not need explicit models for movement epenthesis.

Smoothed Disparity Maps for Continuous American Sign Language Recognition

“…For our experiments, we use a publicly available Boston-104 database, which has been used in several other works [2,11] and consits of 201 American Sign Language sentences performed by 3 different signers (161 are used for training and 40 for testing [2]). On the average, these sentences consist of 5 words out of a vocabulary of 104 unique words.…”

Weakly Supervised Training of a Sign Language Recognition System Using Multiple Instance Learning Density Matrices

“…Few researchers have addressed the problem of the movement epenthesis without explicitly modeling these movements. Yang et al [9] proposed an American Sign Language recognition method based on an enhanced level building algorithm and a trigram grammar model. Their method was based on a dynamic programming approach to spot signs without explicit movement epenthesis models.…”

Sign Language Recognition by Combining Statistical DTW and Independent Classification

“…One advantage of our approach is that the negative influence of irrelevant variations in preparations, transitions, and retractions can be reduced by applying feature selection on the registered feature set. Instead, Yang et al [22] have resolved this problem by including a separate model with constant distance that is fitted to sequences that do not fit well to any known sign. The disadvantage is that it introduces the possibility of falsely inserting the transition model in the place of a sign that differs more from its model than is accounted for.…”