American sign language fingerspelling recognition with phonological feature-based tandem models

Abstract

We study the recognition of fingerspelling sequences in American Sign Language from video using tandem-style models, in which the outputs of multilayer perceptron (MLP) classifiers are used as observations in a hidden Markov model (HMM)-based recognizer. We compare a baseline HMMbased recognizer, a tandem recognizer using MLP letter classifiers, and a tandem recognizer using MLP classifiers of phonological features. We present experiments on a database of fingerspelling videos. We find that the tandem approaches outperform an HMM-based baseline, and that phonological feature-based tandem models outperform letter-based tandem models.

Fingerspelling Recognition with Semi-Markov Conditional Random Fields

“…The lists contained English and foreign words, including proper names and common English nouns. For comparison with prior work, we use the same data from signers 1 and 2 as [18,19], as well as additional data from signers 3 and 4. The recording settings, including differences in environment and camera placement across recording sessions, are illustrated in Figure 3.…”

“…Hand localization and segmentation For every signer, we trained a model for hand detection similar to that used in [19,20]. Using manually annotated hand regions, marked as polygonal regions of interest (ROI) in 30 frames, we fit a mixture of Gaussians P hand to the color of the hand pixels in L*a*b color space.…”

“…The only unrestricted fingerspelling recognition work of which we are aware is [19], using HMM-based approaches; we consider this work as the most competitive baseline and compare to it in the experiments section.…”

“…For this purpose we use a set of frame classifiers, each of which classifies either letters or linguistic handshape features. As in [19], we use the linguistic handshape feature set developed by Brentari [6], who proposed seven features to describe handshape in ASL. Each such linguistic feature (not to be confused with feature functions) has 2-7 possible values.…”

“…The use of such segmental feature functions is useful for gesture modeling, where it is natural to consider the trajectory of some measurement or the statistics of an entire segment. In this work we define feature functions based on scores of letter classifiers, as well as classifiers of handshape features suggested by linguistics research on ASL [6,19]. Linguistic handshape features summarize certain important aspects of a given letter, such as the "active" fingers or the flexed/non-flexed status of the fingers; these are hopefully the most salient aspects of each letter, which should be more discriminative for recognition.…”