Lexicon-free fingerspelling recognition from video: Data, models, and signer adaptation

Abstract

We study the problem of recognizing video sequences of fingerspelled letters in American Sign Language (ASL). Fingerspelling comprises a significant but relatively understudied part of ASL. Recognizing fingerspelling is challenging for a number of reasons: It involves quick, small motions that are often highly coarticulated; it exhibits significant variation between signers; and there has been a dearth of continuous fingerspelling data collected. In this work we collect and annotate a new data set of continuous fingerspelling videos, compare several types of recognizers, and explore the problem of signer variation. Our best-performing models are segmental (semiMarkov) conditional random fields using deep neural network-based features. In the signer-dependent setting, our recognizers achieve up to about 92% letter accuracy. The multi-signer setting is much more challenging, but with neural network adaptation we achieve up to 83% letter accuracies in this setting.

Fingerspelling Recognition in the Wild With Iterative Visual Attention

“…Early work on sign language recognition from video 2 mainly focused on isolated signs [7,2]. More recent work has focused on continuous sign language recognition and data sets [9,10,18,17]. Specifically for fingerspelling, the ChicagoFSVid data set includes 2400 fingerspelling sequences from 4 native ASL signers.…”

“…Much previous work on sign language recognition, and the vast majority of previous work on fingerspelling recognition, uses some form of hand detection or segmentation to localize the region(s) of interest as an initial step. Kim et al [18,19,17] estimate a signerdependent skin color model using manually annotated hand regions for fingerspelling recognition. Huang et al [15] learn a hand detector based on Faster R-CNN [33] using manually annotated signing hand bounding boxes, and apply it to general sign language recognition.…”

Emphatic fingerspelling as code-mixing in American Sign Language

“…We use a corpus of highly naturalistic data compiled by Kim et al (2017) in an ongoing study. Videos of ASL signers were mined from public, online sources, all available on YouTube.…”

American Sign Language Fingerspelling Recognition in the Wild

“…The ChicagoFSVid data set is the largest of which we are aware; it includes 4 native ASL signers fingerspelling 600 sequences each, and has been used in 3 The data set is available for download from http://ttic.edu/livescu/chicago-fingerspellingin-the-wild. recent work on lexicon-free recognition and signer adaptation [13,8]. The National Center for Sign Language and Gesture Resources (NCSLGR) Corpus includes about 1,500 fingerspelling sequences (as well as a variety of other ASL signs) [14,15].…”

“…As in a number of other domains, convolutional neural networks (CNNs) have recently been replacing engineered features in sign language recognition research [17,18,19,11,8]. For sequence modeling, most previous work has used hidden Markov models (HMMs) [20,17,18,13], and some has used segmental conditional random fields [21,22,13]. Much of this work relies on frame-level labels for the training data.…”