Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled

Abstract

This work presents a new approach to learning a framebased classifier on weakly labelled sequence data by embedding a CNN within an iterative EM algorithm. This allows the CNN to be trained on a vast number of example images when only loose sequence level information is available for the source videos. Although we demonstrate this in the context of hand shape recognition, the approach has wider application to any video recognition task where frame level labelling is not available. The iterative EM algorithm leverages the discriminative ability of the CNN to iteratively refine the frame level annotation and subsequent training of the CNN. By embedding the classifier within an EM framework the CNN can easily be trained on 1 million hand images. We demonstrate that the final classifier generalises over both individuals and data sets. The algorithm is evaluated on over 3000 manually labelled hand shape images of 60 different classes which will be released to the community. Furthermore, we demonstrate its use in continuous sign language recognition on two publicly available large sign language data sets, where it outperforms the current state-of-the-art by a large margin. To our knowledge no previous work has explored expectation maximization without Gaussian mixture models to exploit weak sequence labels for sign language recognition.

SubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition

“…As shown in Table 3, our Hand SubUNet surpasses the hand shape recognition performance of the state-of-the-art CNN-based method proposed by Koller et al [27], by a margin of 18% Top-1 accuracy, which is a relative improvement of 30%. Koller et al [27] iteratively realigned and retrained his network whereas the SubUNet architecture automatically overcomes the frame alignment issue.…”

“…As frame level annotations are hard to come by in continuous datasets, most of the work to date required an alignment step to localize individual signs in videos [10]. The work that is most relevant to this paper is by Koller et al [27] which combines deep-representations with traditional HMM based temporal modelling.…”

“…To evaluate the performance of our network we used the 3361 manually annotated hand images provided by [27], which are from the Development set of the RWTH-PHOENIX-Weather-2014 dataset. Again, because we are interested in the more challenging alignment & recognition problem, we run the system on the full (unseen) test sequences from which these images were taken.…”

“…We use the One-Million Hands [27] dataset for training the Hand SubUNet. The dataset consists of cropped hand images collated from publicly available datasets, including Danish [29], New Zealand [31] and German (RWTH-PHOENIXWeather-2014 [14] sign languages.…”

Particle Filter Based Probabilistic Forced Alignment for Continuous Gesture Recognition

“…In [24], Koller et al propose a CNN-HMM hybrid that learns to localize and recognize hand shapes. They first train a CNN using weak frame level annotations.…”