May the force be with you: Force-aligned signwriting for automatic subunit annotation of corpora

Koller, Oscar; Ney, Hermann; Bowden, Richard

doi:10.1109/fg.2013.6553777

Cited by 17 publications

(13 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Le et al (2015) followed this line of thought for gesture recognition, but only employed a shallow legacy neural network that was trained to distinguish twelve artificial actions. Koller et al (2013Koller et al ( , 2014 achieved important results using GMM-HMMs for weakly supervised learning in the domain of sign language. However, hybrid models strongly outperformed the results (Koller et al 2016a), which constituted first and preliminary work in this direction.…”

Section: Related Workmentioning

confidence: 99%

Deep Sign: Enabling Robust Statistical Continuous Sign Language Recognition via Hybrid CNN-HMMs

et al. 2018

Self Cite

View full text Add to dashboard Cite

This manuscript introduces the end-to-end embedding of a CNN into a HMM, while interpreting the outputs of the CNN in a Bayesian framework. The hybrid CNN-HMM combines the strong discriminative abilities of CNNs with the sequence modelling capabilities of HMMs. Most current approaches in the field of gesture and sign language recognition disregard the necessity of dealing with sequence data both for training and evaluation. With our presented end-to-end embedding we are able to improve over the state-of-the-art on three challenging benchmark continuous sign language recognition tasks by between 15 and 38% relative reduction in word error rate and up to 20% absolute. We analyse the effect of the CNN structure, network pretraining and number of hidden states. We compare the hybrid modelling to a tandem approach and evaluate the gain of model combination.

show abstract

Section: Related Workmentioning

confidence: 99%

Deep Sign: Enabling Robust Statistical Continuous Sign Language Recognition via Hybrid CNN-HMMs

et al. 2018

Self Cite

View full text Add to dashboard Cite

show abstract

“…The German SignWriting lexicon currently comprises 24.293 entries. Inspired by [19], we parsed all entries to create the mapping ψ from sign annotations to possible hand shape sequences, where we remove all hand pose related information (such as rotations) of the hand annotations. This mapping will be made available, in order to make our results reproducible.…”

Section: Data Setsmentioning

confidence: 99%

Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled

Koller

Ney

Bowden

2016

2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

215

150

View full text Add to dashboard Cite

This work presents a new approach to learning a framebased classifier on weakly labelled sequence data by embedding a CNN within an iterative EM algorithm. This allows the CNN to be trained on a vast number of example images when only loose sequence level information is available for the source videos. Although we demonstrate this in the context of hand shape recognition, the approach has wider application to any video recognition task where frame level labelling is not available. The iterative EM algorithm leverages the discriminative ability of the CNN to iteratively refine the frame level annotation and subsequent training of the CNN. By embedding the classifier within an EM framework the CNN can easily be trained on 1 million hand images. We demonstrate that the final classifier generalises over both individuals and data sets. The algorithm is evaluated on over 3000 manually labelled hand shape images of 60 different classes which will be released to the community. Furthermore, we demonstrate its use in continuous sign language recognition on two publicly available large sign language data sets, where it outperforms the current state-of-the-art by a large margin. To our knowledge no previous work has explored expectation maximization without Gaussian mixture models to exploit weak sequence labels for sign language recognition.

show abstract

“…Cooper et al used hand labeled data and compared three types of subunits: appearance-based, 2D tracking-based and 3D tracking-based [30]. Koller et al used gloss annotations and gloss time boundaries to generate sequences of subunits using HMM-based modeling and expectation-maximization algorithm [31]. (Gloss is a written form in sign language to provide semantic labels to sign.…”

Section: Relevant Literaturementioning

confidence: 99%

Subunits Inference and Lexicon Development Based on Pairwise Comparison of Utterances and Signs

Tornay

Magimai.-Doss

2019

Information

View full text Add to dashboard Cite

Communication languages convey information through the use of a set of symbols or units. Typically, this unit is word. When developing language technologies, as words in a language do not have the same prior probability, there may not be sufficient training data for each word to model. Furthermore, the training data may not cover all possible words in the language. Due to these data sparsity and word unit coverage issues, language technologies employ modeling of subword units or subunits, which are based on prior linguistic knowledge. For instance, development of speech technologies such as automatic speech recognition system presume that there exists a phonetic dictionary or at least a writing system for the target language. Such knowledge is not available for all languages in the world. In that direction, this article develops a hidden Markov model-based abstract methodology to extract subword units given only pairwise comparison between utterances (or realizations of words in the mode of communication), i.e., whether two utterances correspond to the same word or not. We validate the proposed methodology through investigations on spoken language and sign language. In the case of spoken language, we demonstrate that the proposed methodology can lead up to discovery of phone set and development of phonetic dictionary. In the case of sign language, we demonstrate how hand movement information can be effectively modeled for sign language processing and synthesized back to gain insight about the derived subunits.

show abstract

May the force be with you: Force-aligned signwriting for automatic subunit annotation of corpora

Cited by 17 publications

References 16 publications

Deep Sign: Enabling Robust Statistical Continuous Sign Language Recognition via Hybrid CNN-HMMs

Deep Sign: Enabling Robust Statistical Continuous Sign Language Recognition via Hybrid CNN-HMMs

Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled

Subunits Inference and Lexicon Development Based on Pairwise Comparison of Utterances and Signs

Contact Info

Product

Resources

About