Read My Lips: Continuous Signer Independent Weakly Supervised Viseme Recognition

Koller, Oscar; Ney, Hermann; Bowden, Richard

doi:10.1007/978-3-319-10590-1_19

Cited by 16 publications

(21 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another approach uses Expectation Maximisation (EM) [7] to fit a model to data observations. Koller et al [20] used EM to fit a Gaussian Mixture Model (GMM) to Active Appearance Model (AAM) mouth features in order to find and model mouth shape sequences in sign language. Other works use EM to link text and image regions [37].…”

Section: State-of-the-artmentioning

confidence: 99%

Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled

Koller

Ney

Bowden

2016

2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

215

150

View full text Add to dashboard Cite

This work presents a new approach to learning a framebased classifier on weakly labelled sequence data by embedding a CNN within an iterative EM algorithm. This allows the CNN to be trained on a vast number of example images when only loose sequence level information is available for the source videos. Although we demonstrate this in the context of hand shape recognition, the approach has wider application to any video recognition task where frame level labelling is not available. The iterative EM algorithm leverages the discriminative ability of the CNN to iteratively refine the frame level annotation and subsequent training of the CNN. By embedding the classifier within an EM framework the CNN can easily be trained on 1 million hand images. We demonstrate that the final classifier generalises over both individuals and data sets. The algorithm is evaluated on over 3000 manually labelled hand shape images of 60 different classes which will be released to the community. Furthermore, we demonstrate its use in continuous sign language recognition on two publicly available large sign language data sets, where it outperforms the current state-of-the-art by a large margin. To our knowledge no previous work has explored expectation maximization without Gaussian mixture models to exploit weak sequence labels for sign language recognition.

show abstract

Section: State-of-the-artmentioning

confidence: 99%

Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled

Koller

Ney

Bowden

2016

2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

215

150

View full text Add to dashboard Cite

show abstract

“…This clustering is motivated by the vision of using this method for facial expression animation in avatar-based SL synthesis. Koller et al [33], [34], utilize the same corpus (RWTH-Phoenix-Weather) as well as a similar approach of extracting high-level facial features to model mouthings in SL. In [33], the authors develop a novel viseme recognition method that is specifically designed for SL, does not require any manual annotation and is signer-independent.…”

Section: Mouth Non-manuals In Existing Aslr Systemsmentioning

confidence: 99%

“…Koller et al [33], [34], utilize the same corpus (RWTH-Phoenix-Weather) as well as a similar approach of extracting high-level facial features to model mouthings in SL. In [33], the authors develop a novel viseme recognition method that is specifically designed for SL, does not require any manual annotation and is signer-independent. In [34], they propose an algorithm that automatically annotates mouthings in SL videos.…”

Section: Mouth Non-manuals In Existing Aslr Systemsmentioning

confidence: 99%

“…Regarding the second line of research, to the best of our knowledge the only method for viseme recognition in the context of ASLR is [33]. Drawbacks of using visemes for description and recognition of mouthings include the fact that there is not a standardized set of visemes (e.g., 13 visemes were used in [35], 16 were used in [25] while 50 visemes were used in [62] to model the effects of coarticulation) and using a low number of visemes may cover a small subspace of the mouth motions represented in the visual domain.…”

Section: B Visual Speech Recognitionmentioning

confidence: 99%

“…Furthermore, the task of viseme recognition is very challenging (even for humans) with reported error rates around 50% [35]. The interested reader may refer to [33] for additional technical challenges in viseme recognition in a setting when there is no manual annotation.…”

Section: B Visual Speech Recognitionmentioning

confidence: 99%

See 2 more Smart Citations

A survey on mouth modeling and analysis for Sign Language recognition

Antonakos

Roussos

Zafeiriou

2015

2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)

View full text Add to dashboard Cite

Abstract-Around 70 million Deaf worldwide use Sign Languages (SLs) as their native languages. At the same time, they have limited reading/writing skills in the spoken language. This puts them at a severe disadvantage in many contexts, including education, work, usage of computers and the Internet. Automatic Sign Language Recognition (ASLR) can support the Deaf in many ways, e.g. by enabling the development of systems for Human-Computer Interaction in SL and translation between sign and spoken language. Research in ASLR usually revolves around automatic understanding of manual signs. Recently, ASLR research community has started to appreciate the importance of non-manuals, since they are related to the lexical meaning of a sign, the syntax and the prosody. Nonmanuals include body and head pose, movement of the eyebrows and the eyes, as well as blinks and squints. Arguably, the mouth is one of the most involved parts of the face in non-manuals. Mouth actions related to ASLR can be either mouthings, i.e. visual syllables with the mouth while signing, or non-verbal mouth gestures. Both are very important in ASLR. In this paper, we present the first survey on mouth non-manuals in ASLR. We start by showing why mouth motion is important in SL and the relevant techniques that exist within ASLR. Since limited research has been conducted regarding automatic analysis of mouth motion in the context of ALSR, we proceed by surveying relevant techniques from the areas of automatic mouth expression and visual speech recognition which can be applied to the task. Finally, we conclude by presenting the challenges and potentials of automatic analysis of mouth motion in the context of ASLR.

show abstract