Sign Language Recognition

We propose a novel deep learning approach to solve simultaneous alignment and recognition problems (referred to as "Sequence-to-sequence" learning). We decompose the problem into a series of specialised expert systems referred to as SubUNets. The spatio-temporal relationships between these SubUNets are then modelled to solve the task, while remaining trainable end-to-end.The approach mimics human learning and educational techniques, and has a number of significant advantages. SubUNets allow us to inject domain-specific expert knowledge into the system regarding suitable intermediate representations. They also allow us to implicitly perform transfer learning between different interrelated tasks, which also allows us to exploit a wider range of more varied data sources.In our experiments we demonstrate that each of these properties serves to significantly improve the performance of the overarching recognition system, by better constraining the learning problem.The proposed techniques are demonstrated in the challenging domain of sign language recognition. We demonstrate state-of-the-art performance on hand-shape recognition (outperforming previous techniques by more than 30%). Furthermore, we are able to obtain comparable sign recognition rates to previous research, without the need for an alignment step to segment out the signs for recognition.

show abstract

“…This is due to its multi-channel nature [11], and the large amounts of expert linguistic knowledge available.…”

Section: Related Workmentioning

confidence: 99%

SubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition

Camgöz

Hadfield

Koller

et al. 2017

2017 IEEE International Conference on Computer Vision (ICCV)

Self Cite

225

113

View full text Add to dashboard Cite

show abstract

“…Specifically, in the domain of SLs, several works have been focused in the automatic recognition of atomic gestures by characterizing postures, shape regions, global movements among others (see [3] for an overview of the domain). These works include the use of a broad spectrum of methods such as tracking of articulated shapes, colour segmentation to characterize postures, and the static and temporal characterization shape articulators.…”

Section: Background On Motion Descriptorsmentioning

confidence: 99%

“…They do not include automatic processing on the primary data, the video. However, there is a growing interest on image and video processing tools, to characterize particular recorded gestures from local and global primitives such as motion, shape, body parts interactions, among others [2], [3]. This paper introduces a new proposal to support SLs annotations based on a motion descriptor that characterize temporal gestures in video sequences.…”

Section: Introductionmentioning

confidence: 99%

A Gaussian Mixture Representation of Gesture Kinematics for On-Line Sign Language Video Annotation

Martínez

Manzanera

Gouiffès

et al. 2015

Advances in Visual Computing

View full text Add to dashboard Cite

Abstract. Sign languages (SLs) are visuo-gestural representations used by deaf communities. Recognition of SLs usually requires manual annotations, which are expert dependent, prone to errors and time consuming. This work introduces a method to support SL annotations based on a motion descriptor that characterizes dynamic gestures in videos. The proposed approach starts by computing local kinematic cues, represented as mixtures of Gaussians which together correspond to gestures with a semantic equivalence in the sign language corpora. At each frame, a spatial pyramid partition allows a fine-to-coarse sub-regional description of motion-cues distribution. Then for each sub-region, a histogram of motion-cues occurrence is built, forming a frame-gesture descriptor which can be used for on-line annotation. The proposed approach is evaluated using a bagof-features framework, in which every frame-level histogram is mapped to an SVM. Experimental results show competitive results in terms of accuracy and time computation for a signing dataset.

show abstract

“…With respect to sign language, several works exist that exploit weak supervision to learn hand-based sign models [20][21][22][23][24]. Facial features have also been used before.…”

Section: State Of the Artmentioning

confidence: 99%

Read My Lips: Continuous Signer Independent Weakly Supervised Viseme Recognition

Koller

Ney

Bowden

2014

Computer Vision – ECCV 2014

Self Cite

View full text Add to dashboard Cite

Abstract. This work presents a framework to recognise signer independent mouthings in continuous sign language, with no manual annotations needed. Mouthings represent lip-movements that correspond to pronunciations of words or parts of them during signing. Research on sign language recognition has focused extensively on the hands as features. But sign language is multi-modal and a full understanding particularly with respect to its lexical variety, language idioms and grammatical structures is not possible without further exploring the remaining information channels. To our knowledge no previous work has explored dedicated viseme recognition in the context of sign language recognition. The approach is trained on over 180.000 unlabelled frames and reaches 47.1% precision on the frame level. Generalisation across individuals and the influence of context-dependent visemes are analysed.

show abstract

Sign Language Recognition

Cited by 175 publications

References 98 publications

SubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition

SubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition

A Gaussian Mixture Representation of Gesture Kinematics for On-Line Sign Language Video Annotation

Read My Lips: Continuous Signer Independent Weakly Supervised Viseme Recognition

Contact Info

Product

Resources

About