Modality Combination Techniques for Continuous Sign Language Recognition

The 21st International ACM SIGACCESS Conference on Computers and Accessibility

Bellard

et al. 2019

Self Cite

266

130

Developing successful sign language recognition, generation, and translation systems requires expertise in a wide range of fields, including computer vision, computer graphics, natural language processing, human-computer interaction, linguistics, and Deaf culture. Despite the need for deep interdisciplinary knowledge, existing research occurs in separate disciplinary silos, and tackles separate portions of the sign language processing pipeline. This leads to three key questions: 1) What does an interdisciplinary view of the current landscape reveal? 2) What are the biggest challenges facing the field? and 3) What are the calls to action for people working in the field? To help answer these questions, we brought together a diverse group of experts for a two-day workshop. This paper presents the results of that interdisciplinary workshop, providing key background that is often overlooked by computer scientists, a review of the state-of-the-art, a set of pressing challenges, and a call to action for the research community.Each group focused on the following questions:

Section: Recognition and Computer Visionmentioning

confidence: 99%

Sign Language Recognition, Generation, and Translation

Bragg

The 21st International ACM SIGACCESS Conference on Computers and Accessibility

Bellard

et al. 2019

Self Cite

266

130

“…Brand proposes an algorithm ("N-head dynamic programming") that reaches O(T (N M ) 2 ) complexity [21]. The asynchronous HMM requires both streams to share the same topology [23]. This is imposed due to the assumption that there is a single underlying hidden process that has multiple distinct probabilities to emit an observation on all or just on a single stream.…”

Section: Related Workmentioning

confidence: 99%

“…However, they all deal with signs in isolation (not in a continuous sentence sequence), which essentially turns the multi-stream HMM with synchronisation at the end of signs into a standard late fusion approach and is therefore not comparable to the approach analysed in this paper. In 2013, Forster et al [23] compared different modality combination techniques to recognise continuous DGS with a vocabulary of up to 455 signs. They use multistream HMMs with synchronisation constraints at the wordlevel.…”

Section: Related Workmentioning

confidence: 99%

Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos

IEEE Trans. Pattern Anal. Mach. Intell.

Camgöz

Ney

et al. 2020

Self Cite

228

117

In this work we present a new approach to the field of weakly supervised learning in the video domain. Our method is relevant to sequence learning problems which can be split up into sub-problems that occur in parallel. Here, we experiment with sign language data. The approach exploits sequence constraints within each independent stream and combines them by explicitly imposing synchronisation points to make use of parallelism that all sub-problems share. We do this with multi-stream HMMs while adding intermediate synchronisation constraints among the streams. We embed powerful CNN-LSTM models in each HMM stream following the hybrid approach. This allows the discovery of attributes which on their own lack sufficient discriminative power to be identified. We apply the approach to the domain of sign language recognition exploiting the sequential parallelism to learn sign language, mouth shape and hand shape classifiers. We evaluate the classifiers on three publicly available benchmark data sets featuring challenging real-life sign language with over 1000 classes, full sentence based lipreading and articulated hand shape recognition on a fine-grained hand shape taxonomy featuring over 60 different hand shapes. We clearly outperform the state-of-the-art on all data sets and observe significantly faster convergence using the parallel alignment approach.

“…, s T defined by the symbol sequences in ψ. For an efficient implementation, following [11], we assume a first order Markov dependency and maximum approximation:…”

Section: Sequential Time-decodingmentioning

confidence: 99%

Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled

2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Ney

Bowden

2016

Self Cite

215

150

This work presents a new approach to learning a framebased classifier on weakly labelled sequence data by embedding a CNN within an iterative EM algorithm. This allows the CNN to be trained on a vast number of example images when only loose sequence level information is available for the source videos. Although we demonstrate this in the context of hand shape recognition, the approach has wider application to any video recognition task where frame level labelling is not available. The iterative EM algorithm leverages the discriminative ability of the CNN to iteratively refine the frame level annotation and subsequent training of the CNN. By embedding the classifier within an EM framework the CNN can easily be trained on 1 million hand images. We demonstrate that the final classifier generalises over both individuals and data sets. The algorithm is evaluated on over 3000 manually labelled hand shape images of 60 different classes which will be released to the community. Furthermore, we demonstrate its use in continuous sign language recognition on two publicly available large sign language data sets, where it outperforms the current state-of-the-art by a large margin. To our knowledge no previous work has explored expectation maximization without Gaussian mixture models to exploit weak sequence labels for sign language recognition.