Watch, Read and Lookup: Learning to Spot Signs from Multiple Supervisors

Momeni, Liliane; Varol, Gül; Albanie, Samuel; Afouras, Triantafyllos; Zisserman, Andrew

doi:10.1007/978-3-030-69544-6_18

Cited by 20 publications

(59 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Alternative approaches investigate the use of Multiple Instance Learning [6,28,48]. Other recent contributions leverage words from audio-aligned subtitles with keyword spotting methods based on mouthing cues [1], dictionaries [45] and attention maps generated by transformers [61] to annotate large numbers of signs, as well as to learn domain invariant features for improved sign recognition through joint training [36].…”

Section: Related Workmentioning

confidence: 99%

“…Similarly to these works, we also aim to automatically annotate sign language videos by making use of audioaligned subtitles. To this end, we make use of prior keyword spotting methods [1,45]. However, differently from all the other methods mentioned above we propose an iterative approach, SPOT-ALIGN, that alternates between repeated sign spotting (to obtain more annotations) and jointly training on the resulting annotations together with dictionary exemplars (to obtain better features for spotting).…”

Section: Related Workmentioning

confidence: 99%

“…However, such approaches are fundamentally brittle-they work best when the user knows exactly which signs of interest were used in the video. Moreover, to build an accurate index of such signs using recent sign spotting techniques [1,26,45] requires a list of appropriate query candidates, which to date have often been obtained from subtitles corresponding to speech transcriptions of the translation, for example from an ASR engine. Our focus is on sign language videos produced by and for signers, that do not contain any speech track, so producing such speech transcriptions is not an option.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Sign Language Video Retrieval with Free-Form Textual Queries

Duarte¹,

Albanie²,

Giró-i-Nieto³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Systems that can efficiently search collections of sign language videos have been highlighted as a useful application of sign language technology. However, the problem of searching videos beyond individual keywords has received limited attention in the literature. To address this gap, in this work we introduce the task of sign language retrieval with free-form 1 textual queries: given a written query (e.g., a sentence) and a large collection of sign language videos, the objective is to find the signing video in the collection that best matches the written query.We propose to tackle this task by learning crossmodal embeddings on the recently introduced large-scale How2Sign dataset of American Sign Language (ASL). We identify that a key bottleneck in the performance of the system is the quality of the sign video embedding which suffers from a scarcity of labelled training data. We, therefore, propose SPOT-ALIGN, a framework for interleaving iterative rounds of sign spotting and feature alignment to expand the scope and scale of available training data. We validate the effectiveness of SPOT-ALIGN for learning a robust sign video embedding through improvements in both sign recognition and the proposed video retrieval task.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Sign Language Video Retrieval with Free-Form Textual Queries

Duarte¹,

Albanie²,

Giró-i-Nieto³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Key shortcomings of existing datasets include: a lack of diversity (in terms of signing environment, number of signer identities, or both), restricted domain of discourse [5,24] (for example, weather broadcasts) and limited scale [3]. MSASL [22], WLASL [25], BSL-DICT [29] and BSL SignBank [14] cover a wide vocabulary, but are restricted to isolated signs. BSLCORPUS [35] provides fine-grained linguistic annotation of conversations and narratives, but is limited to pairs of signers under lab conditions.…”

Section: Related Workmentioning

confidence: 99%

SeeHear: Signer Diarisation and a New Dataset

Albanie

Varol

Momeni

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

In this work, we propose a framework to collect a large-scale, diverse sign language dataset that can be used to train automatic sign language recognition models.The first contribution of this work is SDTRACK, a generic method for signer tracking and diarisation in the wild. Our second contribution is SEEHEAR, a dataset of 90 hours of British Sign Language (BSL) content featuring a wide range of signers, and including interviews, monologues and debates. Using SDTRACK, the SEEHEAR dataset is annotated with 35K active signing tracks, with corresponding signer identities and subtitles, and 40K automatically localised sign labels. As a third contribution, we provide benchmarks for signer diarisation and sign recognition on SEEHEAR.

show abstract

“…However, most other people do not understand any sign language, which severely hinders deaf people from blending into the society. To facilitate communication between the deaf and normal people, researchers have paid attention to sign language recognition (SLR), which aims at recognizing the words or sentences from videos [2], [3], and sign language annotation, which focuses on temporally locating instances of signs among the sequences of continuous gestures [4], [5].…”

mentioning

confidence: 99%

SignBERT: A BERT-Based Deep Learning Framework for Continuous Sign Language Recognition

2021

View full text Add to dashboard Cite

Continuous sign language recognition (CSLR) is a very challenging task in intelligent systems, since it requires to produce real-time responses while performing computationally intensive video analytics and language modeling. Previous studies mainly focus on adopting hidden Markov models or recurrent neural networks with a limited capability to model specific sign languages, and the accuracy can drop significantly when recognizing the signs performed by different signers with non-standard gestures or non-uniform speeds. In this work, we develop a deep learning framework named SignBERT, integrating the bidirectional encoder representations from transformers (BERT) with the residual neural network (ResNet), to model the underlying sign languages and extract spatial features for CSLR. We further propose a multimodal version of SignBERT, which combines the input of hand images with an intelligent feature alignment, to minimize the distance between the probability distributions of the recognition results generated by the BERT model and the hand images. Experimental results indicate that when compared to the performance of alternative approaches for CSLR, our method has better accuracy with significantly lower word error rate on three challenging continuous sign language datasets.INDEX TERMS bidirectional encoder representations from transformers, continuous sign language recognition, deep learning, video analytics I. INTRODUCTION

show abstract

Watch, Read and Lookup: Learning to Spot Signs from Multiple Supervisors

Cited by 20 publications

References 37 publications

Sign Language Video Retrieval with Free-Form Textual Queries

Sign Language Video Retrieval with Free-Form Textual Queries

SeeHear: Signer Diarisation and a New Dataset

SignBERT: A BERT-Based Deep Learning Framework for Continuous Sign Language Recognition

Contact Info

Product

Resources

About