Transferring Cross-Domain Knowledge for Video Sign Language Recognition

Li, Dongxu; Yu, Xin; Xu, Chenchen; Petersson, Lars; Li, Hongdong

doi:10.1109/cvpr42600.2020.00624

Cited by 103 publications

(49 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Alternative approaches investigate the use of Multiple Instance Learning [6,28,48]. Other recent contributions leverage words from audio-aligned subtitles with keyword spotting methods based on mouthing cues [1], dictionaries [45] and attention maps generated by transformers [61] to annotate large numbers of signs, as well as to learn domain invariant features for improved sign recognition through joint training [36].…”

Section: Related Workmentioning

confidence: 99%

Sign Language Video Retrieval with Free-Form Textual Queries

Duarte¹,

Albanie²,

Giró-i-Nieto³

et al. 2022

Preprint

View full text Add to dashboard Cite

Systems that can efficiently search collections of sign language videos have been highlighted as a useful application of sign language technology. However, the problem of searching videos beyond individual keywords has received limited attention in the literature. To address this gap, in this work we introduce the task of sign language retrieval with free-form 1 textual queries: given a written query (e.g., a sentence) and a large collection of sign language videos, the objective is to find the signing video in the collection that best matches the written query.We propose to tackle this task by learning crossmodal embeddings on the recently introduced large-scale How2Sign dataset of American Sign Language (ASL). We identify that a key bottleneck in the performance of the system is the quality of the sign video embedding which suffers from a scarcity of labelled training data. We, therefore, propose SPOT-ALIGN, a framework for interleaving iterative rounds of sign spotting and feature alignment to expand the scope and scale of available training data. We validate the effectiveness of SPOT-ALIGN for learning a robust sign video embedding through improvements in both sign recognition and the proposed video retrieval task.

show abstract

Section: Related Workmentioning

confidence: 99%

Sign Language Video Retrieval with Free-Form Textual Queries

Duarte¹,

Albanie²,

Giró-i-Nieto³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…On the other hand, Zhang et al in [ 70 ], proposed the Multiple extraction and Multiple prediction (MEMP) network that consists of alternating 3D-CNN networks and Convolutional LSTM layers that extracted spatio-temporal features from video sequences multiple times, enabling the network to achieve 99.06% and 78.85% accuracy in the LSA64 and IsoGD datasets, respectively. Li et al in [ 71 ], proposed a SLR method that was based on the transferring of cross-domain knowledge of news signs to a base model and improve its performance using domain-invariant features.…”

Section: Sign Language Recognitionmentioning

confidence: 99%

Artificial Intelligence Technologies for Sign Language

Papastratis

Chatzikonstantinou

Konstantinidis

et al. 2021

Sensors

View full text Add to dashboard Cite

AI technologies can play an important role in breaking down the communication barriers of deaf or hearing-impaired people with other communities, contributing significantly to their social inclusion. Recent advances in both sensing technologies and AI algorithms have paved the way for the development of various applications aiming at fulfilling the needs of deaf and hearing-impaired communities. To this end, this survey aims to provide a comprehensive review of state-of-the-art methods in sign language capturing, recognition, translation and representation, pinpointing their advantages and limitations. In addition, the survey presents a number of applications, while it discusses the main challenges in the field of sign language technologies. Future research direction are also proposed in order to assist prospective researchers towards further advancing the field.

show abstract

“…Much of the prior work has been limited to data collected in a controlled environment. There has been a growing interest in sign language recognition "in the wild" (naturally occurring sign language media), which includes challenging visual conditions such as lighting variation, visual clutter, and motion blur, and often also more natural signing styles [15,16,3,24,32,33,47,46]. Two recently released datasets of fingerspelling in the wild [47,46] include data from 168 signers and tens of thousands of fingerspelling segments; these are the testbeds used in our experiments.…”

Section: Related Workmentioning

confidence: 99%

Fingerspelling Detection in American Sign Language

Shi¹,

Brentari²,

Shakhnarovich³

et al. 2021

Preprint

View full text Add to dashboard Cite

Figure 1: Fingerspelling detection and recognition in video of American Sign Language. The goal of detection is to find intervals corresponding to fingerspelling (here indicated by open/close parentheses), and the goal of recognition is to transcribe each of those intervals into letter sequences. Our focus in this paper is on detection that enables accurate recognition. In this example (with downsampled frames), the fingerspelled words are PIRATES and PATRICK, shown along with their canonical handshapes aligned roughly with the most-canonical corresponding frames. Non-fingerspelled signs are labeled with their glosses. The English translation is "Moving furtively, pirates steal the boy Patrick."

show abstract

Transferring Cross-Domain Knowledge for Video Sign Language Recognition

Cited by 103 publications

References 31 publications

Sign Language Video Retrieval with Free-Form Textual Queries

Sign Language Video Retrieval with Free-Form Textual Queries

Artificial Intelligence Technologies for Sign Language

Fingerspelling Detection in American Sign Language

Contact Info

Product

Resources

About