Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translation

Voskou, Andreas; Panousis, Konstantinos P.; Kosmopoulos, Dimitrios; Metaxas, Dimitris N.; Chatzis, Sotirios

doi:10.1109/iccv48922.2021.01173

Cited by 20 publications

(12 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Camgöz et al mention that the most common errors are related to numbers, dates, and places: these can be difficult to derive from context in weather broadcasts [6,37]. The same kind of errors is made by the models of Partaourides et al [70] and Voskou et al [81]. Zheng et al illustrate how their model improves accuracy for longer sentences [64].…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Machine translation from signed to spoken languages: state of the art and challenges

Coster

Shterionov

Herreweghe

et al. 2023

Univ Access Inf Soc

View full text Add to dashboard Cite

Automatic translation from signed to spoken languages is an interdisciplinary research domain on the intersection of computer vision, machine translation (MT), and linguistics. While the domain is growing in terms of popularity—the majority of scientific papers on sign language (SL) translation have been published in the past five years—research in this domain is performed mostly by computer scientists in isolation. This article presents an extensive and cross-domain overview of the work on SL translation. We first give a high level introduction to SL linguistics and MT to illustrate the requirements of automatic SL translation. Then, we present a systematic literature review of the state of the art in the domain. Finally, we outline important challenges for future research. We find that significant advances have been made on the shoulders of spoken language MT research. However, current approaches often lack linguistic motivation or are not adapted to the different characteristics of SLs. We explore challenges related to the representation of SL data, the collection of datasets and the evaluation of SL translation models. We advocate for interdisciplinary research and for grounding future research in linguistic analysis of SLs. Furthermore, the inclusion of deaf and hearing end users of SL translation applications in use case identification, data collection, and evaluation, is of utmost importance in the creation of useful SL translation models.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Six papers use features extracted using a 2D CNNs by first training a CSLR model on RWTH-PHOENIX-Weather 2014 6 [6,36,38,81,92,93]. These papers use the full frame as inputs to the feature extractor.…”

Section: Sign Language Representationsmentioning

confidence: 99%

Machine translation from signed to spoken languages: state of the art and challenges

Coster

Shterionov

Herreweghe

et al. 2023

Univ Access Inf Soc

View full text Add to dashboard Cite

show abstract

“…In the related LWTA literature, the competition process is usually deterministic, i.e., the unit with the highest linear activation is deemed the winner each time. However, novel data-driven stochastic arguments for the competition process have been recently proposed in Theodoridis 2019, 2021;Panousis et al 2021;Voskou et al 2021).…”

Section: Foundational Principlesmentioning

confidence: 99%

Competing Mutual Information Constraints with Stochastic Competition-Based Activations for Learning Diversified Representations

Panousis

Antoniadis

Chatzis

2022

AAAI

Self Cite

View full text Add to dashboard Cite

This work aims to address the long-established problem of learning diversified representations. To this end, we combine information-theoretic arguments with stochastic competition-based activations, namely Stochastic Local Winner-Takes-All (LWTA) units. In this context, we ditch the conventional deep architectures commonly used in Representation Learning, that rely on non-linear activations; instead, we replace them with sets of locally and stochastically competing linear units. In this setting, each network layer yields sparse outputs, determined by the outcome of the competition between units that are organized into blocks of competitors. We adopt stochastic arguments for the competition mechanism, which perform posterior sampling to determine the winner of each block. We further endow the considered networks with the ability to infer the sub-part of the network that is essential for modeling the data at hand; we impose appropriate stick-breaking priors to this end. To further enrich the information of the emerging representations, we resort to information-theoretic principles, namely the Information Competing Process (ICP). Then, all the components are tied together under the stochastic Variational Bayes framework for inference. We perform a thorough experimental investigation for our approach using benchmark datasets on image classification. As we experimentally show, the resulting networks yield significant discriminative representation learning abilities. In addition, the introduced paradigm allows for a principled investigation mechanism of the emerging intermediate network representations.

show abstract

“…Training was performed collaboratively for the entire system (both tasks). The need for that intermediate step has been alleviated in later works such as [ 10 ], where a winner-takes-all activation is integrated into the Transformer architecture. In [ 35 ], the authors introduced a context-aware continuous sign language recognition using a generative adversarial network architecture.…”

Section: Related Workmentioning

confidence: 99%

“…Recent methods based on networks with self-attention (transformers) [ 9 , 10 ], which currently represent the state-of-the-art in SLT, have yielded promising results owing to their ability to learn without depending too much on expert knowledge. Nevertheless, to fully unleash their performance and generalization potential, such systems require large corpora for training, which increase with the size of the vocabulary.…”

Section: Introductionmentioning

confidence: 99%

Context-Aware Automatic Sign Language Video Transcription in Psychiatric Interviews

Pikoulis

Bifis

Trigka

et al. 2022

Sensors

Self Cite

View full text Add to dashboard Cite

Sign language (SL) translation constitutes an extremely challenging task when undertaken in a general unconstrained setup, especially in the absence of vast training datasets that enable the use of end-to-end solutions employing deep architectures. In such cases, the ability to incorporate prior information can yield a significant improvement in the translation results by greatly restricting the search space of the potential solutions. In this work, we treat the translation problem in the limited confines of psychiatric interviews involving doctor-patient diagnostic sessions for deaf and hard of hearing patients with mental health problems.To overcome the lack of extensive training data and be able to improve the obtained translation performance, we follow a domain-specific approach combining data-driven feature extraction with the incorporation of prior information drawn from the available domain knowledge. This knowledge enables us to model the context of the interviews by using an appropriately defined hierarchical ontology for the contained dialogue, allowing for the classification of the current state of the interview, based on the doctor’s question. Utilizing this information, video transcription is treated as a sentence retrieval problem. The goal is predicting the patient’s sentence that has been signed in the SL video based on the available pool of possible responses, given the context of the current exchange. Our experimental evaluation using simulated scenarios of psychiatric interviews demonstrate the significant gains of incorporating context awareness in the system’s decisions.

show abstract

Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translation

Cited by 20 publications

References 24 publications

Machine translation from signed to spoken languages: state of the art and challenges

Machine translation from signed to spoken languages: state of the art and challenges

Competing Mutual Information Constraints with Stochastic Competition-Based Activations for Learning Diversified Representations

Context-Aware Automatic Sign Language Video Transcription in Psychiatric Interviews

Contact Info

Product

Resources

About