Subunits Inference and Lexicon Development Based on Pairwise Comparison of Utterances and Signs

Abstract

Communication languages convey information through the use of a set of symbols or units. Typically, this unit is word. When developing language technologies, as words in a language do not have the same prior probability, there may not be sufficient training data for each word to model. Furthermore, the training data may not cover all possible words in the language. Due to these data sparsity and word unit coverage issues, language technologies employ modeling of subword units or subunits, which are based on prior linguistic knowledge. For instance, development of speech technologies such as automatic speech recognition system presume that there exists a phonetic dictionary or at least a writing system for the target language. Such knowledge is not available for all languages in the world. In that direction, this article develops a hidden Markov model-based abstract methodology to extract subword units given only pairwise comparison between utterances (or realizations of words in the mode of communication), i.e., whether two utterances correspond to the same word or not. We validate the proposed methodology through investigations on spoken language and sign language. In the case of spoken language, we demonstrate that the proposed methodology can lead up to discovery of phone set and development of phonetic dictionary. In the case of sign language, we demonstrate how hand movement information can be effectively modeled for sign language processing and synthesized back to gain insight about the derived subunits.

Towards Multilingual Sign Language Recognition

“…The difficulty in using these methods is that it is not clear if the derived subunits are signerindependent and are language independent akin to phonemes in spoken language (which can be considered as speaker and language independent). More recently, a HMM-based approach was developed [15], where signer-independent hand movement subunits are derived based on light supervision. Through a preliminary crosslingual study it was demonstrated that the subunits derived could be shared across languages.…”

“…The hand movement subunits extraction was done according to the method presented in [15]. Briefly, the 3D skeleton position and velocity of both hands according to three different coordinate centers (head, shoulder and hip center) were used as feature observation (resulting a vector of size 36).…”

“…Similar to the earlier work [15], we used the DeepHand net which is trained on one-million hands dataset [13] for hand shape posterior estimation. The one-million hands is a composition of three different sign languages, namely Danish sign language, New Zealand sign language and German sign language.…”

“…In that framework, while modeling hand shape information with DeepHand [13], it was found that hand movement information can be modeled as discrete units using HMMs. In a more recent work, it was found that such discrete unit representation of hand movements obtained using HMMs, also referred to as hand movement subunits, tend to exhibit language independence [15]. The present paper builds upon these two recent works to investigate a multilingual sign language recognition approach, where resources from multiple sign languages are shared to model both hand movement and hand shape information for sign language recognition.…”

A Phonology-based Approach for Isolated Sign Production Assessment in Sign Language

“…The hand movement subunits extraction was inspired by the method presented in [14]. Briefly, a 36 dimensional position and velocity features for both hands was extracted from the 3D skeleton with three different coordinate centers (head, shoulder and hip center).…”