Language Identification Using Visual Features

Abstract

Abstract-Automatic visual language identification (VLID) is the technology of using information derived from the visual appearance and movement of the speech articulators to identify the language being spoken, without the use of any audio information. This technique for language identification (LID) is useful in situations in which conventional audio processing is ineffective (very noisy environments), or impossible (no audio signal is available). Research in this field is also beneficial in the related field of automatic lip-reading. This paper introduces several methods for visual language identification (VLID). They are based upon audio LID techniques, which exploit language phonology and phonotactics to discriminate languages. We show that VLID is possible in a speaker-dependent mode by discriminating different languages spoken by an individual, and we then extend the technique to speaker-independent operation, taking pains to ensure that discrimination is not due to artefacts, either visual (e.g. skin-tone) or audio (e.g. rate of speaking). Although the low accuracy of visual speech recognition currently limits the performance of VLID, we can obtain an error-rate of < 10% in discriminating between Arabic and English on 19 speakers and using about 30s of visual speech.

Discrimination Between Native and Non-Native Speech Using Visual Features Only

“…The same assumption holds for the test data as well. This behaviour is in conformity with evidence from research in visual-only language identification, according to which performance increases with the length of speech data [22].…”

“…The authors show through a series of experimental scenarios that observers perform much higher than chance level on visual discrimination between English and Spanish, even when the visual speech segments are presented to them in reverse temporal order. Recently, Newman and Cox [22] present an automated approach for both speaker-dependent and speaker-independent visual-only language identification, based on features that capture phonology and phonotactics characteristics of visual speech. They use Active Shape Models and Active Appearance Models (ASM, AAM) [32] for visual feature extraction, which is carried out frame-wise, and then feature vectors are "tokenised" in visually-transcribed phonemes.…”

“…Furthermore, recent findings indicate that human observers can actually perform language identification through the visual modality only [21]. Automated approaches for visual-only language identification have also been proposed (e.g., [22]). Another study shows that visual identification of accent is a feasible task for human observers [23].…”

On the Comparison of Line Spectral Frequencies and Mel-Frequency Cepstral Coefficients Using Feedforward Neural Network for Language Identification

“…A tutorial on LID has been presented in [2] in which syntactic, morphological, and acoustic, phonetic, phonotactic, and prosodic level information have been discussed in details. Around 87 prosodic features has been used for LID system in [3] which provides better recognition performance, while [4] utilizes visual features with error rate less than 10%. In [5], a highly accurate and computationally efficient framework of i-vector presentation is proposed for rapid language identification.…”

Discriminating Native from Non-Native Speech Using Fusion of Visual Cues

“…However, the beneficial role of visual information to speech comprehension has been well documented and experimentally validated [12]. Specifically, automated visual-only approaches have been developed for language identification [8]. Another study shows that visual identification of accent is feasible for human observers [5].…”