Speechreading using shape and intensity information

Luettin, Juergen; Thacker, Neil A.; Beet, S.W.

doi:10.1109/icslp.1996.607024

Cited by 52 publications

(32 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Various feature sets have been tested including DCT coefficients [26], active shape models [27] (ASMs), active appearance models (AAMs) and sieve features [22]. The previous section has described papers in which the recognition performance of some of these features has been studied, and the current finding is that AAM features give the best recognition performance overall [24], despite their poor performance in [6].…”

Section: Active Appearance Modelsmentioning

confidence: 99%

Language Identification Using Visual Features

Newman

Cox

2012

IEEE Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Abstract-Automatic visual language identification (VLID) is the technology of using information derived from the visual appearance and movement of the speech articulators to identify the language being spoken, without the use of any audio information. This technique for language identification (LID) is useful in situations in which conventional audio processing is ineffective (very noisy environments), or impossible (no audio signal is available). Research in this field is also beneficial in the related field of automatic lip-reading. This paper introduces several methods for visual language identification (VLID). They are based upon audio LID techniques, which exploit language phonology and phonotactics to discriminate languages. We show that VLID is possible in a speaker-dependent mode by discriminating different languages spoken by an individual, and we then extend the technique to speaker-independent operation, taking pains to ensure that discrimination is not due to artefacts, either visual (e.g. skin-tone) or audio (e.g. rate of speaking). Although the low accuracy of visual speech recognition currently limits the performance of VLID, we can obtain an error-rate of < 10% in discriminating between Arabic and English on 19 speakers and using about 30s of visual speech.

show abstract

Section: Active Appearance Modelsmentioning

confidence: 99%

Language Identification Using Visual Features

Newman

Cox

2012

IEEE Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

show abstract

“…Within this category belong geometric type features, such as mouth height, width, and area [19], [22], [26], [28], [29], [32][33][34][35], [49][50][51][52], Fourier and image moment descriptors of the lip contours [28], [53], statistical models of shape, such as active shape models [48], [54], or other parameters of lip-tracking models [44], [55][56][57]. Finally, features from both categories can be concatenated into a joint shape and appearance vector [27], [44], [58], [59], or a joint statistical model can be learned on such vectors, as is the case of the active appearance model [60], used for speechreading in [48].…”

Section: The Visual Front Endmentioning

confidence: 99%

“…1). The first scenario is primarily useful in benchmarking the performance of visual feature extraction algorithms, with visual-only ASR results typically reported on small-vocabulary tasks [24], [25], [28][29][30][31], [36], [40][41][42][43], [46], [59], [66], [78], [84][85][86][87][88][89][90][91][92]. Visual speech modeling is required in this process, its two central aspects being the choice of speech classes, that are assumed to generate the observed features, and the statistical modeling of this generation process.…”

Section: Visual Speech Modeling For Asrmentioning

confidence: 99%

“…Then, we seek MLLR adapted HMM parameters (see also (2) Closed form solutions for the unknown matrices exist, if the HMM covariances are diagonal [129]. Tulips1 I-4 digits 12 US [24], [46], [59], [78], [84], [85] M2VTS I-digits 37 FR [27], [114], [134] UIUC C-digits 100 US [32], [86] CUAVE I/C-digits 36 US [28], [87], [135] IBM C-digits 50 US [40] U.Karlsruhe C-letters 6 D [36], [41], [42], [88] U.LeMans I-letters 2 FR [34], [35], [93], [100] U.Sheffield I-letters 10 UK [25], [89] AT&T C-letters 49 US [31], [43], [90] UT.Austin I-500 words 1 US [95] AMP-CMU I-78 words 10 US [29], [30], [66], [91], [92] ATR I-words 1 JP [38], [123] AV-TIMIT 150-sent.…”

Section: A Mllr Adaptationmentioning

confidence: 99%

See 1 more Smart Citation

Recent advances in the automatic recognition of audiovisual speech

et al. 2003

View full text Add to dashboard Cite

Abstract-Visual speech information from the speaker's mouth region has been successfully shown to improve noise robustness of automatic speech recognizers, thus promising to extend their usability into the human computer interface. In this paper, we review the main components of audio-visual automatic speech recognition and present novel contributions in two main areas: First, the visual front end design, based on a cascade of linear image transforms of an appropriate video region-of-interest, and subsequently, audio-visual speech integration. On the later topic, we discuss new work on feature and decision fusion combination, the modeling of audio-visual speech asynchrony, and incorporating modality reliability estimates to the bimodal recognition process. We also briefly touch upon the issue of audiovisual speaker adaptation. We apply our algorithms to three multi-subject bimodal databases, ranging from small-to largevocabulary recognition tasks, recorded at both visually controlled and challenging environments. Our experiments demonstrate that the visual modality improves automatic speech recognition over all conditions and data considered, however less so for visually challenging environments and large vocabulary tasks.

show abstract

“…The visual features are often derived from the shape of the mouth [9] [4], [2]. Although very popular, these methods rely exclusively on the accurate detection of the lip contours which is often a challenging task under varying illumination conditions or rotations of the face.…”

Section: Related Workmentioning

confidence: 99%

A coupled HMM for audio-visual speech recognition

Nefian¹,

Liang²,

Pi³

et al. 2002

IEEE International Conference on Acoustics Speech and Signal Processing

View full text Add to dashboard Cite

In recent years several speech recognition systems that use visual together with audio information showed significant increase in performance over the standard speech recognition systems. The use of visual features is justified by both the bimodality of the speech generation and by the need of features that are invariant to acoustic noise perturbation. The audio-visual speech recognition system presented in this paper introduces a novel audio-visual fusion technique that uses a coupled hidden Markov model (HMM). The statistical properties of the coupled-HMM allow us to model the state asynchrony of the audio and visual observations sequences while still preserving their natural correlation over time. The experimental results show that the coupled HMM outperforms the multistream HMM in audio visual speech recognition.

show abstract

Speechreading using shape and intensity information

Abstract: We describe a speechreading system that uses both, shape information from the lip contours and intensity information from the mouth area.

Cited by 52 publications

References 10 publications

Language Identification Using Visual Features

Language Identification Using Visual Features

Recent advances in the automatic recognition of audiovisual speech

A coupled HMM for audio-visual speech recognition

Contact Info

Product

Resources

About