Characteristics of the use of coupled hidden Markov models for audio-visual polish speech recognition

Abstract

Abstract. This paper focuses on combining audio-visual signals for Polish speech recognition in conditions of the highly disturbed audio speech signal. Recognition of audio-visual speech was based on combined hidden Markov models (CHMM). The described methods were developed for a single isolated command, nevertheless their effectiveness indicated that they would also work similarly in continuous audiovisual speech recognition. The problem of a visual speech analysis is very difficult and computationally demanding, mostly because of an extreme amount of data that needs to be processed. Therefore, the method of audio-video speech recognition is used only while the audiospeech signal is exposed to a considerable level of distortion. There are proposed the authors’ own methods of the lip edges detection and a visual characteristic extraction in this paper. Moreover, the method of fusing speech characteristics for an audio-video signal was proposed and tested. A significant increase of recognition effectiveness and processing speed were noted during tests - for properly selected CHMM parameters and an adequate codebook size, besides the use of the appropriate fusion of audio-visual characteristics. The experimental results were very promising and close to those achieved by leading scientists in the field of audio-visual speech recognition.

A Method of Speech Coding for Speech Recognition Using a Convolutional Neural Network

“…To encode the sounds using the RGB image, the MFCC coefficients [23,24] for the R component were applied, the time characteristic was used for the G component, and the signal source was used for the B component. Convolution of the three components R, G and B is performed in parallel.…”

The use of hidden Markov models to verify the identity based on facial asymmetry

“…It is based on searching face elements which have significant value variations in the level of pixel brightness in small areas (so-called gradient method) [23][24][25]. There may be observed significant value variations in the eye area where the white of the eye is similar to the maximum white colour and the pupil to the maximum black colour, although the iris may contain also white light reflections.…”

Using Facial Asymmetry Properties and Hidden Markov Models for Biometric Authentication in Security Systems

“…In case of eyes, the localization takes place by the checked method, based on searching elements on a face which have significant value variation in the level of pixel brightness in small areas (so called gradient method) [11,12]. Significant value variation can be observed in eye area where white of the eye is similar to maximum white colour and the pupil to maximum black colour, although the iris can contain also white light reflections.…”