Audiovisual automatic speech recognition

Potamianos, Gerasimos; Neti, C.; Luettin, Juergen; Matthews, Iain

doi:10.1017/cbo9780511843891.011

Cited by 81 publications

(123 citation statements)

References 0 publications

Supporting

Mentioning

119

Contrasting

Unclassified

Order By: Relevance

“…First of all, a region of interest (ROI) around the mouth, which contains the largest amount of information about the utterance, has to be extracted [23]. This can be done by hand or with the help of a face tracker.…”

Section: Related Workmentioning

confidence: 99%

“…In general three types of features are used: texture-based features, shapebased features, or a combination of both [23,5]. Texture-based features exploit the pixel values in a ROI -usually closely around the mouth or including the jaws [23].…”

Section: Related Workmentioning

confidence: 99%

“…The GMMs model the acoustics, i.e. the phonemes, or visemes in visual speech, while the states of an HMM model the time evolution within a phoneme and the overall evolution within and between words [23].…”

Section: Related Workmentioning

confidence: 99%

“…The research interest in this topic arises from several factors; first, visual speech recognition can be used in automatic audio-visual speech recognition together with the audio data [23]. It has been shown that highly noisy audio data can thus be supported and higher recognition rates can be achieved in such scenarios [23]. In these cases, the supporting visual information helps similarly to its contribution in human-human interaction, where a listener's concentration on the lip movement increases in noisy environments.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Visual Speech Recognition Using PCA Networks and LSTMs in a Tandem GMM-HMM System

Zimmermann

Ghazi

Ekenel

et al. 2017

Computer Vision – ACCV 2016 Workshops

View full text Add to dashboard Cite

Abstract. Automatic visual speech recognition is an interesting problem in pattern recognition especially when audio data is noisy or not readily available. It is also a very challenging task mainly because of the lower amount of information in the visual articulations compared to the audible utterance. In this work, principle component analysis is applied to the image patches -extracted from the video data -to learn the weights of a two-stage convolutional network. Block histograms are then extracted as the unsupervised learning features. These features are employed to learn a recurrent neural network with a set of long short-term memory cells to obtain spatiotemporal features. Finally, the obtained features are used in a tandem GMM-HMM system for speech recognition. Our results show that the proposed method has outperformed the baseline techniques applied to the OuluVS2 audiovisual database for phrase recognition with the frontal view cross-validation and testing sentence correctness reaching 79% and 73%, respectively, as compared to the baseline of 74% on cross-validation.The final publication is available at Springer via http://dx

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Visual Speech Recognition Using PCA Networks and LSTMs in a Tandem GMM-HMM System

Zimmermann

Ghazi

Ekenel

et al. 2017

Computer Vision – ACCV 2016 Workshops

View full text Add to dashboard Cite

show abstract

“…In [8], the results of visual ASR experiments involving the use of the IBM ViaVoice database were presented in their comparison of four types of visual features, namely discrete cosine transform (DCT) [9], discrete wavelet transform (DWT) [10], principal components analysis (PCA) [11], and active appearance models (AAM) [12]. A solution using hidden Markov models (HMMs) [13] as the classifier found that DCT based visual features were the most promising for the recognition task.…”

Section: Introductionmentioning

confidence: 99%

Geometrical-based lip-reading using template probabilistic multi-dimension dynamic time warping

Ibrahim

Mulvaney

2015

Journal of Visual Communication and Image Representation

View full text Add to dashboard Cite

Message Digest (MD5) Algorithm and Secure Hash Algorithm (SHA)

Encyclopedia of Multimedia

View full text Add to dashboard Cite

Definition:Medical image segmentation refers to the segmentation of known anatomic structures from medical images.Structures of interest include organs or parts thereof, such as cardiac ventricles or kidneys, abnormalities such as tumors and cysts, as well as other structures such as bones, vessels, brain structures etc. The overall objective of such methods is referred to as computer-aided diagnosis; in other words, they are used for assisting doctors in evaluating medical imagery or in recognizing abnormal findings in a medical image.In contrast to generic segmentation methods, methods used for medical image segmentation are often application-specific; as such, they can make use of prior knowledge for the particular objects of interest and other expected or possible structures in the image. This has led to the development of a wide range of segmentation methods addressing specific problems in medical applications. Some methods proposed in the literature are extensions of methods originally proposed for generic image segmentation. In [1], a modification of the watershed transform is proposed for knee cartilage and gray matter/white matter segmentation in magnetic resonance images (MRI). This introduces prior information in the watershed method via the use of a previous probability calculation for the classes present in the image and via the combination of the watershed transform with atlas registration for the automatic generation of markers.Other methods are more application specific; for example in [2], segmentation tools are developed for use in the study of the function of the brain, i.e. for the classification of brain areas as activating, deactivating, or not activating, using functional magnetic resonance imaging (FMRI) data. The method of [2] performs segmentation based on intensity histogram information, augmented with adaptive spatial regularization using Markov random fields. The latter contributes to improved segmentation as compared to non-spatial mixture models, while not requiring the heuristic fine-tuning that is necessary for non-adaptive spatial regularization previously proposed.Another important appHcation of segmentation tools is in the study of the function of the heart. In [3], a contour detection algorithm based on a radial edge-detection filter is MESSAGE DIGEST (MD5) ALGORITHM AND SECURE HASH ALGORITHM (SHA)Definition: Message Digest and Secure Cash are the standard algorithms to provide data security for multimedia authentication.The MD5 algorithm takes as input a message of arbitrary length and produces as output a 128-bit "fingerprint" or "message digest" of the input message [1]. MD5 is currently a standard, Internet Engineering Task Force (IETF) Request for Comments (RFC) 1321. In 160 256 384 512Each SHA algorithm processes a message in two stages: preprocessing and hash computation. Preprocessing involves padding a message, parsing the padded message into 512-or 1024-bit blocks, and setting initialization values to be used in the hash computation. The hash computation generates a ...

show abstract

Audiovisual automatic speech recognition

Cited by 81 publications

References 0 publications

Visual Speech Recognition Using PCA Networks and LSTMs in a Tandem GMM-HMM System

Visual Speech Recognition Using PCA Networks and LSTMs in a Tandem GMM-HMM System

Geometrical-based lip-reading using template probabilistic multi-dimension dynamic time warping

Message Digest (MD5) Algorithm and Secure Hash Algorithm (SHA)

Contact Info

Product

Resources

About