An HMM-based speech-to-video synthesizer

Abstract

Emerging broadband communication systems promise a future of multimedia telephony, e.g. the addition of visual information to telephone conversations. It is useful to consider the problem of generating the critical information useful for speechreading, based on existing narrowband communications systems used for speech. This paper focuses on the problem of synthesizing visual articulatory movements given the acoustic speech signal. In this application, the acoustic speech signal is analyzed and the corresponding articulatory movements are synthesized for speechreading. This paper describes a hidden Markov model (HMM)-based visual speech synthesizer. The key elements in the application of HMMs to this problem are the decomposition of the overall modeling task into key stages and the judicious determination of the observation vector's components for each stage. The main contribution of this paper is a novel correlation HMM model that is able to integrate independently trained acoustic and visual HMMs for speech-to-visual synthesis. This model allows increased flexibility in choosing model topologies for the acoustic and visual HMMs. Moreover the propose model reduces the amount of training data compared to early integration modeling techniques. Results from objective experiments analysis show that the propose approach can reduce time alignment errors by 37.4% compared to conventional temporal scaling method. Furthermore, subjective results indicated that the purpose model can increase speech understanding.

Lip Assistant: Visualize Speech for Hearing Impaired People in Multimedia Services

“…The speech signal was processed in frames of 25ms with a 15ms overlapping (rate=lOOHz). The speech frames were pre-emphasized with an FIR filter (H(z) 1 1-az-1,a a 0.97), and weighted by a Hamming window to avoid spectral distortions. After pre-processing, we exacted Mel Frequency Cepstral Coefficients (MFCCs) as the acoustic features.…”

“…Although hearing impaired people are the ideal subjects for measuring the perceptual quality of the synthesized visual speech, it is quite difficult to get an impartial evaluation because we have to find individuals with equal levels of lipreading proficiency [1]. Therefore, we used a substitution evaluation approach, in which a group of 8 subjects with normal hearing were recruited.…”

“…It is estimated that about 26.1 million people in the United States have hard hearing problem [1]. This figure is much larger in China.…”

Partial linear regression for speech-driven talking head application

“…The conversion problem is treated as one of finding the best approximation from given sets of training data. These approaches were briefly discussed in Chen and Rao [10], including vector quantization [25], Hidden Markov Models (HMM) [2,3,9,13,31], and neural networks [19,20,30]. However, the speech-driven systems were generally made to be user-independent for satisfactory average performance, which means a decrease in accuracy rate for a specific user.…”

Mapping from Speech to Images Using Continuous State Space Models

“…Once the sequence of mouth movements has been determined, the mouth is mapped back to a background face of the speaker. Other authors have proposed methods based on modeling of phonemes by correlational HMM's [5] or neural networks [6].…”