Integration of acoustic and visual speech signals using neural networks

Abstract

A UTOMATIC SPEECH RECOGNITION SYSTEMSrely almost exclusively on the acoustic speech signal and, consequently, these systems often perform poorly in noisy environments [ I ] . Attempts to clean up the acoustic input have had limited success [2]. Another approach is to use other sources of speech information, such as visual speech signals. performance of the system was degraded by this eqrly encoding.The need for early categorization of speech signals can be traced to the computational limitations of currently available hardware. On a digital computer, the inherently analog visual speech signals must first be converted to digital format. Next, a significant amount of preprocessing and encoding must be performed before these signals can be compared to a stored set of patterns. Finally. the symbolic descriptions of the segmented visual signal stream are combined with the auditory symbol stream, using rules that require a significant amount of programming. The von Neumann architecture requires that all of these steps be performed sequentially. We propose an alternative method for processing visual speech signals, based on analog computation in a distributed network architecture. By using many interconnected processors working in parallel, large amounts ofdata can be handled concurrently. In addition to speeding up the computation, this approach does not require segmentation in the early stages of processing; rather, analog signals from the visual and auditory pathways would flow through networks in real time and would be combined directly in the final analog Very Large-scale Integration (VLSI) implementation.Results are presented from a series of experiments that use neural networks to process the visual speech signals of a male talker. In these preliminary cxperiments, the results are limited to static images of vowels. We demonstrate that these networks are able to extract speech information from the visual images, and that this information can be used to improve automatic vowel recognition. The first section of this article reviews the structure of speech, and its corresponding acoustic and visual signals. The next section describes the specific data that was used in our experiments along with the network architectures and algorithms. In the final section, we present the results of integrating the visual and auditory signals for vowel recognition in the presence of acoustic noise.

Survey of the State of the Art in Human Language Technology

“…Regardless of the signal-to-noise ratio, most systems perform better using both acoustical and optical sources of information than when using only one source of information (Bregler, Omohundro, et al, 1994;Bregler, Hild, et al, 1993;Mak & Allen, 1994;Petajan, 1984;Petajan, Bischoff, et al, 1988;Silsbee, 1994;Silsbee, 1993;Smith, 1989;Stork, Wolff, et al, 1992;Yuhas, Goldstein, et al, 1989). At a signal-to-noise ratio of zero with a 500-word task Silsbee (1993) achieves word accuracy recognition rates of 38%, 22%, and 58% respectively, using acoustical information, optical information, and both sources of information.…”

“…This comparator may consist of a set of rules (e.g., if the top two phones from the acoustic recognizer is /t/ or /p/, then choose the one that has a higher ranking from the optical recognizer) (Petajan, Bischoff, et al, 1988) or a fuzzy logic integrator (e.g., provides linear weights associated with the acoustically and optically recognized phones) (Silsbee, 1993;Silsbee, 1994). The second approach performs recognition using a vector that includes both acoustical and optical information, such systems typically use neural networks to combine the optical information with the acoustic to improve the signal-to-noise ratio before phonemic recognition (Yuhas, Goldstein, et al, 1989;Bregler, Omohundro, et al, 1994;Bregler, Hild, et al, 1993;Stork, Wolff, et al, 1992;Silsbee, 1994).…”

Nonlinear manifold learning for visual speech recognition

“…Our bimodal database consists of subjects of var- [10,20,19,5,16,12]. Most approaches attempt to show that com puter lip reading is able to improve speech recog nition, especially in noisy environments.…”

Evolutionary Speech Recognition

“…Equivalently for a neural network, the number of units and the number of layers would be adjusted, as well as the weights' values. The third group is increasingly addressed and corresponds to the combination of at least two sources of information to improve the robustness (Yuhas et al 1989) (McGurk and MacDonald 1976). For example, the speaker's lips movements and their corresponding acoustic signals are processed and integrated to improve the robustness of the speech recognition systems.…”