Visionary Speech: Looking Ahead to Practical Speechreading Systems

Hennecke, Marcus E.; Stork, David G.; Prasad, K. Venkatesh

doi:10.1007/978-3-662-13015-5_25

Cited by 73 publications

(35 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Depending on the stage at which the audio and visual streams are fused, one can generally classify these approaches into three main categories, namely early, intermediate, and late integration techniques [46], ranging from methods that enforce strict stream alignment to methods that process each stream independently. Intermediate integration techniques, which allow moderate asynchrony between the modalities, are perhaps best suited for modeling audiovisual speech.…”

Section: Synchronous and Asynchronous Integration Modelsmentioning

confidence: 99%

Adaptive Multimodal Fusion by Uncertainty Compensation With Application to Audiovisual Speech Recognition

Papandreou

Katsamanis

Pitsikalis

et al. 2009

IEEE Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Abstract-While the accuracy of feature measurements heavily depends on changing environmental conditions, studying the consequences of this fact in pattern recognition tasks has received relatively little attention to date. In this paper, we explicitly take feature measurement uncertainty into account and show how multimodal classification and learning rules should be adjusted to compensate for its effects. Our approach is particularly fruitful in multimodal fusion scenarios, such as audiovisual speech recognition, where multiple streams of complementary time-evolving features are integrated. For such applications, provided that the measurement noise uncertainty for each feature stream can be estimated, the proposed framework leads to highly adaptive multimodal fusion rules which are easy and efficient to implement. Our technique is widely applicable and can be transparently integrated with either synchronous or asynchronous multimodal sequence integration architectures. We further show that multimodal fusion methods relying on stream weights can naturally emerge from our scheme under certain assumptions; this connection provides valuable insights into the adaptivity properties of our multimodal uncertainty compensation approach. We show how these ideas can be practically applied for audiovisual speech recognition. In this context, we propose improved techniques for person-independent visual feature extraction and uncertainty estimation with active appearance models, and also discuss how enhanced audio features along with their uncertainty estimates can be effectively computed. We demonstrate the efficacy of our approach in audiovisual speech recognition experiments on the CUAVE database using either synchronous or asynchronous multimodal integration models.Index Terms-Active appearance models (AAMs), audiovisual automatic speech recognition (AV-ASR), multimodal fusion, uncertainty compensation.

show abstract

Section: Synchronous and Asynchronous Integration Modelsmentioning

confidence: 99%

Adaptive Multimodal Fusion by Uncertainty Compensation With Application to Audiovisual Speech Recognition

Papandreou

Katsamanis

Pitsikalis

et al. 2009

IEEE Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

show abstract

“…For instance, in audio-visual speech recognition, the issue of early versus late combination is well known (e.g., Hennecke, Stork, and Venkatesh Prasad 1996). Early combination fuses different sources of information into a single, large feature vector which forms the input to a single recognition system.…”

Section: Conclusion and Discussionmentioning

confidence: 99%

Obituary: W A (Bill) Ainsworth

2003

Computer Speech & Language

View full text Add to dashboard Cite

A growing body of recent work documents the potential benefits of sub-band processing over wideband processing in automatic speech recognition and, less usually, speaker recognition. It is often found that the subband approach delivers performance improvements (especially in the presence of noise), but not always so. This raises the question of precisely when and how sub-band processing might be advantageous, which is difficult to answer because there is as yet only a rudimentary theoretical framework guiding this work. We describe a simple sub-band speaker recognition system designed to facilitate experimentation aimed at increasing understanding of the approach. This splits the time-domain speech signal into 16 sub-bands using a bank of second-order filters spaced on the psychophysical mel scale. Each sub-band has its own separate cepstral-based recognition system, the outputs of which are combined using the sum rule to produce a final decision. We find that sub-band processing leads to worthwhile reductions in both the verification and identification error rates relative to the wideband system, decreasing the identification error rate from 3.33% to 0.56% and equal error rate for verification by approximately 50% for clean speech. The hypothesis is advanced that, unlike the wideband system, sub-band processing effectively constrains the free parameters of the speaker models to be more uniformly deployed across frequency: as such, it offers a practical solution to the bias/variance dilemma of data modeling. Much remains to be done to explore fully the new paradigm of sub-band processing. Accordingly, several avenues for future work are identified. In particular, we aim to explore the hypothesis of a practical solution to the bias/variance dilemma in more depth.

show abstract

“…On theoretical grounds and on the necessity of maintaining temporal relationships and low-level dependencies, many argue for feature fusion [26]. On the other hand, comparative empirical studies have demonstrated that decision fusion techniques are performing better than feature fusion methods, having more practical benefits [27].…”

Section: Fusion Of Audio and Visual Featuresmentioning

confidence: 99%

Multimedia content analysis for emotional characterization of music video clips

et al. 2013

View full text Add to dashboard Cite

Nowadays, tags play an important role in the search and retrieval process in multimedia content sharing social networks. As the amount of multimedia contents explosively increases, it is a challenging problem to find a content that will be appealing to the users. Furthermore, the retrieval of multimedia contents, which can match users' current mood or affective state, can be of great interest. One approach to indexing multimedia contents is to determine the potential affective state, which they can induce in users. In this paper, multimedia content analysis is performed to extract affective audio and visual cues from different music video clips. Furthermore, several fusion techniques are used to combine the information extracted from the audio and video contents of music video clips. We show that using the proposed methodology, a relatively high performance (up to 90%) of affect recognition is obtained.

show abstract

Visionary Speech: Looking Ahead to Practical Speechreading Systems

Cited by 73 publications

References 0 publications

Adaptive Multimodal Fusion by Uncertainty Compensation With Application to Audiovisual Speech Recognition

Adaptive Multimodal Fusion by Uncertainty Compensation With Application to Audiovisual Speech Recognition

Obituary: W A (Bill) Ainsworth

Multimedia content analysis for emotional characterization of music video clips

Contact Info

Product

Resources

About