Speaker Adaptive Audio-Visual Fusion for the Open-Vocabulary Section of AVICAR

Abstract

This experimental study establishes the first audiovisual speech recognition baseline for the TIMIT sentence portion of the AVICAR dataset, a dataset recorded in a real, noisy car environment. We use an automatic speech recognizer trained on a larger dataset to generate an audio-only recognition baseline for AVICAR. We utilize the forced alignment of the audio modality of AVICAR to get training targets for the convolutional neural network based visual front end. Based on our observation that there is a great amount of variation between visual features of different speakers, we apply feature space maximum likelihood linear regression (fMMLR) based speaker adaptation to the visual features. We find that the quality of fMLLR is sensitive to the quality of the alignment probabilities used to compute it; experimental tests compare the quality of fMLLR trained using audiovisual versus audio-only alignment probabilities. We report the first audiovisual results for TIMIT subset of AVICAR and show that the word error rate of the proposed audiovisual system is significantly better than that of the audio-only system.

MobiLipNet: Resource-Efficient Deep Learning Based Lipreading

“…The field of visual speech recognition (VSR), or lipreading, has witnessed dramatic breakthroughs recently, primarily due to the paradigm shift from hand-crafted features to deep learning based models [1][2][3][4][5][6][7][8], coupled with the public release of large suitable corpora in a variety of environments [9][10][11][12][13][14][15], as also reviewed in [16,17]. Such models however, while reducing recognition errors compared to previous approaches, are not as efficient to compute and store.…”