Visual Speech Recognition of Korean Words Using Convolutional Neural Network

Abstract

In recent studies, speech recognition performance is greatly improved by using HMM and CNN. HMM is studying statistical modeling of voice to construct an acoustic model and to reduce the error rate by predicting voice through image of mouth region using CNN. In this paper, we propose visual speech recognition (VSR) using lip images. To implement VSR, we repeatedly recorded three subjects speaking 53 words chosen from an emergency medical service vocabulary book. To extract images of consonants, vowels, and final consonants in the recorded video, audio signals were used. The Viola-Jones algorithm was used for lip tracking on the extracted images. The lip tracking images were grouped and then classified using CNNs. To classify the components of a syllable including consonants, vowels, and final consonants, the structure of the CNN used VGG-s and modified LeNet-5, which has more layers. All syllable components were classified, and then the word was found by the Euclidean distance. From this experiment, a classification rate of 72.327% using 318 total testing words was obtained when VGG-s was used. When LeNet-5 applied this classifier for words, however, the classification rate was 22.327%.

OLKAVS: An Open Large-Scale Korean Audio-Visual Speech Dataset

“…Jo et al [16] collected predefined syllables of a single speaker with 7 views. Also, Lee and Park [22] and Lee et al [23] collected predefined word utterances, such as digits and city names, of 56 and 9 speakers, respectively. Unfortunately, the size of all the datasets is too minuscule to support deep learning-driven models; moreover, some are not publicly available.…”

“…1) Korean speech recognition: Not only the size of OLKAVS greatly outplays the previous audio-visual speech datasets [16,22,23], but also comparable to the audio-only Korean speech dataset [25]. Also, our pre-trained audio-visual speech recognition model can be useful when fine-tuned to other languages [10].…”