Multimodal Corpus Design for Audio-Visual Speech Recognition in Vehicle Cabin

Abstract

This paper introduces a new methodology aimed at comfort for the driver in-the-wild multimodal corpus creation for audiovisual speech recognition in driver monitoring systems. The presented methodology is universal and can be used for corpus recording for different languages. We present an analysis of speech recognition systems and voice interfaces for driver monitoring systems based on the analysis of both audio and video data. Multimodal speech recognition allows using audio data when video data are useless (e.g. at nighttime), as well as applying video data in acoustically noisy conditions (e.g., at highways). Our methodology identifies the main steps and requirements for multimodal corpus designing, including the development of a new framework for audiovisual corpus creation. We identify the main research questions related to the speech corpus creation task and discuss them in detail in this paper. We also consider some main cases of usage that require speech recognition in a vehicle cabin for interaction with a driver monitoring system. We also consider other important use cases when the system detects dangerous states of driver's drowsiness and starts a question-answer game to prevent dangerous situations. At the end based on the proposed methodology, we developed a mobile application that allows us to record a corpus for the Russian language. We created RUSAVIC corpus using the developed mobile application that at the moment a unique audiovisual corpus for the Russian language that is recorded in-the-wild condition. INDEX TERMS Driver monitoring, automatic speech recognition, multimodal corpus, human-computer interaction.

RETRACTED ARTICLE: Audio-Visual Automatic Speech Recognition Towards Education for Disabilities

“…Chen et al (2022) have proposed an improved K-singular value decomposition and atom optimization techniques to reduce image noise. The authors have developed audiovisual speech recognition scheme for driver monitoring system (Kashevnik et al, 2021). Multimodal speech recognition allows for the use of audio data when video data is unavailable at night, as well as the use of video data in acoustically loud environments at highways.…”

Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices

“…Another modern trend that appeared recently is the web-based datasets: datasets collected from open sources such as YouTube or TV shows [ 59 ]. The most well-known of them are: LRW dataset [ 20 ], LRS2-BBC, LRS3-TED datasets [ 63 ], VGG-SOUND [ 64 ], Modality dataset [ 65 ], and vehicle AVSR [ 66 ]. A survey [ 67 ] regarding this topic provides essential knowledge of the current state-of-the-art situation.…”

Development of Visual and Audio Speech Recognition Systems Using Deep Neural Networks

“…Almost no publicly-accessible audio-visual Russian speech datasets are available and suitable for NN training. The most recent one was introduced in the work [1] and was specifically designed for the task of robust speech recognition in acoustically-noisy car environment.…”

“…In this paper, we present lip-reading pipeline and acoustic speech recognition pipeline with the use of deep 3D CNNs. We trained and evaluated our models using RUSAVIC [1] dataset on a limited vocabulary of 50 phrases. To handle the over-fitting problem due to the increased number of parameters from the 3D kernels, we applied the idea from [2] to inflate the pre-trained weights of the several stateof-the-art models, such as MobileNetV2 [3], DenseNet121 [4], NASNetMobile [5], etc.…”