Noise-Robust Multimodal Audio-Visual Speech Recognition System for Speech-Based Interaction Applications

Abstract

Speech is a commonly used interaction-recognition technique in edutainment-based systems and is a key technology for smooth educational learning and user–system interaction. However, its application to real environments is limited owing to the various noise disruptions in real environments. In this study, an audio and visual information-based multimode interaction system is proposed that enables virtual aquarium systems that use speech to interact to be robust to ambient noise. For audio-based speech recognition, a list of words recognized by a speech API is expressed as word vectors using a pretrained model. Meanwhile, vision-based speech recognition uses a composite end-to-end deep neural network. Subsequently, the vectors derived from the API and vision are classified after concatenation. The signal-to-noise ratio of the proposed system was determined based on data from four types of noise environments. Furthermore, it was tested for accuracy and efficiency against existing single-mode strategies for extracting visual features and audio speech recognition. Its average recognition rate was 91.42% when only speech was used, and improved by 6.7% to 98.12% when audio and visual information were combined. This method can be helpful in various real-world settings where speech recognition is regularly utilized, such as cafés, museums, music halls, and kiosks.

Multimodal audiovisual speech recognition architecture using a three‐feature multi‐fusion method for noise‐robust systems

“…In addition, when comparing A + V ( L + S ), A , and V , it improved by 2.228% and 2.305%, respectively, and we demonstrated that adding the log‐Mel spectrogram of speech to visual information helped improve performance in noisy environments. Unlike the previous OCSR API [31, 35], which exhibited excellent performance only in certain environments, it exhibited similar performance in all nine noise environments and an evenly stable performance overall.…”

“…Depending on the composition of the auditory and visual information, seven models were evaluated as A, V(L) for visual information that used only lip information, V(L + S) for visual information using lip and log-Mel spectrogram information, and A + V(L) for auditory and lip and log-Mel spectrogram information. The use of only the auditory data (black dotted lines) of the single-modal approach exhibited approximately 3%-4% WER performance in nine noisy environments, and the OCSR API, which performed well only in certain environments, such as indoors or vehicles, was better than the quantitative performance shown in two previous studies [31,35]. Therefore, in the clean environment listed in Table S4, the OCSR API exhibited an accuracy of 3.939 ± 0.313% (Table S5).…”

“…To construct a word table, we referred to the speech command dataset introduced by Google [46, 47] and collected data for Version 3 following a previous study (Table S2) [31, 35]. A total of 40 participants (20 males, 20 females; average age 29.14 years) were recruited based on their familiarity with speech recognition devices as they used speech recognition technology at least five times a day in noise environments.…”

“…The proposed audio recognition module is illustrated in Figure 3. In previous studies [34,35], we used an open cloud-based speech recognition API using Microsoft's Azure Cognitive Services API [19], which had approximately 5%-10% better word recognition rates than Google Assistant and Amazon Transcribe. To mitigate the impact of performance changes over time, a new API that surpasses the current API is provided whenever available.…”

“…Additionally, following the "transition" structure shown in Figure S1A(b) (Table S1), a standard dropout layer was connected, and pixels were randomly dropped to prevent strong correlation in feature maps between successive frames [39]. In addition, the spatial dropout layer connected to the "transition" structure, shown in Figure S2C, was effectively used to extract fine movement features such as lips, teeth, and tongue with strong spatial correlation [31][32][33][34][35][36][37][38][39][40][41][42][43]. Therefore, the proposed dense spatial-temporal CNN network comprises one layer that represents a nonlinear transformation Hl, and the output of the layer can be expressed as x l (3), where x 0 , x 1 , ÁÁÁ, and x (l-1) denote the volume of the 3D feature created in the previous layer and [ÁÁÁ] denotes a concatenation operation.…”