Time-Domain Multi-Modal Bone/Air Conducted Speech Enhancement

Abstract

Integrating modalities, such as video signals with speech, has been shown to provide a standard quality and intelligibility for speech enhancement (SE). However, video clips usually contain large amounts of data and pose a high cost in terms of computational resources, which may complicate the respective SE. By contrast, a bone-conducted speech signal has a moderate data size while it manifests speech-phoneme structures, and thus complements its air-conducted counterpart, benefiting the enhancement. In this study, we propose a novel multi-modal SE structure that leverages bone-and air-conducted signals. In addition, we examine two strategies, early fusion and late fusion (LF), to process the two types of speech signals, and adopt a deep learning-based fully convolutional network to conduct the enhancement. The experiment results indicate that this newly presented multimodal structure significantly outperforms the single-source SE counterparts (with a bone-or air-conducted signal only) in various speech evaluation metrics. In addition, the adoption of an LF strategy other than an EF in this novel SE multi-modal structure achieves better results.

EMA2S: An End-to-End Multimodal Articulatory-to-Speech System

“…Multimodal learning [25] aims to learn relating information from multiple modalities and fill the missing modality given the observed ones. Numerous research has investigated the effectiveness of incorporating different features into speechrelated systems, including text [27]- [30], videos [31]- [33], bone-conducted microphone signals [34], electropalatography [35], and articulatory movements [36]- [38].…”

EMA2S: An End-to-End Multimodal Articulatory-to-Speech System

“…Multimodal learning [25] aims to learn relating information from multiple modalities and fill the missing modality given the observed ones. Numerous research has investigated the effectiveness of incorporating different features into speechrelated systems, including text [27]- [30], videos [31]- [33], bone-conducted microphone signals [34], electropalatography [35], and articulatory movements [36]- [38].…”

Improved Lite Audio-Visual Speech Enhancement

“…Another well-known advantage of DL models is that they can flexibly fuse data from different domains [64], [65]. Recently, researchers have tried to incorporate text [66], bone-conducted signals [67], and visual cues [68], [69], [70], [71], [72], [73] into speech applications as auxiliary and complementary information to achieve better performance. Among them, visual cues are the most common and intuitive because most devices can capture audio and visual data simultaneously.…”