Phonetic Feedback for Speech Enhancement with and Without Parallel Speech Data

Plantinga, Peter; Bagchi, Deblin; Fosler‐Lussier, Eric

doi:10.1109/icassp40776.2020.9054001

Cited by 4 publications

(4 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recently, deep learning (DL) has became a popular and effective machine learning algorithm [32], [33], [34] and has brought significant progress in the SE field [35], [36], [37], [38], [39], [40], [41], [42], [43]. Based on the deep structure, an effective representation of the noisy input signal can be extracted and used to reconstruct a clean signal [44], [45], [46], [47], [48], [49], [50]. Various DL-based model structures, including deep denoising autoencoders [51], [52], fully connected neural networks [53], [54], [55], convolutional neural networks (CNNs) [56], [57], recurrent neural networks (RNNs), and long short-term memory (LSTM) [58], [59], [60], [61], [62], [63], have been used as the core model of an SE system and have been proven to provide better performance than traditional statistical and machine-learning methods.…”

Section: Introductionmentioning

confidence: 99%

Improved Lite Audio-Visual Speech Enhancement

Chuang

Wang

Tsao

2022

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Numerous studies have investigated the effectiveness of audio-visual multimodal learning for speech enhancement (AVSE) tasks, seeking a solution that uses visual data as auxiliary and complementary input to reduce the noise of noisy speech signals. Recently, we proposed a lite audio-visual speech enhancement (LAVSE) algorithm for a car-driving scenario. Compared to conventional AVSE systems, LAVSE requires less online computation and to some extent solves the user privacy problem on facial data. In this study, we extend LAVSE to improve its ability to address three practical issues often encountered in implementing AVSE systems, namely, the additional cost of processing visual data, audio-visual asynchronization, and low-quality visual data. The proposed system is termed improved LAVSE (iLAVSE), which uses a convolutional recurrent neural network architecture as the core AVSE model. We evaluate iLAVSE on the Taiwan Mandarin speech with video dataset. Experimental results confirm that compared to conventional AVSE systems, iLAVSE can effectively overcome the aforementioned three practical issues and can improve enhancement performance. The results also confirm that iLAVSE is suitable for real-world scenarios, where high-quality audio-visual sensors may not always be available.

show abstract

Section: Introductionmentioning

confidence: 99%

Improved Lite Audio-Visual Speech Enhancement

Chuang

Wang

Tsao

2022

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

show abstract

“…As an example, Turian and Henry have demonstrated that spectral measures of distance do not capture pitch differences, which is important for tasks such as tonal language recognition [13]. Another example of the limitations of spectral measures is that they can miss low-energy phonemes due to over-emphasis on energy differences [14].…”

Section: Introductionmentioning

confidence: 99%

“…1. Our approach uses a recognition model pre-trained with phoneme targets and clean input speech to generate a phonetic perceptual loss 1 for improved enhancement training [14]- [16]. This approach preserves modularity, and achieved state-of-the-art recognition scores on the CHiME-2 challenge [17] for systems using the default language model.…”

Section: Introductionmentioning

confidence: 99%

Perceptual Loss with Recognition Model for Single-Channel Enhancement and Robust ASR

Plantinga¹,

Bagchi²,

Fosler‐Lussier³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Single-channel speech enhancement approaches do not always improve automatic recognition rates in the presence of noise, because they can introduce distortions unhelpful for recognition. Following a trend towards end-to-end training of sequential neural network models, several research groups have addressed this problem with joint training of front-end enhancement module with back-end recognition module. While this approach ensures enhancement outputs are helpful for recognition, the enhancement model can overfit to the training data, weakening the recognition model in the presence of unseen noise.To address this, we used a pre-trained acoustic model to generate a perceptual loss that makes speech enchancement more aware of the phonetic properties of the signal. This approach keeps some benefits of joint training, while alleviating the overfitting problem. Experiments on Voicebank + DEMAND dataset for enhancement show that this approach achieves a new state of the art for some objective enhancement scores. In combination with distortion-independent training, our approach gets a WER of 2.80% on the test set, which is more than 20% relative better recognition performance than joint training, and 14% relative better than distortion-independent mask training.

show abstract

“…More recently, deep learning (DL) has became a popular and effective machine learning algorithm [32,33,34] and has brought significant progress in the SE field [35,36,37,38,39,40,41,42,43]. Based on the deep structure, an effective representation of the noisy input signal can be extracted and used to reconstruct a clean signal [44,45,46,47,48,49,50]. Various DL-based model structures, including deep denoising autoencoders [51,52], fully connected neural networks [53,54,55], convolutional neural networks (CNNs) [56,57], recurrent neural networks (RNNs), and long short-term memory (LSTM) [58,59,60,61,62,63], have been used as the core model of an SE system and have been proven to provide better performance than traditional statistical and machine-learning methods.…”

Section: Introductionmentioning

confidence: 99%

Improved Lite Audio-Visual Speech Enhancement

Chuang¹,

Wang²,

Yu³

2020

Preprint

View full text Add to dashboard Cite

Numerous studies have investigated the effectiveness of audio-visual multimodal learning for speech enhancement (AVSE) tasks, seeking a solution that uses visual data as auxiliary and complementary input to reduce the noise of noisy speech signals. Recently, we proposed a lite audio-visual speech enhancement (LAVSE) algorithm. Compared to conventional AVSE systems, LAVSE requires less online computation and moderately solves the user privacy problem on facial data. In this study, we extend LAVSE to improve its ability to address three practical issues often encountered in implementing AVSE systems, namely, the requirement for additional visual data, audio-visual asynchronization, and lowquality visual data. The proposed system is termed improved LAVSE (iLAVSE), which uses a convolutional recurrent neural network architecture as the core AVSE model. We evaluate iLAVSE on the Taiwan Mandarin speech with video dataset. Experimental results confirm that compared to conventional AVSE systems, iLAVSE can effectively overcome the aforementioned three practical issues and can improve enhancement performance. The results also confirm that iLAVSE is suitable for real-world scenarios, where high-quality audio-visual sensors may not always be available.

show abstract

Phonetic Feedback for Speech Enhancement with and Without Parallel Speech Data

Cited by 4 publications

References 19 publications

Improved Lite Audio-Visual Speech Enhancement

Improved Lite Audio-Visual Speech Enhancement

Perceptual Loss with Recognition Model for Single-Channel Enhancement and Robust ASR

Improved Lite Audio-Visual Speech Enhancement

Contact Info

Product

Resources

About