Deep learning based speaker separation and dereverberation can generalize across different languages to improve intelligibility

Abstract

The practical efficacy of deep learning based speaker separation and/or dereverberation hinges on its ability to generalize to conditions not employed during neural network training. The current study was designed to assess the ability to generalize across extremely different training versus test environments. Training and testing were performed using different languages having no known common ancestry and correspondingly large linguistic differences—English for training and Mandarin for testing. Additional generalizations included untrained speech corpus/recording channel, target-to-interferer energy ratios, reverberation room impulse responses, and test talkers. A deep computational auditory scene analysis algorithm, employing complex time-frequency masking to estimate both magnitude and phase, was used to segregate two concurrent talkers and simultaneously remove large amounts of room reverberation to increase the intelligibility of a target talker. Significant intelligibility improvements were observed for the normal-hearing listeners in every condition. Benefit averaged 43.5% points across conditions and was comparable to that obtained when training and testing were performed both in English. Benefit is projected to be considerably larger for individuals with hearing impairment. It is concluded that a properly designed and trained deep speaker separation/dereverberation network can be capable of generalization across vastly different acoustic environments that include different languages.

Restoring speech intelligibility for hearing aid users with deep learning

“…Non-spatial (single microphone) noise reduction algorithms employed in hearing aids have so far not been able to provide improvements in speech intelligibility 6 , 10 – 15 . A few recent studies have shown that deep learning-based denoising 16 – 18 or separation of multiple competing speakers 19 , 20 can provide improvements in speech intelligibility for cochlear implant users 21 and hearing aid users with severe-profound 22 hearing loss under fixed signal-to-noise (SNR) conditions 16 – 18 . For the majority of hearing aid users, with less severe hearing loss 22 , it is more challenging to provide intelligibility improvements through denoising.…”

Recovering speech intelligibility with deep learning and multiple microphones in noisy-reverberant situations for people using cochlear implants

“…This is an important consideration for deep learning algorithms, which can overfit to training or within-domain data (Pandey and Wang, 2020;Rehr and Gerkmann, 2021). These findings are promising, though further work is required to assess whether different languages (Healy et al, 2021), microphone configurations and/or spatial arrangements between the listener and target speaker would also produce similar results. An advantage of deep-learning approaches is that, by choosing the variability of acoustic scenarios included in the training data, the algorithms can be fine-tuned to specific scenarios or made robust to a wide range of scenarios, albeit with some expected drop in performance over scenario-specific training.…”

A deep learning algorithm for target speaker extraction to improve speech intelligibility in noisy multi-talker environments

“…However, further exploration of different languages, in particular tonal languages, remains to be investigated. Related research showed that such generalization can be achieved by deep learning algorithms (Healy et al, 2021c).…”