iMetricGAN: Intelligibility Enhancement for Speech-in-Noise Using Generative Adversarial Network-Based Metric Learning

Abstract

The intelligibility of natural speech is seriously degraded when exposed to adverse noisy environments. In this work, we propose a deep learning-based speech modification method to compensate for the intelligibility loss, with the constraint that the root mean square (RMS) level and duration of the speech signal are maintained before and after modifications. Specifically, we utilize an iMetricGAN approach to optimize the speech intelligibility metrics with generative adversarial networks (GANs). Experimental results show that the proposed iMetricGAN outperforms conventional state-of-the-art algorithms in terms of objective measures, i.e., speech intelligibility in bits (SIIB) and extended short-time objective intelligibility (ESTOI), under a Cafeteria noise condition. In addition, formal listening tests reveal significant intelligibility gains when both noise and reverberation exist.

Multi-Metric Optimization Using Generative Adversarial Networks for Near-End Speech Intelligibility Enhancement

“…Moreover, the signal examplê y, which is pre-enhanced using other reference algorithms (e.g., SSDRC [7] and OptSII [5]), is also fed into D int in the training. As demonstrated in our earlier study [28], learning such additional examples can stabilize the training process and improve performance. Given all the above notations, the loss function of D int is represented as follows:…”

“…We comprehensively evaluate the system's performance under different conditions with unseen noises and reverberations. Our experiments show that the improved system significantly increases the intelligibility and quality of speech over our original system [28] with far less parameters. Moreover, it also outperforms the state-of-the-art SSDRC baseline in both objective and subjective evaluations.…”

“…Thus, the number of nodes are accordingly set to 2. Similar to our previous study [28], we apply spectral normalization with 1-Lipschitz continuity [43] to all the layers used in D qua to stabilize the training process.…”

“…Inspired by progresses in black-box function optimization [26], [27], we previously proposed a generative adversarial network (GAN)-based system [28] for near-end intelligibility enhancement. The system was composed of a generator that enhances the intelligibility of input speech and a discriminator that acts as a learned surrogate of evaluation metrics to guide the training scheme of the generator.…”

“…In this paper, we propose a causal and light-weight system as an extension to our earlier system [28]. We substitute the original bidirectional long-short term memory (BLSTM) with causal convolution.…”