Speech Bandwidth Extension with Wavenet

Abstract

Large-scale mobile communication systems tend to contain legacy transmission channels with narrowband bottlenecks, resulting in characteristic 'telephone-quality' audio. While higher quality codecs exist, due to the scale and heterogeneity of the networks, transmitting higher sample rate audio with modern high-quality audio codecs can be difficult in practice. This paper proposes an approach where a communication node can instead extend the bandwidth of a band-limited incoming speech signal that may have been passed through a low-rate codec. To this end, we propose a WaveNet-based model conditioned on a log-mel spectrogram representation of a bandwidth-constrained speech audio signal of 8 kHz and audio with artifacts from GSM full-rate (FR) compression to reconstruct the higher-resolution signal. In our experimental MUSHRA evaluation, we show that a model trained to upsample to 24kHz speech signals from audio passed through the 8kHz GSM-FR codec is able to reconstruct audio only slightly lower in quality to that of the Adaptive Multi-Rate Wideband audio codec (AMR-WB) codec at 16kHz, and closes around half the gap in perceptual quality between the original encoded signal and the original speech sampled at 24kHz. We further show that when the same model is passed 8kHz audio that has not been compressed, is able to again reconstruct audio of slightly better quality than 16kHz AMR-WB, in the same MUSHRA evaluation.

Bandwidth Extension is All You Need

“…Kuleshov et al [22] used a convolutional encoder-decoder network inspired by image super resolution. WaveNet [23] and its variants for BWE [24,25] use dilated convolutions to enable large receptive field while preserving the original resolution. Feng et al [6] used FFTNet [26] which resembles the classical FFT process.…”

Speaker and Phoneme-Aware Speech Bandwidth Extension with Residual Dual-Path Network

“…Learning such a long feature sequence poses a challenge to conventional sequential modeling networks, including recurrent neural network and 1-D convolutional networks. Dilated convolutional layers were proposed to alleviate such problem [13,14]. Despite the increased receptive field over the input signals, they have not captured the utterance-level temporal information.…”

“…Prior studies [5][6][7][8][9][10] are focused on estimating high-frequency magnitude and phase spectra in frequency domain. To overcome the inherent difficulty of phase estimation, time-domain frameworks [11][12][13][14][15] are proposed, that offer competitive voice quality.…”

Real-Time Speech Frequency Bandwidth Extension

“…Early methods were based on the source-filter model of speech production and exploit DNNs to estimate the upper frequency envelope [3]. Inspired by the early success in image superresolution [4], end-to-end audio-based solutions were proposed, based on wave-to-wave UNet [5], WaveNet [6,7], hybrid time/frequency-domain models [8]. All these methods are trained by minimizing a reconstruction loss, typically the The design of SEANet is similar to [12], but we adopt the losses proposed in [13], in which the reconstruction loss is computed in the feature space of the discriminator, at different scales and at different layers.…”