Real-Time Speech Frequency Bandwidth Extension

Liu, Yunpeng; Tagliasacchi, Marco; Rybakov, Oleg; Ungureanu, Victor; Roblek, Dominik

doi:10.1109/icassp39728.2021.9413439

Cited by 22 publications

(22 citation statements)

References 15 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While deep learning based audio super resolution methods have been proven effective, deploying such solutions to a resource-limited embedded system has not been fully investigated. Similar to our proposal, several super resolution deep learning methods [25,28,32] have proven the feasibility of applying the super resolution method on a smartphone. Other state-of-the-art speech super resolution models require considerable computation resources and cause significant latency, which is not suitable for edge device deployment.…”

Section: Audio Super Resolution Techniquesmentioning

confidence: 67%

See 1 more Smart Citation

Enabling Real-time On-chip Audio Super Resolution for Bone Conduction Microphones

Li¹,

Wang²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Voice communication using the air conduction microphone in noisy environments suffers from the degradation of speech audibility. Bone conduction microphones (BCM) are robust against ambient noises but suffer from limited effective bandwidth due to their sensing mechanism. Although existing audio super resolution algorithms can recover the high frequency loss to achieve high-fidelity audio, they require considerably more computational resources than available in low-power hearable devices. This paper proposes the first-ever real-time on-chip speech audio super resolution system for BCM. To accomplish this, we built and compared a series of lightweight audio super resolution deep learning models. Among all these models, ATS-UNet is the most cost-efficient because the proposed novel Audio Temporal Shift Module (ATSM) reduces the network's dimensionality while maintaining sufficient temporal features from speech audios. Then we quantized and deployed the ATS-UNet to low-end ARM micro-controller units for real-time embedded prototype. Evaluation results show that our system achieved real-time inference speed on Cortex-M7 and higher quality than the baseline audio super resolution method. Finally, we conducted a user study with ten experts and ten amateur listeners to evaluate our method's effectiveness to human ears. Both groups perceived a significantly higher speech quality with our method when compared to the solutions with the original BCM or air conduction microphone with cutting-edge noise reduction algorithms. CCS Concepts: • Computing methodologies → Artificial intelligence; • Computer systems organization → Realtime systems; System on a chip; Embedded software.

show abstract

Section: Audio Super Resolution Techniquesmentioning

confidence: 67%

“…Adversarial learning is another popular training technique. In this technique, a discriminator that works either in the time domain [15,19,28] or frequency domain [11,22,27] guides the generator to predict more realistic high-resolution audio from low-resolution inputs.…”

Section: Audio Super Resolution Techniquesmentioning

confidence: 99%

Enabling Real-time On-chip Audio Super Resolution for Bone Conduction Microphones

Li¹,

Wang²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The model architecture is based on the decoder described in [27,28], which is a real-time streaming-capable version of MelGAN [29]. Its structure with parameters is shown in Table 1.…”

Section: Neural Network Based Spectrogram Inversionmentioning

confidence: 99%

Real time spectrogram inversion on mobile phone

Rybakov¹,

Tagliasacchi²,

Liu³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

With the growth of computing power on mobile phones and privacy concerns over user's data, on-device real time speech processing has become an important research topic. In this paper, we focus on methods for real time spectrogram inversion, where an algorithm receives a portion of the input signal (e.g., one frame) and processes it incrementally, i.e., operating in streaming mode. We present a real time Griffin Lim(GL) algorithm using a sliding window approach in STFT domain. The proposed algorithm is 2.4x faster than real time on the ARM CPU of a Pixel4. In addition we explore a neural vocoder operating in streaming mode and demonstrate the impact of looking ahead on perceptual quality. As little as one hop size (12.5ms) of lookahead is able to significantly improve perceptual quality in comparison to a causal model. We compare GL with the neural vocoder and show different trade-offs in terms of perceptual quality, on-device latency, algorithmic delay, memory footprint and noise sensitivity. For fair quality assessment of the GL approach, we use input log magnitude spectrogram without mel transformation. We evaluate presented real time spectrogram inversion approaches on clean, noisy and atypical speech.

show abstract

“…Although only a few studies have applied GAN models to bandwidth extension of music signals [10], [39], many recent works have applied them for speech [14], [13], [40]. Eskimez et al [40] proposed one of the earliest works using an adversarial approach for speech super-resolution.…”

Section: B Gans For Audio Bandwidth Extensionmentioning

confidence: 99%

“…During recent years, many works have used modern deep learning technologies for bandwidth extension, but usually with the final goal of increasing the sampling rate of modern digital audio signals. Only a few exceptions are relevant to music signal processing [8], [9], [10], whereas most of these studies focus on processing speech [11], [12], [13], [14], [15].…”

Section: Introductionmentioning

confidence: 99%

BEHM-GAN: Bandwidth Extension of Historical Music using Generative Adversarial Networks

Moliner¹,

Välimäki²

2022

Preprint

View full text Add to dashboard Cite

Audio bandwidth extension aims to expand the spectrum of narrow-band audio signals. Although this topic has been broadly studied during recent years, the particular problem of extending the bandwidth of historical music recordings remains an open challenge. This paper proposes BEHM-GAN, a model based on generative adversarial networks, as a practical solution to this problem. The proposed method works with the complex spectrogram representation of audio and, thanks to a dedicated regularization strategy, can effectively extend the bandwidth of out-of-distribution real historical recordings. The BEHM-GAN is designed to be applied as a second step after denoising the recording to suppress any additive disturbances, such as clicks and background noise. We train and evaluate the method using solo piano classical music. The proposed method outperforms the compared baselines in both objective and subjective experiments. The results of a formal blind listening test show that BEHM-GAN significantly increases the perceptual sound quality in early-20th-century gramophone recordings. For several items, there is a substantial improvement in the mean opinion score after enhancing historical recordings with the proposed bandwidthextension algorithm. This study represents a relevant step toward data-driven music restoration in real-world scenarios.

show abstract

Real-Time Speech Frequency Bandwidth Extension

Cited by 22 publications

References 15 publications

Enabling Real-time On-chip Audio Super Resolution for Bone Conduction Microphones

Enabling Real-time On-chip Audio Super Resolution for Bone Conduction Microphones

Real time spectrogram inversion on mobile phone

BEHM-GAN: Bandwidth Extension of Historical Music using Generative Adversarial Networks

Contact Info

Product

Resources

About