Low latency sound source separation using convolutional recurrent neural networks

Naithani, Gaurav; Barker, Trevor; Parascandolo, Giambattista; Bramsløw, Lars; Pontoppidan, Niels Henrik; Virtanen, Tuomas

doi:10.1109/waspaa.2017.8169997

Cited by 31 publications

(27 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the advent of deep neural networks (DNNs), a large improvement in the performance of supervised speech separation has been reported starting with Wang and Wang (2013). Various network architectures have been employed, e.g., feedforward DNNs (Grais et al, 2014;Xu et al, 2015), recurrent neural networks (Erdogan et al, 2015;Huang et al, 2015;Weninger et al, 2014), deep autoencoders (Lu et al, 2013), convolutional neural networks (Chandna et al, 2017;Park and Lee, 2016), convolutional recurrent neural networks (Naithani et al, 2017), etc. These DNN-based approaches have employed either time-frequency masking (Huang et al, 2015;Weninger et al, 2014;Williamson and Wang, 2017) or spectral mapping (Grais et al, 2014;Park and Lee, 2016;Xu et al, 2014Xu et al, , 2015 approaches.…”

Section: Introductionmentioning

confidence: 99%

Improving competing voices segregation for hearing impaired listeners using a low-latency deep neural network algorithm

Bramsløw

Naithani

Hafez

et al. 2018

The Journal of the Acoustical Society of America

Self Cite

View full text Add to dashboard Cite

Hearing aid users are challenged in listening situations with noise and especially speech-on-speech situations with two or more competing voices. Specifically, the task of attending to and segregating two competing voices is particularly hard, unlike for normal-hearing listeners, as shown in a small sub-experiment. In the main experiment, the competing voices benefit of a deep neural network (DNN) based stream segregation enhancement algorithm was tested on hearing-impaired listeners. A mixture of two voices was separated using a DNN and presented to the two ears as individual streams and tested for word score. Compared to the unseparated mixture, there was a 13%-point benefit from the separation, while attending to both voices. If only one output was selected as in a traditional target-masker scenario, a larger benefit of 37%-points was found. The results agreed well with objective metrics and show that for hearing-impaired listeners, DNNs have a large potential for improving stream segregation and speech intelligibility in difficult scenarios with two equally important targets without any prior selection of a primary target stream. An even higher benefit can be obtained if the user can select the preferred target via remote control.

show abstract

Section: Introductionmentioning

confidence: 99%

Improving competing voices segregation for hearing impaired listeners using a low-latency deep neural network algorithm

Bramsløw

Naithani

Hafez

et al. 2018

The Journal of the Acoustical Society of America

Self Cite

View full text Add to dashboard Cite

show abstract

“…This DNN predicts a soft mask that is applied to the mixture to yield magnitude estimates. We compare oMISI to its offline counterpart and to the amplitude mask (AM) used as a baseline [20].…”

Section: A Dataset and Protocolmentioning

confidence: 99%

Online Spectrogram Inversion for Low-Latency Audio Source Separation

Magron

Virtanen

2020

IEEE Signal Process. Lett.

Self Cite

View full text Add to dashboard Cite

Audio source separation is usually achieved by estimating the short-time Fourier transform (STFT) magnitude of each source, and then applying a spectrogram inversion algorithm to retrieve time-domain signals. In particular, the multiple input spectrogram inversion (MISI) algorithm has been exploited successfully in several recent works. However, this algorithm suffers from two drawbacks, which we address in this paper. First, it has originally been introduced in a heuristic fashion: we propose here a rigorous optimization framework in which MISI is derived, thus proving the convergence of this algorithm. Besides, while MISI operates offline, we propose here an online version of MISI called oMISI, which is suitable for low-latency source separation, an important requirement for e.g., hearing aids applications. oMISI also allows one to use alternative phase initialization schemes exploiting the temporal structure of audio signals. Experiments conducted on a speech separation task show that oMISI performs as well as its offline counterpart, thus demonstrating its potential for real-time source separation.

show abstract

“…Considering a practical situation in the real world, some research on DNN-based speech enhancement has focused on real-time application [19][20][21][22][23]. To apply an enhancement method in real time, the system must be causal, i.e., it uses past information only and does not require future information to estimate the enhanced signal.…”

Section: Introductionmentioning

confidence: 99%

“…To apply an enhancement method in real time, the system must be causal, i.e., it uses past information only and does not require future information to estimate the enhanced signal. Therefore, uni-directional LSTM are often used in that task [19][20][21][22]. However, as the price of mitigating the vanishing gradient problem, LSTM consists of a lot of parameters as in Fig.…”

Section: Introductionmentioning

confidence: 99%

Real-Time Speech Enhancement Using Equilibriated RNN

Takeuchi

Yatabe

Oikawa

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

We propose a speech enhancement method using a causal deep neural network (DNN) for real-time applications. DNN has been widely used for estimating a time-frequency (T-F) mask which enhances a speech signal. One popular DNN structure for that is a recurrent neural network (RNN) owing to its capability of effectively modelling time-sequential data like speech. In particular, the long short-term memory (LSTM) is often used to alleviate the vanishing/exploding gradient problem which makes the training of an RNN difficult. However, the number of parameters of LSTM is increased as the price of mitigating the difficulty of training, which requires more computational resources. For real-time speech enhancement, it is preferable to use a smaller network without losing the performance. In this paper, we propose to use the equilibriated recurrent neural network (ERNN) for avoiding the vanishing/exploding gradient problem without increasing the number of parameters. The proposed structure is causal, which requires only the information from the past, in order to apply it in real-time. Compared to the uni-and bi-directional LSTM networks, the proposed method achieved the similar performance with much fewer parameters.Index Terms-Real-time speech enhancement, equiribriated recurrent neural network, vanishing/exploding gradient problem.

show abstract

Low latency sound source separation using convolutional recurrent neural networks

Cited by 31 publications

References 20 publications

Improving competing voices segregation for hearing impaired listeners using a low-latency deep neural network algorithm

Improving competing voices segregation for hearing impaired listeners using a low-latency deep neural network algorithm

Online Spectrogram Inversion for Low-Latency Audio Source Separation

Real-Time Speech Enhancement Using Equilibriated RNN

Contact Info

Product

Resources

About