Neural Full-Rank Spatial Covariance Analysis for Blind Source Separation

Abstract

This paper describes a neural blind source separation (BSS) method based on amortized variational inference (AVI) of a non-linear generative model of mixture signals. A classical statistical approach to BSS is to fit a linear generative model that consists of spatial and source models representing the interchannel covariances and power spectral densities of sources, respectively. Although the variational autoencoder (VAE) has successfully been used as a non-linear source model with latent features, it should be pretrained from a sufficient amount of isolated signals. Our method, in contrast, enables the VAE-based source model to be trained only from mixture signals. Specifically, we introduce a neural mixture-to-feature inference model that directly infers the latent features from the observed mixture and integrate it with a neural feature-to-mixture generative model consisting of a full-rank spatial model and a VAE-based source model. All the models are optimized jointly such that the likelihood for the training mixtures is maximized in the framework of AVI. Once the inference model is optimized, it can be used for estimating the latent features of sources included in unseen mixture signals. The experimental results show that the proposed method outperformed the state-of-the-art BSS methods based on linear generative models and was comparable to a method based on supervised learning of the VAE-based source model. Index Terms-Neural source separation, unsupervised training, deep generative models, variational autoencoders I. INTRODUCTION S OUND source separation is a fundamental function to understand acoustic scenes computationally [1]-[4]. Blind source separation (BSS), for example, has been actively investigated to separate a multichannel mixture signal into source signals with little prior information about the sources and microphones [5]-[10]. Such a modern statistical multichannel method is based on a generative model consisting of spatial Manuscript

Joint Separation and Localization of Moving Sound Sources Based on Neural Full-Rank Spatial Covariance Analysis

“…The deep spectral model can be trained in an unsupervised manner by using only multichannel mixtures based on amortized variational inference [21], [33]. This method called neural FCA utilizes an inference (encoder) network to predict the speech features Z ≜ {z nt } N,T n,t=1 from an input mixture X ≜ {x f t } F,T f,t=1 as the posterior distribution q ϕ (Z | X), where ϕ represents the network parameters.…”

“…1). As the estimates of z nt , following the original neural FCA [21], the inference model predicts the posterior distribution q ϕ (Z | X) as follows:…”

“…After training these networks, they can be used to separate and localize moving sources in an unseen mixture. The source signals are obtained with the source images Y nf t ≜ g θ,f (µ ϕ,nt (X))H nf t by a multichannel Wiener filter [14], [21].…”

“…We used almost the same network configuration as that of the original neural FCA [21], whose inference and generative models consisted of temporal convolutional networks. We added the dropout (p = 0.1) to avoid bad local optima and two output layers to the inference model for estimating the TF mask and DoA.…”

“…Multichannel nonnegative matrix factorization (MNMF) [15]- [17] improves the performance of FCA by assuming the source spectra to be low-rank. Its nonlinear extension called neural FCA [21], [22] can precisely represent the source spectra with a neural network called a deep spectral model [23]- [27]. This model can be trained blindly by maximizing the log-marginal likelihood for the training data of multichannel mixtures in advance and was reported to be comparable with a multichannel supervised model [21], [28].…”