Deep Bayesian Unsupervised Source Separation Based On A Complex Gaussian Mixture Model

Bando, Yoshiaki; Sasaki, Yoko; Yoshii, Kazuyoshi

doi:10.1109/mlsp.2019.8918699

Cited by 10 publications

(5 citation statements)

References 26 publications

(70 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For monocular depth estimation, self-supervised learning is an active research topic because oracle depth maps are not easy to obtain [19]. Self-supervised learning of neural sound source separation has also been investigated by using a spatial model of multichannel audio signals [20], [21]. These methods do not require clean source signals, which are often unavailable in the real recordings.…”

Section: B Self-supervised Learning With Audio and Visual Datamentioning

confidence: 99%

“…In order to achieve localization of objects with similar appearance, we present a self-supervised training method for multichannel neural AV-SSL that uses pairs of 360 • images and the corresponding multichannel audio mixtures. The selfsupervised training is formulated in a probabilistic manner based on a spatial audio model called a cGMM as in [21].…”

Section: Self-supervised Training Of Audio-visual Sound Source Locali...mentioning

confidence: 99%

See 1 more Smart Citation

Self-supervised Neural Audio-Visual Sound Source Localization via Probabilistic Spatial Modeling

Masuyama

Bando

Yatabe

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Detecting sound source objects within visual observation is important for autonomous robots to comprehend surrounding environments. Since sounding objects have a large variety with different appearances in our living environments, labeling all sounding objects is impossible in practice. This calls for self-supervised learning which does not require manual labeling. Most of conventional self-supervised learning uses monaural audio signals and images and cannot distinguish sound source objects having similar appearances due to poor spatial information in audio signals. To solve this problem, this paper presents a self-supervised training method using 360 • images and multichannel audio signals. By incorporating with the spatial information in multichannel audio signals, our method trains deep neural networks (DNNs) to distinguish multiple sound source objects. Our system for localizing sound source objects in the image is composed of audio and visual DNNs. The visual DNN is trained to localize sound source candidates within an input image. The audio DNN verifies whether each candidate actually produces sound or not. These DNNs are jointly trained in a self-supervised manner based on a probabilistic spatial audio model. Experimental results with simulated data showed that the DNNs trained by our method localized multiple speakers. We also demonstrate that the visual DNN detected objects including talking visitors and specific exhibits from real data recorded in a science museum.

show abstract

Section: B Self-supervised Learning With Audio and Visual Datamentioning

confidence: 99%

Section: Self-supervised Training Of Audio-visual Sound Source Locali...mentioning

confidence: 99%

Self-supervised Neural Audio-Visual Sound Source Localization via Probabilistic Spatial Modeling

Masuyama

Bando

Yatabe

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…TF masks for the integration into the mixture model are obtained with a bidirectional long short-term memory (LSTM) network in [18]. Both [16] and [21] take advantage of spatial clustering methods to train neural networks in an unsupervised way, as well as to compute the final masks in the end. In [17], a convolutional neural network (CNN) with utterance-level PIT is employed prior to the mixture model-based mask estimation.…”

Section: Introductionmentioning

confidence: 99%

DOA-guided source separation with direction-based initialization and time annotations using complex angular central Gaussian mixture models

Bohlender

Severen

Sterckx

et al. 2022

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

By means of spatial clustering and time-frequency masking, a mixture of multiple speakers and noise can be separated into the underlying signal components. The parameters of a model, such as a complex angular central Gaussian mixture model (cACGMM), can be determined based on the given signal mixture itself. Then, no misfit between training and testing conditions arises, as opposed to approaches that require labeled datasets to be trained. Whereas the separation can be performed in a completely unsupervised way, it may be beneficial to take advantage of a priori knowledge. The parameter estimation is sensitive to the initialization, and it is necessary to address the frequency permutation problem. In this paper, we therefore consider three techniques to overcome these limitations using direction of arrival (DOA) estimates. First, we propose an initialization with simple DOA-based masks. Secondly, we derive speaker specific time annotations from the same masks in order to constrain the cACGMM. Thirdly, we employ an approach where the mixture components are specific to each DOA instead of each speaker. We conduct experiments with sudden DOA changes, as well as a gradually moving speaker. The results demonstrate that particularly the DOA-based initialization is effective to overcome both of the described limitations. In this case, even methods based on normally unavailable oracle information are not observed to be more beneficial to the permutation resolution or the initialization. Lastly, we also show that the proposed DOA-guided source separation works quite robustly in the presence of adverse conditions and realistic DOA estimation errors.

show abstract

“…Teacher-student approach has also been studied to address the mismatch between the real and simulated environments recently [10][11][12][13]. When multiple microphones are available, a standard approach is to use an unsupervised spatial clustering approach as a teacher for a single-channel separation network student [10][11][12]. However, this approach relies on multimodal training data and has not been evaluated with recent developed end-to-end separation models.…”

Section: Introductionmentioning

confidence: 99%

Teacher-Student MixIT for Unsupervised and Semi-Supervised Speech Separation

Zhang¹,

Zorilă²,

Doddipatla³

et al. 2021

Interspeech 2021

View full text Add to dashboard Cite

In this paper, we introduce a novel semi-supervised learning framework for end-to-end speech separation. The proposed method first uses mixtures of unseparated sources and the mixture invariant training (MixIT) criterion to train a teacher model. The teacher model then estimates separated sources that are used to train a student model with standard permutation invariant training (PIT). The student model can be fine-tuned with supervised data, i.e., paired artificial mixtures and clean speech sources, and further improved via model distillation. Experiments with single and multi channel mixtures show that the teacher-student training resolves the over-separation problem observed in the original MixIT method. Further, the semisupervised performance is comparable to a fully-supervised separation system trained using ten times the amount of supervised data.

show abstract

Deep Bayesian Unsupervised Source Separation Based On A Complex Gaussian Mixture Model

Cited by 10 publications

References 26 publications

Self-supervised Neural Audio-Visual Sound Source Localization via Probabilistic Spatial Modeling

Self-supervised Neural Audio-Visual Sound Source Localization via Probabilistic Spatial Modeling

DOA-guided source separation with direction-based initialization and time annotations using complex angular central Gaussian mixture models

Teacher-Student MixIT for Unsupervised and Semi-Supervised Speech Separation

Contact Info

Product

Resources

About