Joint Localization and Classification of Multiple Sound Sources Using a Multi-task Neural Network

He, Weipeng; Motlíček, Petr; Odobez, Jean-Marc

doi:10.21437/interspeech.2018-1269

Cited by 32 publications

(19 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They have been used with various inputs: binaural features [24], GCC features [25], the eigenvectors of the spatial covariance matrix [26], raw short-time Fourier transform (STFT) of signals [27]- [29], including for Ambisonics signals in [29]. Different architectures have been tested: feed-forward neural networks [24], convolutional neural networks (CNNs) [27], [30], deep residual networks [31], convolutional and recurrent networks (CRNNs) [29]. Yet, most of these methods have only been evaluated in simulated environments similar to the training conditions, which is not sufficient to verify their generalization to real-life applications.…”

Section: Introductionmentioning

confidence: 99%

CRNN-Based Multiple DoA Estimation Using Acoustic Intensity Features for Ambisonics Recordings

Perotin

Serizel

Vincent

et al. 2019

IEEE J. Sel. Top. Signal Process.

117

126

View full text Add to dashboard Cite

Localizing audio sources is challenging in real reverberant environments, especially when several sources are active. We propose to use a neural network built from stacked convolutional and recurrent layers in order to estimate the directions of arrival of multiple sources from a first-order Ambisonics recording. It returns the directions of arrival over a discrete grid of a known number of sources. We propose to use features derived from the acoustic intensity vector as inputs. We analyze the behavior of the neural network by means of a visualization technique called layerwise relevance propagation. This analysis highlights which parts of the input signal are relevant in a given situation. We also conduct experiments to evaluate the performance of our system in various environments, from simulated rooms to real recordings, with one or two speech sources. The results show that the proposed features significantly improve performances with respect to raw Ambisonics inputs.

show abstract

Section: Introductionmentioning

confidence: 99%

CRNN-Based Multiple DoA Estimation Using Acoustic Intensity Features for Ambisonics Recordings

Perotin

Serizel

Vincent

et al. 2019

IEEE J. Sel. Top. Signal Process.

117

126

View full text Add to dashboard Cite

show abstract

“…Most often, the number of the sound sources desired to be localized is known or set in advance, although localization of an a-priori unknown number of sources was also demonstrated (Cao et al 2021;Chazan et al 2019;He et al 2018b).…”

Section: Learning-based Sound Source Localizationmentioning

confidence: 99%

Improvement of learning-based methods for localization of multiple sound sources

Sakavicius¹

2021

View full text Add to dashboard Cite

“…Such ideal conditions hardly hold true in real-world applications and usually require special treatments [5,6]. Data-driven methods, and in particular deep learning, have recently outperformed classical signal-processing methods for various audio tasks [7,8] including SSL [9,10].…”

Section: Introductionmentioning

confidence: 99%

Data-Efficient Framework for Real-World Multiple Sound Source 2d Localization

Moing

Vinayavekhin

Agravante

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Deep neural networks have recently led to promising results for the task of multiple sound source localization. Yet, they require a lot of training data to cover a variety of acoustic conditions and microphone array layouts. One can leverage acoustic simulators to inexpensively generate labeled training data. However, models trained on synthetic data tend to perform poorly with real-world recordings due to the domain mismatch. Moreover, learning for different microphone array layouts makes the task more complicated due to the infinite number of possible layouts. We propose to use adversarial learning methods to close the gap between synthetic and real domains. Our novel ensemble-discrimination method significantly improves the localization performance without requiring any label from the real data. Furthermore, we propose a novel explicit transformation layer to be embedded in the localization architecture. It enables the model to be trained with data from specific microphone array layouts while generalizing well to unseen layouts during inference.

show abstract

Joint Localization and Classification of Multiple Sound Sources Using a Multi-task Neural Network

Cited by 32 publications

References 17 publications

CRNN-Based Multiple DoA Estimation Using Acoustic Intensity Features for Ambisonics Recordings

CRNN-Based Multiple DoA Estimation Using Acoustic Intensity Features for Ambisonics Recordings

Improvement of learning-based methods for localization of multiple sound sources

Data-Efficient Framework for Real-World Multiple Sound Source 2d Localization

Contact Info

Product

Resources

About