A Survey of Sound Source Localization with Deep Learning Methods

Grumiaux, Pierre-Amaury; Kitić, Srđan; Girin, Laurent; Guérin, Alexandre

doi:10.48550/arxiv.2109.03465

Cited by 12 publications

(14 citation statements)

References 187 publications

(461 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, these approaches are sensitive to room acoustics and noisy backgrounds and may be unreliable when multiple sources are present. More recently, machine learning has been used for direction of arrival estimation with some success [12,13,19]. Although these methods improve upon the traditional approaches, the lack of visual information limits the efficacy of these systems in real-word settings.…”

Section: Related Workmentioning

confidence: 99%

Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization

Jiang¹,

Murdock²,

Ithapu³

2022

Preprint

View full text Add to dashboard Cite

Augmented reality devices have the potential to enhance human perception and enable other assistive functionalities in complex conversational environments. Effectively capturing the audio-visual context necessary for understanding these social interactions first requires detecting and localizing the voice activities of the device wearer and the surrounding people. These tasks are challenging due to their egocentric nature: the wearer's head motion may cause motion blur, surrounding people may appear in difficult viewing angles, and there may be occlusions, visual clutter, audio noise, and bad lighting. Under these conditions, previous state-of-the-art active speaker detection methods do not give satisfactory results. Instead, we tackle the problem from a new setting using both video and multi-channel microphone array audio. We propose a novel end-to-end deep learning approach that is able to give robust voice activity detection and localization results. In contrast to previous methods, our method localizes active speakers from all possible directions on the sphere, even outside the camera's field of view, while simultaneously detecting the device wearer's own voice activity. Our experiments show that the proposed method gives superior results, can run in real time, and is robust against noise and clutter.

show abstract

Section: Related Workmentioning

confidence: 99%

Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization

Jiang¹,

Murdock²,

Ithapu³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Among the most widely used methods are narrowband realizations of steered response power (SRP) [29,30], where the direction is determined by maximizing the output power of a beamformer, as well as subspace decomposition-based methods like MUSIC [31]. Alternatively, a deep learning approach can also be used [32].…”

Section: Doa-guided Source Separationmentioning

confidence: 99%

DOA-guided source separation with direction-based initialization and time annotations using complex angular central Gaussian mixture models

Bohlender

Severen

Sterckx

et al. 2022

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

By means of spatial clustering and time-frequency masking, a mixture of multiple speakers and noise can be separated into the underlying signal components. The parameters of a model, such as a complex angular central Gaussian mixture model (cACGMM), can be determined based on the given signal mixture itself. Then, no misfit between training and testing conditions arises, as opposed to approaches that require labeled datasets to be trained. Whereas the separation can be performed in a completely unsupervised way, it may be beneficial to take advantage of a priori knowledge. The parameter estimation is sensitive to the initialization, and it is necessary to address the frequency permutation problem. In this paper, we therefore consider three techniques to overcome these limitations using direction of arrival (DOA) estimates. First, we propose an initialization with simple DOA-based masks. Secondly, we derive speaker specific time annotations from the same masks in order to constrain the cACGMM. Thirdly, we employ an approach where the mixture components are specific to each DOA instead of each speaker. We conduct experiments with sudden DOA changes, as well as a gradually moving speaker. The results demonstrate that particularly the DOA-based initialization is effective to overcome both of the described limitations. In this case, even methods based on normally unavailable oracle information are not observed to be more beneficial to the permutation resolution or the initialization. Lastly, we also show that the proposed DOA-guided source separation works quite robustly in the presence of adverse conditions and realistic DOA estimation errors.

show abstract

“…Estimating the direction of arrival (DOA) of a sound while also classifying the type of event is an important type of frontend processing for a wide variety of monitoring and robotic applications. Due to its wide applicability, this task, often referred to in the literature as sound event localization and detection (SELD), has recently seen a surge of interest [1][2][3][4]. However, SELD remains challenging because sound sources can move, cease to produce sound, have their positions obscured by room reverberation, and are often mixed with interfering sounds.…”

Section: Introductionmentioning

confidence: 99%

Locate This, Not That: Class-Conditioned Sound Event DOA Estimation

Slizovskaia¹,

Wichern²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Existing systems for sound event localization and detection (SELD) typically operate by estimating a source location for all classes at every time instant. In this paper, we propose an alternative class-conditioned SELD model for situations where we may not be interested in localizing all classes all of the time. This class-conditioned SELD model takes as input the spatial and spectral features from the sound file, and also a one-hot vector indicating the class we are currently interested in localizing. We inject the conditioning information at several points in our model using feature-wise linear modulation (FiLM) layers. Through experiments on the DCASE 2020 Task 3 dataset, we show that the proposed class-conditioned SELD model performs better in terms of common SELD metrics than the baseline model that locates all classes simultaneously, and also outperforms specialist models that are trained to locate only a single class of interest. We also evaluate performance on the DCASE 2021 Task 3 dataset, which includes directional interference (sound events from classes we are not interested in localizing) and notice especially strong improvement from the class-conditioned model.

show abstract

A Survey of Sound Source Localization with Deep Learning Methods

Cited by 12 publications

References 187 publications

Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization

Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization

DOA-guided source separation with direction-based initialization and time annotations using complex angular central Gaussian mixture models

Locate This, Not That: Class-Conditioned Sound Event DOA Estimation

Contact Info

Product

Resources

About