Localization, Detection and Tracking of Multiple Moving Sound Sources
                        with a Convolutional Recurrent Neural Network

Adavanne, Sharath; Politis, Archontis; Virtanen, Tuomas

doi:10.33682/xb0q-a335

Cited by 33 publications

(38 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most of these considerations were addressed in the recent dataset for the new DCASE2020 challenge [55]. A significant advance is the introduction of reverberant moving sources, still based on captured RIRs from real spaces [55], [56]. Moreover, ambient noise occurs at varying levels, reverberant conditions are stronger and more varied, and event locations do not occur in a sparse regular grid but can vary more or less continuously.…”

Section: Discussionmentioning

confidence: 99%

Overview and Evaluation of Sound Event Localization and Detection in DCASE 2019

Politis

Mesaros

Adavanne

et al. 2021

IEEE/ACM Trans. Audio Speech Lang. Process.

Self Cite

View full text Add to dashboard Cite

Sound event localization and detection is a novel area of research that emerged from the combined interest of analyzing the acoustic scene in terms of the spatial and temporal activity of sounds of interest. This paper presents an overview of the first international evaluation on sound event localization and detection, organized as a task of the DCASE 2019 Challenge. A large-scale realistic dataset of spatialized sound events was generated for the challenge, to be used for training of learning-based approaches, and for evaluation of the submissions in an unlabeled subset. The overview presents in detail how the systems were evaluated and ranked and the characteristics of the best-performing systems. Common strategies in terms of input features, model architectures, training approaches, exploitation of prior knowledge, and data augmentation are discussed. Since ranking in the challenge was based on individually evaluating localization and event classification performance, part of the overview focuses on presenting metrics for the joint measurement of the two, together with a reevaluation of submissions using these new metrics. The new analysis reveals submissions that performed better on the joint task of detecting the correct type of event close to its original location than some of the submissions that were ranked higher in the challenge. Consequently, ranking of submissions which performed strongly when evaluated separately on detection or localization, but not jointly on both, was affected negatively.

show abstract

Section: Discussionmentioning

confidence: 99%

Overview and Evaluation of Sound Event Localization and Detection in DCASE 2019

Politis

Mesaros

Adavanne

et al. 2021

IEEE/ACM Trans. Audio Speech Lang. Process.

Self Cite

View full text Add to dashboard Cite

show abstract

“…In this context, most studies have evaluated sound localization performance in the frontal horizontal hemisphere with static sound scenarios (Grantham et al 2007;Kerber & Seeber 2012;Jones et al 2014;Dorman et al 2016). In the future, modern hearing aid and implant audio processors will apply advanced, data science-based auditory scene analysis for improved hearing outcomes (Ma et al 2017;Adavanne et al 2019).…”

Section: Discussionmentioning

confidence: 99%

Dynamic sound field audiometry: Static and dynamic spatial hearing tests in the full horizontal plane

et al. 2020

View full text Add to dashboard Cite

Although spatial hearing is of great importance in everyday life, today's routine audiological test batteries and static test setups assess sound localization, discrimination and tracking abilities rudimentarily and thus provide only a limited interpretation of treatment outcomes regarding spatial hearing performance. To address this limitation, we designed a dynamic sound field test setup and evaluated the sound localization, discrimination and tracking performance of 12 normal-hearing subjects. During testing, participants provided feedback either through a touchpad or through eye tracking. In addition, the influence of head movement on sound-tracking performance was investigated. Our results show that tracking and discrimination performance was significantly better in the frontal azimuth than in the dorsal azimuth. Particularly good performance was observed in the backward direction across localization, discrimination and tracking tests. As expected, free head movement improved sound-tracking abilities. Furthermore, feedback via gaze detection led to larger tracking errors than feedback via the touchpad. We found statistically significant correlations between the static and dynamic tests, which favor the snapshot theory for auditory motion perception. .

show abstract

“…Similar to the approaches proposed in SELDnet [31] and DOAnet [32], we extracted magnitude and phase components from spectrograms of each channel of the microphone array. The magnitude and phase components were then stacked along the channel dimension and treated as a M/2 by 2 × C image, where M is the window length of the Fourier transformation and C is the number of channels (Fig.…”

Section: A Feature Extraction 1) Audiomentioning

confidence: 99%

“…After processing the whole audio sequence with 50% overlap on window size, the shape of the audio data was T × M/2 × 2C, where T is the number of data points in the time dimension. The parameters used for our feature extraction followed those in SELDnet [31] and DOAnet [32]. 1 Our code is available at: https://intuitivecomputing.jhu.edu/openscience.html…”

Section: A Feature Extraction 1) Audiomentioning

confidence: 99%

Object Permanence Through Audio-Visual Representations

Bu¹,

Huang

2021

IEEE Access

View full text Add to dashboard Cite

As robots perform manipulation tasks and interact with objects, it is probable that they accidentally drop objects that subsequently bounce out of their visual fields (e.g., due to an inadequate grasp of an unfamiliar object). To enable robots to recover from such errors, we draw upon the concept of object permanence-objects remain in existence even when they are not being sensed (e.g., seen) directly. In particular, we developed a multimodal neural network model-using a partial, observed bounce trajectory and the audio resulting from drop impact as its inputs-to predict the full bounce trajectory and the end location of a dropped object. We empirically show that: (1) our multimodal method predicted end locations close in proximity (i.e., within the visual field of the robot's wrist camera) to the actual locations and (2) the robot was able to retrieve dropped objects by applying minimal vision-based pick-up adjustments. Additionally, we show that our method outperformed five comparison baselines in retrieving dropped objects. Our results contribute to enabling object permanence for robots and error recovery from object drops.

show abstract

Localization, Detection and Tracking of Multiple Moving Sound Sources with a Convolutional Recurrent Neural Network

Cited by 33 publications

References 26 publications

Overview and Evaluation of Sound Event Localization and Detection in DCASE 2019

Overview and Evaluation of Sound Event Localization and Detection in DCASE 2019

Dynamic sound field audiometry: Static and dynamic spatial hearing tests in the full horizontal plane

Object Permanence Through Audio-Visual Representations

Contact Info

Product

Resources

About