Multimodal fusion for indoor sound source localization

Chen, Jinhui; Takashima, Ryoichi; Guo, Xingchen; Xu, Xuexin; Takiguchi, Tetsuya; Hancock, Edwin R.

doi:10.1016/j.patcog.2021.107906

Cited by 13 publications

(5 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To train this model, 360°images with multi-channel audio signals were fed into visual and auditory DNNs. Chen et al [23] replaced the DNN with SVM and trained the models using 1-channel audio from a single microphone to simultaneously estimate the DoA and the distance between the sound source and the microphone. Regrettably, although the above systems can accurately locate multiple sound sources at the same time, they are unable to identify the sources and focus on a specific target, which is actually a highly desired capability of robot audition in cocktail party environments.…”

Section: Related Work 21 Audio-visual Sound Source Localization (Av-ssl)mentioning

confidence: 99%

Audio–Visual Sound Source Localization and Tracking Based on Mobile Robot for The Cocktail Party Problem

2023

View full text Add to dashboard Cite

Locating the sound source is one of the most important capabilities of robot audition. In recent years, single-source localization techniques have increasingly matured. However, localizing and tracking specific sound sources in multi-source scenarios, which is known as the cocktail party problem, is still unresolved. In order to address this challenge, in this paper, we propose a system for dynamically localizing and tracking sound sources based on audio–visual information that can be deployed on a mobile robot. Our system first locates specific targets using pre-registered voiceprint and face features. Subsequently, the robot moves to track the target while keeping away from other sound sources in the surroundings instructed by the motion module, which helps the robot gather clearer audio data of the target to perform downstream tasks better. Its effectiveness has been verified via extensive real-world experiments with a 20% improvement in the success rate of specific speaker localization and a 14% reduction in word error rate in speech recognition compared to its counterparts.

show abstract

Section: Related Work 21 Audio-visual Sound Source Localization (Av-ssl)mentioning

confidence: 99%

Audio–Visual Sound Source Localization and Tracking Based on Mobile Robot for The Cocktail Party Problem

2023

View full text Add to dashboard Cite

show abstract

“…4) Deep Learning-Based SOL Approaches: Recently, SOL-related works are all based on deep learning [113], [114], [115], [116], whose key idea is to perform audio and visual feature embedding. And most of them can be roughly divided into two groups: 1) the class activation mapping-(CAM-) based ones and 2) the feature similarity-based ones.…”

Section: Sounding Object Localization (Sol)mentioning

confidence: 99%

A Comprehensive Survey on Video Saliency Detection With Auditory Information: The Audio-Visual Consistency Perceptual is the Key!

Chen

Mengke

Song

et al. 2023

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

Video saliency detection (VSD) aims at fast locating the most attractive objects/things/patterns in a given video clip. Existing VSD-related works have mainly relied on the visual system but paid less attention to the audio aspect. In contrast, our audio system is the most vital complementary part of our visual system. Also, audio-visual saliency detection (AVSD), one of the most representative research topics for mimicking human perceptual mechanisms, is currently in its infancy, and none of the existing survey papers have touched on it, especially from the perspective of saliency detection. Thus, the ultimate goal of this paper is to provide an extensive review to bridge the gap between audio-visual fusion and saliency detection. In addition, as another highlight of this review, we have provided a deep insight into key factors that could directly determine AVSD deep models' performances. We claim that the audio-visual consistency degree (AVC) -a long-overlooked issue, can directly influence the effectiveness of using audio to benefit its visual counterpart when performing saliency detection. Moreover, to make the AVC issue more practical and valuable for future followers, we have newly equipped almost all existing publicly available AVSD datasets with additional frame-wise AVC labels. Based on these upgraded datasets, we have conducted extensive quantitative evaluations to ground our claim on the importance of AVC in the AVSD task. In a word, our ideas and new sets serve as a convenient platform with preliminaries and guidelines, all of which can potentially facilitate future works in further promoting stateof-the-art (SOTA) performance.

show abstract

“…However, when encountering a strong reflection condition, noise source localization will encounter serious problems [9] . At present, there are many dereverberation processing methods for speech signal [10][11] . However, there are few methods for steady noise source localization in a strong reflection condition, which still needs further research.…”

Section: Introductionmentioning

confidence: 99%

Study on noise source localization method in strong reflection condition

Guan

Liao

et al. 2023

International Conference on Image, Signal Processing, and Pattern Recognition (ISPP 2023)

View full text Add to dashboard Cite

Noise source localization is very important in the field of acoustic measurement. However, the localization accuracy in strong reflection conditions using traditional localization method will be seriously disturbed by the sidelobes. In this study, a new noise source localization method of multi-frequency energy superposition (MES) was proposed to reduce the effect of sidelobes. The noise source maps in strong reflection conditions obtained by MES method were analyzed and compared with delay and sum method. The results show that the main-to-sidelobe suppression ratio of source localization obtained from MES method is higher than that from traditional delay and sum method, indicating that the proposed MES method is more accurate for noise source localization in strong reflection conditions.

show abstract

Multimodal fusion for indoor sound source localization

Cited by 13 publications

References 43 publications

Audio–Visual Sound Source Localization and Tracking Based on Mobile Robot for The Cocktail Party Problem

Audio–Visual Sound Source Localization and Tracking Based on Mobile Robot for The Cocktail Party Problem

A Comprehensive Survey on Video Saliency Detection With Auditory Information: The Audio-Visual Consistency Perceptual is the Key!

Study on noise source localization method in strong reflection condition

Contact Info

Product

Resources

About