Audio-visual sensing from a quadcopter: dataset and baselines for source localization and sound enhancement

Wang, Lin; Sánchez-Matilla, Ricardo; Cavallaro, Andrea

doi:10.1109/iros40897.2019.8968183

Cited by 17 publications

(20 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use two multi-channel drone-sound datasets, AS [8] and AVQ [51], and a single-channel speech and noise dataset. Table I summarizes the details of the noise data used in this paper.…”

Section: A Experiments Setupmentioning

confidence: 99%

Deep Learning Assisted Time-Frequency Processing for Speech Enhancement on Drones

Wang

Cavallaro

2021

IEEE Trans. Emerg. Top. Comput. Intell.

Self Cite

View full text Add to dashboard Cite

This article fills the gap between the growing interest in signal processing based on Deep Neural Networks (DNN) and the new application of enhancing speech captured by microphones on a drone. In this context, the quality of the target sound is degraded significantly by the strong ego-noise from the rotating motors and propellers. We present the first work that integrates single-channel and multi-channel DNN-based approaches for speech enhancement on drones. We employ a DNN to estimate the ideal ratio masks at individual time-frequency bins, which are subsequently used to design three potential speech enhancement systems, namely singlechannel ego-noise reduction (DNN-S), multi-channel beamforming (DNN-BF), and multi-channel time-frequency spatial filtering (DNN-TF). The main novelty lies in the proposed DNN-TF algorithm, which infers the noise-dominance probabilities at individual time-frequency bins from the DNN-estimated soft masks, and then incorporates them into a time-frequency spatial filtering framework for ego-noise reduction. By jointly exploiting the direction of arrival of the target sound, the time-frequency sparsity of the acoustic signals (speech and ego-noise) and the time-frequency noise-dominance probability, DNN-TF can suppress the ego-noise effectively in scenarios with very low signal-to-noise ratios (e.g. SNR lower than −15 dB), especially when the direction of the target sound is close to that of a source of the ego-noise. Experiments with real and simulated data show the advantage of DNN-TF over competing methods, including DNN-S, DNN-BF and the state-of-the-art time-frequency spatial filtering.

show abstract

“…We use two multi-channel drone-sound datasets, AS [8] and AVQ [51], and a single-channel speech and noise dataset. Table I summarizes the details of the noise data used in this paper.…”

Section: A Experiments Setupmentioning

confidence: 99%

Deep Learning Assisted Time-Frequency Processing for Speech Enhancement on Drones

Wang

Cavallaro

2021

IEEE Trans. Emerg. Top. Comput. Intell.

Self Cite

View full text Add to dashboard Cite

show abstract

“…First, TFS requires the knowledge of the target sound direction and the location of the microphones to estimate the DOA of the sound at each time-frequency bin and calculate the correlation matrix of the target sound. Recently, several sound source localization algorithms were proposed for the drone platform [14], [15], [41]- [45]. Second, TFS is sensitive to the direction of the target sound.…”

Section: A Ego-noise Reductionmentioning

confidence: 99%

“…TFS requires the knowledge of the DOA of the target sound, which can be estimated with sound source localization algorithms [15], [44], or with an onboard camera and an object detector [14], [43]. While the proposed method considers one source only, it can be extended to multiple sources as long as their DOAs are known.…”

Section: G Remarksmentioning

confidence: 99%

“…Eq. (14). We also compare with two traditional beamforming algorithms: fixed delay-and-sum beamformer (FBF) and adaptive beamformer (ABF).…”

Section: A Evaluation Setupmentioning

confidence: 99%

“…Microphone arrays have been widely used for sound enhancement and source localization [9], [10], but most algorithms are designed for indoor settings with relatively high SNR at the microphones, and thus are unsuitable for drone-based applications [11], [12]. Several microphone-array drone sound datasets have been made publicly available recently [13], [14]. The acoustic sensing performance of traditional microphone-array algorithms usually drops significantly due to the extremely low SNR, the time-frequency dynamics of the ego-noise, and additional natural wind noise [8].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Blind Source Separation Framework for Ego-Noise Reduction on Multi-Rotor Drones

Wang

Cavallaro

2020

IEEE/ACM Trans. Audio Speech Lang. Process.

Self Cite

View full text Add to dashboard Cite

Acoustic sensing from a multi-rotor drone is heavily degraded by the strong ego-noise produced by the rotating motors and propellers. To address this problem, we propose a blind source separation (BSS) framework that extracts a target sound from noisy multi-channel signals captured by a microphone array mounted on a drone. The proposed method addresses the challenging problem of permutation alignment, in extremely low signal-to-noise-ratio scenarios (e.g. SNR <-15 dB), by performing clustering on the time activities of the separated signals across frequencies. Since initialization plays an important role to the success of clustering, we propose a pre-processing algorithm which uses time-frequency spatial filtering (TFS) to generate a reference to pre-align the permutation. The pre-alignment not only improves the performance of clustering and permutation alignment, but also solves the target-channel selection problem for BSS. The proposed method integrates the advantages of both TFS and BSS. Experimental results with real-recorded data show that the proposed method is capable of processing the audio stream continuously in a blockwise manner and also remarkably outperforms the state of the art.

show abstract

Multi-sensory sound source enhancement for unmanned aerial vehicle recordings

et al. 2022

View full text Add to dashboard Cite

Audio-visual sensing from a quadcopter: dataset and baselines for source localization and sound enhancement

Cited by 17 publications

References 27 publications

Deep Learning Assisted Time-Frequency Processing for Speech Enhancement on Drones

Deep Learning Assisted Time-Frequency Processing for Speech Enhancement on Drones

A Blind Source Separation Framework for Ego-Noise Reduction on Multi-Rotor Drones

Multi-sensory sound source enhancement for unmanned aerial vehicle recordings

Contact Info

Product

Resources

About