Tracking a moving sound source from a multi-rotor drone

Wang, Lin; Sánchez-Matilla, Ricardo; Cavallaro, Andrea

doi:10.1109/iros.2018.8594483

Cited by 21 publications

(24 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unlike DNN-BF, DNN-TF requires the knowledge of the target DOA and the location of the microphones. In practice, the target DOA can be estimated using existing sound source localization algorithms that are dedicated to drone sound processing [18], [19], or be estimated by exploiting additional vision information [13], [51]. Furthermore, DNN-TF can separate multiple speakers talking simultaneously if their DOAs are known.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Deep Learning Assisted Time-Frequency Processing for Speech Enhancement on Drones

Wang

Cavallaro

2021

IEEE Trans. Emerg. Top. Comput. Intell.

Self Cite

View full text Add to dashboard Cite

This article fills the gap between the growing interest in signal processing based on Deep Neural Networks (DNN) and the new application of enhancing speech captured by microphones on a drone. In this context, the quality of the target sound is degraded significantly by the strong ego-noise from the rotating motors and propellers. We present the first work that integrates single-channel and multi-channel DNN-based approaches for speech enhancement on drones. We employ a DNN to estimate the ideal ratio masks at individual time-frequency bins, which are subsequently used to design three potential speech enhancement systems, namely singlechannel ego-noise reduction (DNN-S), multi-channel beamforming (DNN-BF), and multi-channel time-frequency spatial filtering (DNN-TF). The main novelty lies in the proposed DNN-TF algorithm, which infers the noise-dominance probabilities at individual time-frequency bins from the DNN-estimated soft masks, and then incorporates them into a time-frequency spatial filtering framework for ego-noise reduction. By jointly exploiting the direction of arrival of the target sound, the time-frequency sparsity of the acoustic signals (speech and ego-noise) and the time-frequency noise-dominance probability, DNN-TF can suppress the ego-noise effectively in scenarios with very low signal-to-noise ratios (e.g. SNR lower than −15 dB), especially when the direction of the target sound is close to that of a source of the ego-noise. Experiments with real and simulated data show the advantage of DNN-TF over competing methods, including DNN-S, DNN-BF and the state-of-the-art time-frequency spatial filtering.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Eq. (19)) and then normalizing it with the highest frequency count [8]. It can be observed that the shape of the spatial likelihood function varies for the two segments.…”

Section: Problem Definitionmentioning

confidence: 99%

Deep Learning Assisted Time-Frequency Processing for Speech Enhancement on Drones

Wang

Cavallaro

2021

IEEE Trans. Emerg. Top. Comput. Intell.

Self Cite

View full text Add to dashboard Cite

show abstract

“…TFS requires the knowledge of the DOA of the target sound, which can be estimated with sound source localization algorithms [15], [44], or with an onboard camera and an object detector [14], [43]. While the proposed method considers one source only, it can be extended to multiple sources as long as their DOAs are known.…”

Section: G Remarksmentioning

confidence: 99%

A Blind Source Separation Framework for Ego-Noise Reduction on Multi-Rotor Drones

Wang

Cavallaro

2020

IEEE/ACM Trans. Audio Speech Lang. Process.

Self Cite

View full text Add to dashboard Cite

Acoustic sensing from a multi-rotor drone is heavily degraded by the strong ego-noise produced by the rotating motors and propellers. To address this problem, we propose a blind source separation (BSS) framework that extracts a target sound from noisy multi-channel signals captured by a microphone array mounted on a drone. The proposed method addresses the challenging problem of permutation alignment, in extremely low signal-to-noise-ratio scenarios (e.g. SNR <-15 dB), by performing clustering on the time activities of the separated signals across frequencies. Since initialization plays an important role to the success of clustering, we propose a pre-processing algorithm which uses time-frequency spatial filtering (TFS) to generate a reference to pre-align the permutation. The pre-alignment not only improves the performance of clustering and permutation alignment, but also solves the target-channel selection problem for BSS. The proposed method integrates the advantages of both TFS and BSS. Experimental results with real-recorded data show that the proposed method is capable of processing the audio stream continuously in a blockwise manner and also remarkably outperforms the state of the art.

show abstract

“…Unmanned aerial vehicles (UAVs), ubiquitously known as drones, have found great use in a wide range of applications-from casual use in photography to search and rescue operations where human lives are at stake. Reports by the United Nations and other humanitarian using embedded microphone arrays in the UAVs to triangulate the sound coming from emergency whistles or humans trapped beneath debris [3][4][5][6][7]. It is evident that a sound source localization (SSL)-based detection system can compliment the visual detection in scenarios where the field of view may be occluded due to obstacles or bad lighting or even operations carried out at night.…”

Section: Introductionmentioning

confidence: 99%

“…Such an algorithm was proposed by [4] which involved a modified version of the MUSIC algorithm based on incremental generalized singular value decomposition (iGSVD-MUSIC). Furthermore, in order to locate and track a moving sound source, an approach involving time-frequency spatial filtering combined with a particle filter was described to perform well under noisy conditions by [6].…”

Section: Introductionmentioning

confidence: 99%

DOANet: a deep dilated convolutional neural network approach for search and rescue with drone-embedded sound source localization

Qayyum

Hassan

Anika

et al. 2020

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

Drone-embedded sound source localization (SSL) has interesting application perspective in challenging search and rescue scenarios due to bad lighting conditions or occlusions. However, the problem gets complicated by severe drone ego-noise that may result in negative signal-to-noise ratios in the recorded microphone signals. In this paper, we present our work on drone-embedded SSL using recordings from an 8-channel cube-shaped microphone array embedded in an unmanned aerial vehicle (UAV). We use angular spectrum-based TDOA (time difference of arrival) estimation methods such as generalized cross-correlation phase-transform (GCC-PHAT), minimum-variance-distortion-less-response (MVDR) as baseline, which are state-of-the-art techniques for SSL. Though we improve the baseline method by reducing ego-noise using speed correlated harmonics cancellation (SCHC) technique, our main focus is to utilize deep learning techniques to solve this challenging problem. Here, we propose an end-to-end deep learning model, called DOANet, for SSL. DOANet is based on a one-dimensional dilated convolutional neural network that computes the azimuth and elevation angles of the target sound source from the raw audio signal. The advantage of using DOANet is that it does not require any hand-crafted audio features or ego-noise reduction for DOA estimation. We then evaluate the SSL performance using the proposed and baseline methods and find that the DOANet shows promising results compared to both the angular spectrum methods with and without SCHC. To evaluate the different methods, we also introduce a well-known parameter—area under the curve (AUC) of cumulative histogram plots of angular deviations—as a performance indicator which, to our knowledge, has not been used as a performance indicator for this sort of problem before.

show abstract

Tracking a moving sound source from a multi-rotor drone

Cited by 21 publications

References 26 publications

Deep Learning Assisted Time-Frequency Processing for Speech Enhancement on Drones

Deep Learning Assisted Time-Frequency Processing for Speech Enhancement on Drones

A Blind Source Separation Framework for Ego-Noise Reduction on Multi-Rotor Drones

DOANet: a deep dilated convolutional neural network approach for search and rescue with drone-embedded sound source localization

Contact Info

Product

Resources

About