Deep Learning Assisted Time-Frequency Processing for Speech Enhancement on Drones

Abstract

This article fills the gap between the growing interest in signal processing based on Deep Neural Networks (DNN) and the new application of enhancing speech captured by microphones on a drone. In this context, the quality of the target sound is degraded significantly by the strong ego-noise from the rotating motors and propellers. We present the first work that integrates single-channel and multi-channel DNN-based approaches for speech enhancement on drones. We employ a DNN to estimate the ideal ratio masks at individual time-frequency bins, which are subsequently used to design three potential speech enhancement systems, namely singlechannel ego-noise reduction (DNN-S), multi-channel beamforming (DNN-BF), and multi-channel time-frequency spatial filtering (DNN-TF). The main novelty lies in the proposed DNN-TF algorithm, which infers the noise-dominance probabilities at individual time-frequency bins from the DNN-estimated soft masks, and then incorporates them into a time-frequency spatial filtering framework for ego-noise reduction. By jointly exploiting the direction of arrival of the target sound, the time-frequency sparsity of the acoustic signals (speech and ego-noise) and the time-frequency noise-dominance probability, DNN-TF can suppress the ego-noise effectively in scenarios with very low signal-to-noise ratios (e.g. SNR lower than −15 dB), especially when the direction of the target sound is close to that of a source of the ego-noise. Experiments with real and simulated data show the advantage of DNN-TF over competing methods, including DNN-S, DNN-BF and the state-of-the-art time-frequency spatial filtering.

Singular Spectrum Analysis for Source Separation in Drone-Based Audio Recording

“…Wiener filter [15]- [17] Kalman filter [18], [19] Neural network [20]- [22] Evolutionary algorithm [23], [24] Non-adaptive Low-pass/high-pass/pass-band filter [25] Wavelet transform [26]- [28] Beam-forming [29] Singular value decomposition Independent component analysis [30] Principal component analysis [31], [32] Singular spectrum analysis [33]- [35] Other [36]- [38] Moreover, the ego-noise is highly non-stationary, as it typically depends on the characteristics of the movements being performed, e.g., speeds and accelerations. The noise produced by a drone has three main components, namely, the mechanical noise generated by the rotation of the motors, the noise generated by the propellers cutting through the air, and the noise of the airflow generated by the propellers themselves.…”

“…Neural network-based approaches can achieve state-ofthe-art results in application domains with great amounts of data. RNNoise [20], put forward by Zhang et al [21], and the Wang et al method [22] are both great alternatives in the voice denoising field. However, they are not as performant in domains with much less data.…”

A Review on Auditory Perception for Unmanned Aerial Vehicles

“…Additionally, the recent LA work of [ 76 ] does not aim to localize either, but to enhance the signal being captured from the ground via a technique based on deep learning. This work can have important implications for further localization or detection from a UAV.…”

“…Mavic Pro [74] Quad [75] Bebop 2 [16] Matrice 100 [16] Quad [76] Quad [77] * Phantom I [78] Quad [79] Quad [80] Hexa [81] Overall, there are some interesting tendencies that are worth pointing out at this moment. In Figure 5, we present a distribution of works revised in this review across recent years and differentiated by their category, i.e., AL, LA, and AA.…”

Applying the majority voting rule in acoustic detection and classification of drones

“…This problem can be addressed using classical signal processing noise reduction algorithms, including frequencyspatial filtering techniques, effective in blind source separation problems [16]. More recently, methods based on Deep Neural Networks (DNN) [17] are also being used to enhance speech signals captured using drones. An example is the work presented in [17] that integrates single-and multi-channel DNN-based approaches for the enhancement of speech signals captured from drones.…”