Joint Time-Frequency and Time Domain Learning for Speech Enhancement

Tang, Chengxiang; Luo, Chong; Zhao, Zhiyuan; Xie, Wenxuan; Zeng, Wenjun

doi:10.24963/ijcai.2020/528

Cited by 47 publications

(22 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, deep neural networks have shown great potential for single-channel speech enhancement (SE) (or noisesuppression) [2,3,4,5,6,7,8]. Although these models substantially remove background noise, most of them degrade the performance of downstream tasks such as automatic speech recognition (ASR) performance significantly, as modern commercial multi-condition trained ASR systems can usually recognize original noisy speech [5] well and the SE models introduces unseen distortions that are particularly harmful to ASR.…”

Section: Introductionmentioning

confidence: 99%

Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement

Eskimez

Wang

Tang³

et al. 2021

Interspeech 2021

View full text Add to dashboard Cite

With the surge of online meetings, it has become more critical than ever to provide high-quality speech audio and live captioning under various noise conditions. However, most monaural speech enhancement (SE) models introduce processing artifacts and thus degrade the performance of downstream tasks, including automatic speech recognition (ASR). This paper proposes a multi-task training framework to make the SE models unharmful to ASR. Because most ASR training samples do not have corresponding clean signal references, we alternately perform two model update steps called SE-step and ASR-step. The SEstep uses clean and noisy signal pairs and a signal-based loss function. The ASR-step applies a pre-trained ASR model to training signals enhanced with the SE model. A cross-entropy loss between the ASR output and reference transcriptions is calculated to update the SE model parameters. Experimental results with realistic large-scale settings using ASR models trained on 75,000-hour data show that the proposed framework improves the word error rate for the SE output by 11.82% with little compromise in the SE quality. Performance analysis is also carried out by changing the ASR model, the data used for the ASR-step, and the schedule of the two update steps.

show abstract

Section: Introductionmentioning

confidence: 99%

Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement

Eskimez

Wang

Tang³

et al. 2021

Interspeech 2021

View full text Add to dashboard Cite

show abstract

“…1) Adaptive time-frequency attention Transformer : To alleviate the heavy computational complexity of conventional self-attention, we introduce an adaptive time-frequency attention (ATFA) mechanism as a lightweight solution to capture long-range correlations exhibited in the temporal and spectral axes, as described in [22], [25]. As illustrated in Fig.…”

Section: A Densely Convolutional Encodermentioning

confidence: 99%

“…In each branch, different from the Vanilla transformer, a GRU-based improved transformer [7] is employed, which is comprised of multi-head self-attention (MHSA) and GRUbased position-wise network, followed by residual connections and LN. Multi-head self-attention has been widely used in the natural language processing and speech area, as it can leverage the contextual information in the feature maps [21], [22], [25], [26]. In MHSA modules, the input features are first linearly mapped with different linear projections h times to get queries (Q), keys (K) and values (V ) representation.…”

Section: A Densely Convolutional Encodermentioning

confidence: 99%

DBT-Net: Dual-branch federative magnitude and phase estimation with attention-in-attention transformer for monaural speech enhancement

Yu¹,

Li²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

The decoupling-style concept begins to ignite in the speech enhancement area, which decouples the original complex spectrum estimation task into multiple easier sub-tasks (i.e., magnitude and phase), resulting in better performance and easier interpretability. In this paper, we propose a dual-branch federative magnitude and phase estimation framework, dubbed DBT-Net, for monaural speech enhancement, which aims at recovering the coarse-and fine-grained regions of the overall spectrum in parallel. From the complementary perspective, the magnitude estimation branch is designed to filter out dominant noise components in the magnitude domain, while the complex spectrum purification branch is elaborately designed to inpaint the missing spectral details and implicitly estimate the phase information in the complex domain. To facilitate the information flow between each branch, interaction modules are introduced to leverage features learned from one branch, so as to suppress the undesired parts and recover the missing components of the other branch. Instead of adopting the conventional RNNs and temporal convolutional networks for sequence modeling, we propose a novel attention-in-attention transformer-based network within each branch for better feature learning. More specially, it is composed of several adaptive spectro-temporal attention transformerbased modules and an adaptive hierarchical attention module, aiming to capture long-term time-frequency dependencies and further aggregate intermediate hierarchical contextual information. Comprehensive evaluations on the WSJ0-SI84 + DNS-Challenge and VoiceBank + DEMAND dataset demonstrate that the proposed approach consistently outperforms previous advanced systems and yields state-of-the-art performance in terms of speech quality and intelligibility.

show abstract

“…Recently, the progress of deep learning algorithms has brought substantial improvements also in the SE field [ 14 , 15 , 16 , 17 , 18 , 19 ]. Deep learning techniques are data-driven approaches that frames the SE task as a supervised learning problem aiming at reconstructing the target speech signals from the noisy mixture.…”

Section: Introductionmentioning

confidence: 99%

Time-Domain Joint Training Strategies of Speech Enhancement and Intent Classification Neural Models

Ali

Falavigna

Brutti

2022

Sensors

View full text Add to dashboard Cite

Robustness against background noise and reverberation is essential for many real-world speech-based applications. One way to achieve this robustness is to employ a speech enhancement front-end that, independently of the back-end, removes the environmental perturbations from the target speech signal. However, although the enhancement front-end typically increases the speech quality from an intelligibility perspective, it tends to introduce distortions which deteriorate the performance of subsequent processing modules. In this paper, we investigate strategies for jointly training neural models for both speech enhancement and the back-end, which optimize a combined loss function. In this way, the enhancement front-end is guided by the back-end to provide more effective enhancement. Differently from typical state-of-the-art approaches employing on spectral features or neural embeddings, we operate in the time domain, processing raw waveforms in both components. As application scenario we consider intent classification in noisy environments. In particular, the front-end speech enhancement module is based on Wave-U-Net while the intent classifier is implemented as a temporal convolutional network. Exhaustive experiments are reported on versions of the Fluent Speech Commands corpus contaminated with noises from the Microsoft Scalable Noisy Speech Dataset, shedding light and providing insight about the most promising training approaches.

show abstract

Joint Time-Frequency and Time Domain Learning for Speech Enhancement

Cited by 47 publications

References 1 publication

Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement

Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement

DBT-Net: Dual-branch federative magnitude and phase estimation with attention-in-attention transformer for monaural speech enhancement

Time-Domain Joint Training Strategies of Speech Enhancement and Intent Classification Neural Models

Contact Info

Product

Resources

About