Conformer: Convolution-augmented Transformer for Speech Recognition

Gulati, Anmol; Qin, James; Chiu, Chung‐Cheng; Parmar, Niki; Zhang, Yu; Yu, Jiahui; Han, Wei; Wang, Shibo; Zhang, Zhengdong; Wu, Yonghui; Pang, Ruoming

doi:10.21437/interspeech.2020-3015

Cited by 1,608 publications

(870 citation statements)

References 0 publications

Supporting

Mentioning

754

Contrasting

Unclassified

Order By: Relevance

“…Following the success of Transformer-based modeling of speech features [7,11], we choose convolution-augmented Transformer or Conformer [7] for the denoising network. For modeling long-term and short-term patterns, it relies on self-attention mechanism and specially designed convolution modules respectively.…”

Section: Baseline Conformer Transformer Enhancement Networkmentioning

confidence: 99%

“…For modeling long-term and short-term patterns, it relies on self-attention mechanism and specially designed convolution modules respectively. Moreover, it combines the power of relative Position Encoding (PE) scheme and Macaron-style half-step Feed-Forward Networks (FFNs) [7]. We additionally include Squeeze-and-Excitation [12] module (squeeze factor of 8) after the 1D Depthwise Convolution inside the convolution module of Conformer.…”

Section: Baseline Conformer Transformer Enhancement Networkmentioning

confidence: 99%

“…Our contributions are as follows. First, we establish a strong baseline based on Conformer Transformer [7] with a simple l1 objective. Second, we propose a rich framework called Perceptual Ensemble Regularization Loss (PERL) which leverages six opensource large-scale pre-trained networks to analyze speech denoising with perceptual losses.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Perceptual Loss Based Speech Denoising with an Ensemble of Audio Pattern Recognition and Self-Supervised Models

Kataria

Villalba

Dehak

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Deep learning based speech denoising still suffers from the challenge of improving perceptual quality of enhanced signals. We introduce a generalized framework called Perceptual Ensemble Regularization Loss (PERL) built on the idea of perceptual losses. Perceptual loss discourages distortion to certain speech properties and we analyze it using six large-scale pre-trained models: speaker classification, acoustic model, speaker embedding, emotion classification, and two self-supervised speech encoders (PASE+, wav2vec 2.0). We first build a strong baseline (w/o PERL) using Conformer Transformer Networks on the popular enhancement benchmark called VCTK-DEMAND. Using auxiliary models one at a time, we find acoustic event and self-supervised model PASE+ to be most effective. Our best model (PERL-AE) only uses acoustic event model (utilizing AudioSet) to outperform state-of-the-art methods on major perceptual metrics. To explore if denoising can leverage full framework, we use all networks but find that our seven-loss formulation suffers from the challenges of Multi-Task Learning. Finally, we report a critical observation that state-of-the-art Multi-Task weight learning methods cannot outperform hand tuning, perhaps due to challenges of domain mismatch and weak complementarity of losses.

show abstract

Section: Baseline Conformer Transformer Enhancement Networkmentioning

confidence: 99%

Section: Baseline Conformer Transformer Enhancement Networkmentioning

confidence: 99%

See 1 more Smart Citation

Perceptual Loss Based Speech Denoising with an Ensemble of Audio Pattern Recognition and Self-Supervised Models

Kataria

Villalba

Dehak

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Conformer [6] is designed as an architecture with a multi-head attention module and a convolutional module, and a pair of feedforward network modules. The multi-head attention module and feedforward network module follow the form of the Transformer, and the convolutional module is the key part of the Conformer, which is mainly a depthwise convolutional layer that is sandwiched between two point-wise convolutional layers.…”

Section: Related Workmentioning

confidence: 99%

“…Over the past few years, some end-to-end models for offline applications [3][4][5][6][7] have gained performance comparable to that of humans. However, these models cannot be directly applied to real-time scenarios because of their high latency.…”

Section: Introductionmentioning

confidence: 99%

A 2D Convolutional Gating Mechanism for Mandarin Streaming Speech Recognition

Wang

Zhao

2021

Information

View full text Add to dashboard Cite

Recent research shows recurrent neural network-Transducer (RNN-T) architecture has become a mainstream approach for streaming speech recognition. In this work, we investigate the VGG2 network as the input layer to the RNN-T in streaming speech recognition. Specifically, before the input feature is passed to the RNN-T, we introduce a gated-VGG2 block, which uses the first two layers of the VGG16 to extract contextual information in the time domain, and then use a SEnet-style gating mechanism to control what information in the channel domain is to be propagated to RNN-T. The results show that the RNN-T model with the proposed gated-VGG2 block brings significant performance improvement when compared to the existing RNN-T model, and it has a lower latency and character error rate than the Transformer-based model.

show abstract