A Transformer with Interleaved Self-attention and Convolution for Hybrid Acoustic Models

Abstract

Transformer with self-attention has achieved great success in the area of nature language processing. Recently, there have been a few studies on transformer for end-to-end speech recognition, while its application for hybrid acoustic model is still very limited. In this paper, we revisit the transformer-based hybrid acoustic model, and propose a model structure with interleaved self-attention and 1D convolution, which is proven to have faster convergence and higher recognition accuracy. We also study several aspects of the transformer model, including the impact of the positional encoding feature, dropout regularization, as well as training with and without time restriction. We show competitive recognition results on the public Librispeech dataset when compared to the Kaldi baseline at both cross entropy training and sequence training stages. For reproducible research, we release our source code and recipe within the PyKaldi2 toolbox.

Exploring Transformers for Large-Scale Speech Recognition

“…With the interleaved convolution, the accuracy loss is much smaller. This may be due to that the interleaved convolution layers can compensate the reordering effect of self-attention operations, and maintain the monotonicity of the input sequence [25]. When the…”

“…The self-attention operation cannot maintain the monotonicity of input sequence, which is particularly harmful for timesynchronous acoustic model such as the hybrid model studies in this paper. The positional encoding approach in [2] is shown to be less effective to the speech recognition problem [12,25], while convolutional layers are proven to be more powerful to encode the positional information. In Table 1, we compare the two schemes of using convolution layers in Transformers in the offline condition, namely, the interleaved 1D convolution with self-attention from our previous study [25], and using the VGG net [26] as the input encoding layer.…”

“…The positional encoding approach in [2] is shown to be less effective to the speech recognition problem [12,25], while convolutional layers are proven to be more powerful to encode the positional information. In Table 1, we compare the two schemes of using convolution layers in Transformers in the offline condition, namely, the interleaved 1D convolution with self-attention from our previous study [25], and using the VGG net [26] as the input encoding layer. The kernel size for the 1D convolution is 3, while the VGG net has 4 layers of 2D convolutions with 3x3 filters.…”