A Bidirectional Context Embedding Transformer for Automatic Speech Recognition

Abstract

Transformers have become popular in building end-to-end automatic speech recognition (ASR) systems. However, transformer ASR systems are usually trained to give output sequences in the left-to-right order, disregarding the right-to-left context. Currently, the existing transformer-based ASR systems that employ two decoders for bidirectional decoding are complex in terms of computation and optimization. The existing ASR transformer with a single decoder for bidirectional decoding requires extra methods (such as a self-mask) to resolve the problem of information leakage in the attention mechanism This paper explores different options for the development of a speech transformer that utilizes a single decoder equipped with bidirectional context embedding (BCE) for bidirectional decoding. The decoding direction, which is set up at the input level, enables the model to attend to different directional contexts without extra decoders and also alleviates any information leakage. The effectiveness of this method was verified with a bidirectional beam search method that generates bidirectional output sequences and determines the best hypothesis according to the output score. We achieved a word error rate (WER) of 7.65%/18.97% on the clean/other LibriSpeech test set, outperforming the left-to-right decoding style in our work by 3.17%/3.47%. The results are also close to, or better than, other state-of-the-art end-to-end models.

Nonlinear Regularization Decoding Method for Speech Recognition

“…The difference from the bidirectional Transformer [ 13 , 14 ] lies in the unrestricted nature of the NLT decoding process in this paper, which can decode in any order without directional constraints. Specifically, while bidirectional LSTMs use both forward and backward information simultaneously during decoding, this paper allows decoding to proceed independently in any direction.…”

“…Additionally, it adopts a more lightweight model, accelerating training speed and holding significant importance for advancing speech recognition. Subsequently, Transformer decoders have undergone various improvements, such as refining positional information [ 25 ], incorporating positional information into encoder outputs [ 26 ], and enhancing bidirectional semantic information [ 13 , 14 , 27 ]. However, regardless of the type of decoder, there are certain limitations.…”

“…However, attention decoding follows an autoregressive decoding manner, decoding sequences linearly from left to right. Despite bidirectional [ 13 , 14 ] Transformers providing reverse decoding information, the decoding sequence remains relatively monotonic, struggling to extract semantic information effectively, especially on small datasets where training resources are relatively scarce and cannot meet the demands of current popular model training. Meanwhile, attention models are not well-suited for the rescoring approach because, during beam search, many decoding errors may occur.…”