Recent research shows end-to-end ASR systems can recognize overlapped speech from multiple speakers. However, all published works have assumed no latency constraints during inference, which does not hold for most voice assistant interactions. This work focuses on multi-speaker speech recognition based on a recurrent neural network transducer (RNN-T) that has been shown to provide high recognition accuracy at a low latency online recognition regime. We investigate two approaches to multi-speaker model training of the RNN-T: deterministic output-target assignment and permutation invariant training. We show that guiding separation with speaker order labels in the former case enhances the high-level speaker tracking capability of RNN-T. Apart from that, with multistyle training on single-and multi-speaker utterances, the resulting models gain robustness against ambiguous numbers of speakers during inference. Our best model achieves a WER of 10.2% on simulated 2-speaker LibriSpeech data, which is competitive with the previously reported state-of-the-art nonstreaming model (10.3%), while the proposed model could be directly applied for streaming applications.

No abstract

No abstract

Recent research shows end-to-end ASR systems can recognize overlapped speech from multiple speakers. However, all published works have assumed no latency constraints during inference, which does not hold for most voice assistant interactions. This work focuses on multi-speaker speech recognition based on a recurrent neural network transducer (RNN-T) that has been shown to provide high recognition accuracy at a low latency online recognition regime. We investigate two approaches to multi-speaker model training of the RNN-T: deterministic output-target assignment and permutation invariant training. We show that guiding separation with speaker order labels in the former case enhances the high-level speaker tracking capability of RNN-T. Apart from that, with multistyle training on single-and multi-speaker utterances, the resulting models gain robustness against ambiguous numbers of speakers during inference. Our best model achieves a WER of 10.2% on simulated 2-speaker LibriSpeech data, which is competitive with the previously reported state-of-the-art nonstreaming model (10.3%), while the proposed model could be directly applied for streaming applications.

Postprocessing of confidence scores in keyword search (KWS) task is known to be an efficient way of improving retrieval performance. In this paper, we extend the existing graph-based re-ranking algorithm proposed for KWS score calibration. We replace the originally used Dynamic Time Warping (DTW) distance measure between prospective hits with distances between their Acoustic Word Embeddings (AWEs) learned from Neural Networks. We argue that AWEs trained to discriminate between the same and different words should improve the graphbased re-ranking performance. Experimental results on two languages from IARPA Babel program show that our approach outperforms the DTW and improves the baseline KWS result between 3.0-7.5% relative on the Maximum Term Weighted Value (MTWV) measure. It was previously shown, that enhancing detection lists with keyword exemplars given high confidence, improved the algorithm performance. We additionally expanded the detection lists with negative query exemplars and observed further improvements in MTWV.