Single-channel multi-talker speech recognition with permutation invariant training

Abstract

Although great progresses have been made in automatic speech recognition (ASR), significant performance degradation is still observed when recognizing multi-talker mixed speech. In this paper, we propose and evaluate several architectures to address this problem under the assumption that only a single channel of mixed signal is available. Our technique extends permutation invariant training (PIT) by introducing the frontend feature separation module with the minimum mean square error (MSE) criterion and the back-end recognition module with the minimum cross entropy (CE) criterion. More specifically, during training we compute the average MSE or CE over the whole utterance for each possible utterance-level output-target assignment, pick the one with the minimum MSE or CE, and optimize for that assignment. This strategy elegantly solves the label permutation problem observed in the deep learning based multi-talker mixed speech separation and recognition systems. The proposed architectures are evaluated and compared on an artificially mixed AMI dataset with both two-and threetalker mixed speech. The experimental results indicate that our proposed architectures can cut the word error rate (WER) by 45.0% and 25.0% relatively against the state-of-the-art singletalker speech recognition system across all speakers when their energies are comparable, for two-and three-talker mixed speech, respectively. To our knowledge, this is the first work on the multi-talker mixed speech recognition on the challenging speakerindependent spontaneous large vocabulary continuous speech task.

End-to-End Training of Time Domain Audio Separation and Recognition

“…Although multi-speaker source separation can already be performed by combining independently trained front-and back-end systems, the source separator produces artifacts unknown to the ASR system which disturb its performance. According to [19], and as also shown in [8,10], such a mismatch can be mitigated by jointly fine-tuning the whole model at once. We here compare three different variants of joint fine-tuning: (a) fine-tuning just the ASR system on the enhanced signals, (b) finetuning just the front-end by propagating gradients through the ASR system but only updating the front-end parameters and (c) jointly fine-tuning both systems.…”

“…Other works already studied the effectiveness of frequency domain source separation techniques as a front-end for ASR. DPCL and PIT have been efficiently used for this purpose, and it was shown that joint retraining for fine-tuning can improve performance [7,8,10]. E2E systems for single-channel multi-speaker ASR have been proposed that no longer consist of individual parts dedicated for source separation and speech recognition, but combine these functionalities into one large monolithic neural network.…”

“…Based on these source separation techniques, multi-speaker ASR systems have been constructed. DPCL and PIT have been used as frequency domain source separation front-ends for a state-of-theart single-speaker ASR system and extended to jointly trained E2E or hybrid systems [7,8,9,10]. They showed that joint (re-)training can improve the performance of these models over a simple cascade system.…”

Utterance-level Permutation Invariant Training with Latency-controlled BLSTM for Single-channel Multi-talker Speech Separation

“…BLSTM is often used in uPIT-based speech separation systems for its capacity of modeling long time dependency in forward and backward directions [8], [13], [12], [14], [15], [16], [17], [18]. BLSTM has a high latency as long as the utterance.…”

“…It is extended to utterance-level PIT (uPIT) [8] with an utterance-level cost function to further improve the performance. Because uPIT is simple and well-performed, it draws a lot of attention [6], [9], [12], [13], [14], [15], [16], [17], [18], [19]. LSTM [20], [21], [22] and BLSTM [23], [24] are widely used for uPIT to exploit utterance-level long time dependency.…”