General Sequence Teacher–Student Learning

Abstract

In automatic speech recognition, performance gains can often be obtained by combining an ensemble of multiple models. However, this can be computationally expensive when performing recognition. Teacher-student learning alleviates this cost by training a single student model to emulate the combined ensemble behaviour. Only this student needs to be used for recognition. Previously investigated teacher-student criteria often limit the forms of diversity allowed in the ensemble, and only propagate information from the teachers to the student at the frame level. This paper addresses both of these issues by examining teacher-student learning within a sequence-level framework, and assessing the flexibility that these approaches offer. Various sequence-level teacher-student criteria are examined in this work, to propagate sequence posterior information. A training criterion based on the KL-divergence between context-dependent state sequence posteriors is proposed that allows for a diversity of state cluster sets to be present in the ensemble. This criterion is shown to be an upper bound to a more general KL-divergence between word sequence posteriors, which places even fewer restrictions on the ensemble diversity, but whose gradient can be expensive to compute. These methods are evaluated on the AMI meeting transcription and MGB-3 television broadcast audio tasks.

High-Accuracy and Low-Latency Speech Recognition with Two-Head Contextual Layer Trajectory LSTM Model

“…where X is an input sequence, θT is the teacher, θS is the student, and H is a hypothesis, which may be expressed as a sequence of words, sub-word units, or states [27]. This study considers state sequences.…”

“…where α k is the combination weight for the kth teacher, θT k , satisfying k α k = 1 and α k ≥ 0. The contribution to the T/S gradient from the teachers can be obtained by performing a separate forwardbackward operation over each of the teachers' lattices, which represent each teachers' hypotheses [27]. This is computationally expensive, especially when using a large amount of training data.…”

“…The second direction is to improve upon the hybrid model training strategy. Specifically, on top of sequence discriminative training, we further improve the model accuracy by performing sequence-level teacher-student (T/S) learning [26,27] toward a strong ensemble teacher.…”

Mutual-learning sequence-level knowledge distillation for automatic speech recognition

“…Wu et al [29] proposed a multi-teacher knowledge distillation framework to compress the model by transferring the knowledge from multiple teachers to a single student model. Wong et al [30] proposed to train the student by an ensemble with a diversity of state cluster sets. Simultaneous distillation algorithms [31] trained simultaneously a set of models that learn from each other in a peer-teaching manner.…”