Learning Speaker Aware Offsets for Speaker Adaptation of Neural Networks

Abstract

In this work, we present an unsupervised long short-term memory (LSTM) layer normalization technique that we call adaptation by speaker aware offsets (ASAO). These offsets are learned using an auxiliary network attached to the main senone classifier. The auxiliary network takes main network LSTM activations as input and tries to reconstruct speaker, (speaker,phone) and (speaker,senone)-level averages of the activations by minimizing the mean-squared error. Once the auxiliary network is jointly trained with the main network, during test time we do not need additional information for the test data as the network will generate the offset itself. Unlike many speaker adaptation studies which only adapt fully connected layers, our method is applicable to LSTM layers in addition to fully-connected layers. In our experiments, we investigate the effect of ASAO of LSTM layers at different depths. We also show its performance when the inputs are already speaker adapted by feature space maximum likelihood linear regression (fMLLR). In addition, we compare ASAO with a speaker adversarial training framework. ASAO achieves higher senone classification accuracy and lower word error rate (WER) than both the unadapted models and the adversarial model on the HUB4 dataset, with an absolute WER reduction of up to 2%.

Auxiliary Networks for Joint Speaker Adaptation and Speaker Change Detection

“…Our earlier work on adaptation by speaker aware offsets [12], [34] can also be grouped under the affine transformation category where speaker embeddings generated through an auxiliary network are used as bias vectors and subtracted from main network activations. As compared to [12], [34], in the current work, we investigate more general affine transformations. We also experiment with adding a nonlinearity to the transformation.…”

“…The proposed system for joint speaker adaptation and change detection combines the speaker adaptation scheme of [34] with Siamese change detection [30], and introduces an attention block to the auxiliary network. The attention block allows us to integrate ideas from the Siamese network into the new joint model.…”

“…In this work, the problem we address is whether we can construct a single acoustic model that can detect the speaker change and automatically adapt to different speakers simultaneously. In order to develop such a system, we combine ideas from neural network based speaker change detection [30] and unsupervised speaker adaptation by an auxiliary network [12], [34]. The key novel contribution of this work is a method that moves the speaker-change decision inside the speech recognition system, in the form of a soft-decision speaker-attention layer.…”

“…The method then trains the speaker-attention layer explicitly in order to minimize the ASR error rate. The speaker-attention layer is used to accumulate a soft-decision speaker embedding, and from that point onward, the network behaves similarly to [34]. In addition to the mean normalization proposed in [34], we also investigate an affine and a nonlinear transformation of these activations that depend on the speaker embedding generated by the auxiliary network.…”

“…The speaker-attention layer is used to accumulate a soft-decision speaker embedding, and from that point onward, the network behaves similarly to [34]. In addition to the mean normalization proposed in [34], we also investigate an affine and a nonlinear transformation of these activations that depend on the speaker embedding generated by the auxiliary network. We also show that the learned speaker embeddings can be used for speaker segmentation although we do not explicitly train the network with this objective.…”