Unsupervised Speaker Adaptation Using Attention-Based Speaker Memory for End-to-End ASR

Abstract

We propose an unsupervised speaker adaptation method inspired by the neural Turing machine for end-to-end (E2E) automatic speech recognition (ASR). The proposed model contains a memory block that holds speaker i-vectors extracted from the training data and reads relevant i-vectors from the memory through an attention mechanism. The resulting memory vector (M-vector) is concatenated to the acoustic features or to the hidden layer activations of an E2E neural network model. The E2E ASR system is based on the joint connectionist temporal classification and attention-based encoderdecoder architecture. M-vector and i-vector results are compared for inserting them at different layers of the encoder neural network using the WSJ and TED-LIUM2 ASR benchmarks. We show that M-vectors, which do not require an auxiliary speaker embedding extraction system at test time, achieve similar word error rates (WERs) compared to i-vectors for single speaker utterances and significantly lower WERs for utterances in which there are speaker changes.

Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview

“…On the other hand, owing to the small amounts of data available for adaptation the gains are usually lower that one could obtain with speaker-level clusters. While many approaches use utterances to directly extract corresponding embeddings to use as an auxiliary input for the acoustic model [56]- [59], one can also build a fixed inventory of speaker, domains, or topic codes [60] or embeddings [61], [62] when learning the acoustic model or acoustic encoder, and then use the test utterance to select a combination of these at test stage. The latter approach alleviates the necessity of estimating an accurate representation from small amounts of data.…”

“…It may be possible to relax the utterance-level constraint by iteratively re-estimating adaptation statistics using a number of preceding segment(s) [57]. Extra care usually needs to be taken to handle silence and speech uttered by different speakers, as failing to do so may deteriorate the overall ASR performance [62]- [64].…”

“…IX) was shown to be an effective way for the speaker adaption of E2E models [120] even with very limited raw data from the target speaker. Embeddings have also been used to train a speaker-aware AED model [62], [121], [122].…”

“…The embedding method is also helpful to the adaptation of E2E systems. Fan et al [121] and Sari et al [62] generated a soft embedding vector by combining a set of i-vectors from multiple speakers with the combination weight calculated from the attention mechanism. The soft embedding vector is appended to the acoustic encoder output of the E2E model, helping the model to normalize speaker variations.…”

“…The soft embedding vector is appended to the acoustic encoder output of the E2E model, helping the model to normalize speaker variations. While the soft embedding vectors in [62], [121] are different at each frame, the speaker i-vectors are concatenated with the speech utterance as the input of every encoder layer in [122] to form a persistent memory through the depth of encoder, hence learning utterance-level speaker knowledge. In addition to acoustic embedding, E2E models can also leverage text embedding to improve their modeling accuracy.…”