Residual Information in Deep Speaker Embedding Architectures

Abstract

Speaker embeddings represent a means to extract representative vectorial representations from a speech signal such that the representation pertains to the speaker identity alone. The embeddings are commonly used to classify and discriminate between different speakers. However, there is no objective measure to evaluate the ability of a speaker embedding to disentangle the speaker identity from the other speech characteristics. This means that the embeddings are far from ideal, highly dependent on the training corpus and still include a degree of residual information pertaining to factors such as linguistic content, recording conditions or speaking style of the utterance. This paper introduces an analysis over six sets of speaker embeddings extracted with some of the most recent and high-performing deep neural network (DNN) architectures, and in particular, the degree to which they are able to truly disentangle the speaker identity from the speech signal. To correctly evaluate the architectures, a large multi-speaker parallel speech dataset is used. The dataset includes 46 speakers uttering the same set of prompts, recorded in either a professional studio or their home environments. The analysis looks into the intra- and inter-speaker similarity measures computed over the different embedding sets, as well as if simple classification and regression methods are able to extract several residual information factors from the speaker embeddings. The results show that the discriminative power of the analyzed embeddings is very high, yet across all the analyzed architectures, residual information is still present in the representations in the form of a high correlation to the recording conditions, linguistic contents and utterance duration. However, we show that this correlation, although not ideal, could still be useful in downstream tasks. The low-dimensional projections of the speaker embeddings show similar behavior patterns across the embedding sets with respect to intra-speaker data clustering and utterance outlier detection.

An analysis on the effects of speaker embedding choice in non auto-regressive TTS

“…To the best of our knowledge, there are no studies which explore how, and if, the choice of speaker representations and their learning strategies affect the synthesised output in multi-speaker TTS. One study which performed a related task on understanding what the SV-derived embeddings encompass is [17]. The study introduced an evaluation over six freely available neural speaker embedding architectures and the extent to which they encompass residual information related to other speech factors, such as F0, duration, signal-to-noise ratio, speaker gender, and linguistic content.…”

“…The choice for the TitaNet architecture is based on the results of [17] where multiple speaker embedding models were jointly analysed in an effort to determine the amount of residual information present within them. The TitaNet-derived embeddings showed some of the best performances.…”