Layer-Wise Analysis of a Self-Supervised Speech Representation Model

Abstract

Many self-supervised speech models, varying in their pretraining objective, input modality, and pre-training data, have been proposed in the last few years. Despite impressive empirical successes on downstream tasks, we still have a limited understanding of the properties encoded by the models and the differences across models. In this work, we examine the intermediate representations for a variety of recent models. Specifically, we measure acoustic, phonetic, and word-level properties encoded in individual layers, using a lightweight analysis tool based on canonical correlation analysis (CCA). We find that these properties evolve across layers differently depending on the model, and the variations relate to the choice of pre-training objective. We further investigate the utility of our analyses for downstream tasks by comparing the property trends with performance on speech recognition and spoken language understanding tasks. We discover that CCA trends provide reliable guidance to choose layers of interest for downstream tasks and that single-layer performance often matches or improves upon using all layers, suggesting implications for more efficient use of pre-trained models. 1

Word Discovery in Visually Grounded, Self-Supervised Speech Models

“…In Table 2, we first note that the highest overall performance for the Area and Boundary metrics is achieved by the VG-HuBERT3 model, while the highest performance for the Word metrics is achieved by the VG-HuBERT4 model. For both VG-HuBERT and VG-W2V2, we find that reinitializing the last few layers before training on the visual grounding task is highly beneficial, which is in line with the results found by [32]. A more extensive comparison of layer reinitialization is shown in Table 3.…”

“…Although increasing the number of clusters does lead to a larger number of detectors (Figure 3), we see significant diminishing returns and thus fix K = 4096 in all of our experiments, unless stated otherwise. In Figure 4, we see that for all models, word detection performance is best in the middle and upper half of the model, which is also consistent with the analysis of [32] Finally, we compare the word detection performance of VG-HuBERT when using oracle word boundaries to determine segments (rather than the model's thresholded self-attention), which we counterintuitively find hurts the model's word detection ability. Combined with the observation that the attention segments tend to concentrate at the nucleus of words, we hypothesize that the model's contextualization is pushing word identity information towards the temporal center of each word.…”

“…We will refer to our visually-grounded variants of HuBERT and W2V2 as VG-HuBERT and VG-W2V2. As will be shown in section 3, similar to [32], we found that reinitializing the last few Transformer layers of the pretrained HuBERT or W2V2 with random weights can significantly improve the word discovery performance. We name models that reinitialize weights of the last x Transformer layers VG-HuBERTx and VG-W2V2x.…”

data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

“…The effect is very pronounced for speech and NLP while for vision there is still a slight advantage of predicting more than a single layer. Pasad et al, 2021).…”

Language-Independent Speaker Anonymization Approach using Self-Supervised Pre-Trained Models

“…Many self-supervised speech models have been proposed in previous studies. It has been observed that different layers of SSL-based models contain different information like speaker identity, content and semantics [35][36][37][38]. In particular, higher and middle layers of SSL-based models tend to capture richer linguistic information.…”